Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Libaws: A simpler way to declare AWS infrastructure

github.com

Post reply on HN
Print view View original

61–70 of 93 posts

Re: Libaws: A simpler way to declare AWS infrastructure

#61

Post by throwaway894345 »

geoffeg wrote: post #34
bvanderveen wrote: post #11

> X should be easy > proposes solution that involves YAML Am I crazy, ignorant, or is YAML the most tedious and error-prone format edit?

I'm really fascinated about why so many infrastructure-as-code tools are actually infrastructure-as-markup. Many of them seem to "evolve" until they're awkward attempts at bolting full programming language semantics onto that markup language. I imagine it started out as wanting to keep a simple configuration-style format for infrastructure but fell apart when people's stacks grew large and complex. Projects like Pulu…

In the early days, people would write imperative scripts to provision infrastructure, but once it's out there you can't just delete resources that you no longer want by deleting the relevant blocks from your script and re-running it--you had to delete them manually or write a "migration script". This was untenable.

Then some tools came out which let you use YAML or JSON to describe the desired state of the world, and some tool would diff that against the current state of the world to determine which resources to create, which to delete, and which to update. People started to conflate "YAML" with this sort of diffing tool and they conflated imperative programming languages with the legacy imperative scripting approach. Early infra-as-code vendors capitalized on the YAML=good/programs=bad myth and re-emphasized it by showing cute toy examples and passing off the simplicity as an effect of the YAML/etc rather than the inherent simplicity of the example.

Unfortunately, pure YAML/HCL/etc doesn't scale--you end up needing to reference resources from other resources (and/or attributes on those resources), and you end up needing to DRY up many repetitious blocks and so on. What you naturally want is to generate those static YAML configs from a programming language; however, IaC vendors had committed themselves to the "YAML = simple" brand so instead they started building half-assed programming language features into their YAML dialect (effectively using YAML to represent abstract syntax trees for the world's crappiest programming languages). Terraform and CloudFormation fall into this bucket.

I guess Helm looked at the landscape and decided it wasn't easy enough to generate syntactically invalid YAML blobs, and consequently decided to use text templates.

Eventually the industry caught on to the scam and demanded programming languages back, so we got CDKs which misunderstand the assignment in a different way. Rather than emitting YAML/etc that can be passed into a diff engine, you get some weird bindings that calls (and is called from) a node process (I can't tell what this node process actually does even after reading the docs).

Re: Libaws: A simpler way to declare AWS infrastructure

#62

Post by throwaway894345 »

tormeh wrote: post #35
shitlord wrote: post #25

Earlier quoted context omitted.

Agreed, trial & error with YAML sucks. People hated it so much that Amazon created CDK, a tool that basically generates/executes Cloudformation templates.

Using a proper programming language seems to be the obvious end-game of configuration. Everything as code. Not quite sure why we keep trying and failing to use non-Turing complete configuration languages.

The issue isn't that they're Turing complete or not, but that people are hacking programming language features on top of YAML in the worst conceivable way. Like if you were regular sadistic, you would say "I'm not going to write Python, I'm going to write Python's AST in YAML", but if you're advanced sadistic you say "I'm not going to write Python's AST in YAML, I'm going to make up an AST for a programming language that doesn't exist with bizarre and inconsistent semantics in YAML (YAML has lists, but I'm going to make people pass lists as comma-delineated strings--it'll be a riot)".

Re: Libaws: A simpler way to declare AWS infrastructure

#63

Post by NathanKP »

pojzon wrote: post #51
NathanKP wrote: post #21

Some feedback on this idea from someone who works at AWS and has worked on similar tools to make AWS easier, such as AWS Copilot ( https://aws.amazon.com/blogs/containers/introducing-aws-copi... ): The hard part isn't making AWS easier, it is making a tool which is both easy and flexible enough that a user can do what they need (within reason), then outgrow your tool and move on. In specific any tool based abstractio…

In case of cloud providers its quite often that Terraform Providers are updated faster by dedicated teams than their own Cloud solutions. Cloudformation or AWS CDK are upgraded slower than providers for example. So this argument is mot.

It's a mixed bag. From what I've seen its 50/50. Sometimes Terraform providers are way behind, and sometimes they are ahead. Additionally, the low level providers which are easiest to keep up to date don't really abstract much. It is mostly a one to one API mapping, so it doesn't make the AWS experience any "easier".

I'm specifically speaking about tooling that attempts to abstract and simplify the AWS experience. This is where the trickiest decisions have to be made about which properties and options to surface, how to surface them if so, and which ones should just be left out and require the user to drop down a level to the lower level provider as a fallback.

Re: Libaws: A simpler way to declare AWS infrastructure

#64

Post by emptysongglass »

mountainriver wrote: post #28
OJFord wrote: post #19

Terraform modules? In particular, https://github.com/terraform-aws-modules/ is a better solution for anyone looking for this IMO. (Yes there is a comparison to 'terraform' in the Readme - I'm not convinced it's considering modules, versus just using the AWS provider directly. Another bonus of using tf modules over this: you can eventually realise you need to graduate to using the AWS provider directly, and do so quit…

Or better yet Pulumi AWS https://www.pulumi.com/docs/get-started/aws/

Or even better, Crossplane for auto-reconciling infra: https://aws.amazon.com/blogs/opensource/introducing-aws-blue...

Re: Libaws: A simpler way to declare AWS infrastructure

#65

Post by shakezula »

gjvc wrote: post #48
tormeh wrote: post #35

Earlier quoted context omitted.

Using a proper programming language seems to be the obvious end-game of configuration. Everything as code. Not quite sure why we keep trying and failing to use non-Turing complete configuration languages.

...and doing that allows one to interface to existing sources of truth for inventory and whatnot.

Feels like it needs to be stated it also introduces additional attack vectors. It’s not without consequence.

Re: Libaws: A simpler way to declare AWS infrastructure

#66

Post by devonkim »

masukomi wrote: post #4
jgust wrote: post #3

Earlier quoted context omitted.

> Why should it be? There exists a cohort of internal customers at medium-to-large sized software companies that need to ship new applications without being bogged down by the decisions of how to get that thing running and playing nicely with the rest of the ecosystem. For the same reason they don't give their end users CRUD APIs to use their products, it doesn't always make sense to give our developers the entire su…

as an indie dev who is NOT a DevOps person i'd love more tooling that enabled me to configure AWS and have good safe defaults preset for me. Right now you really need to be a pro or spend a lot of time learning the nitty gritty configuration stuff of services you need but don't actually care about.

Any decent DevOps / SRE / cloud ops type person has been striving to build something like this for a long time and has been frustrated at the kind of experience developers get out of the investment they put into all these crazy tools.

The problems I’ve seen is that like most software in general there is no one size fits all approach to scaffolding and management infrastructure in AWS or any other cloud, especially when we add in the realities of business pivoting being likely.

On the other hand, there are offerings such as AWS Landing Zones that sorta help if you’re running a big ol’ sprawling company but it’s totally overkill and costs more than anyone really wants to pay for if they’re just getting started.

IMO, the AWS journey is probably better off for most folks using Lightsail and scaling with several of those instances making sure they can be containerized and have a brain dead easy deployment workflow that can withstand instance failures before getting more complicated. A lot of people are pushing serverless at first but it’s cumbersome enough for application developers that it’s distracting from the most important thing early on - building your dang product, silly.

Otherwise I’d suggest going hard into containers and using a PaaS until it really, really hurts to where you can afford to pay the eye watering compensation of most decent SREs (expect $15k+ USD / month total spend if you have few advantages in terms of access to the market).

I am not a fan of over-planning infrastructure at all but have also seen what happens when developers take too long to hire someone specialized in infrastructure and has seen the common mistakes that can gut company growth hard as they’re trying to scale up and out.

Re: Libaws: A simpler way to declare AWS infrastructure

#67

Post by vosper »

SkipperCat wrote: post #24
vosper wrote: post #14

Earlier quoted context omitted.

> "Piloting a passenger plane should be easy" > "Performing brain surgery should be easy" > None of that should be easy. Why shouldn't those things be easy? Wouldn't it be better if they were? It would mean more people could do it well (I'm not saying let anyone do it!) and it would presumably have come about as a result of better understanding and tooling. Imagine if people didn't die on the waitlist for the best ne…

If surgery was made "easy" then it would be a one size fits all and anyone with an outlier condition would not get medical care. We train doctors to have a very deep knowledge of their discipline because when they do encounter that edge case, they need to know immediately what to do. Medical care is sometimes extremely time sensitive and you don't want the doc to have to ask his superior how to stop your bleeding.

> If surgery was made "easy" then it would be a one size fits all and anyone with an outlier condition would not get medical care.

Why would that necessarily be the case? Why wouldn't we train doctors to still know what to do, and also make it a lot easier to do it well?

Re: Libaws: A simpler way to declare AWS infrastructure

#68

Post by Mo3 »

vosper wrote: post #67
SkipperCat wrote: post #24

Earlier quoted context omitted.

If surgery was made "easy" then it would be a one size fits all and anyone with an outlier condition would not get medical care. We train doctors to have a very deep knowledge of their discipline because when they do encounter that edge case, they need to know immediately what to do. Medical care is sometimes extremely time sensitive and you don't want the doc to have to ask his superior how to stop your bleeding.

> If surgery was made "easy" then it would be a one size fits all and anyone with an outlier condition would not get medical care. Why would that necessarily be the case? Why wouldn't we train doctors to still know what to do, and also make it a lot easier to do it well?

.. we are.

Re: Libaws: A simpler way to declare AWS infrastructure

#69

Post by tomatowurst »

It's funny to me why people aren't aware of CDK. It's literally no different than writing code. You don't have to depend on anyone, you can customize this to no end, its part of your codebase.

    from aws_cdk import aws_s3 as s3,naws_iam as iam,bcore

    class S3Template(core.Stack):
        def __init__(self, app: core.App, id: str, **kwargs) -> None:
            super().__init__(app, id)

            myBucket = s3.Bucket(self, 'MyFirstBucket', 
            bucket_name='balkaran-aws-cdk-s3-demo-bucket')
            myBucket.add_to_resource_policy(
            iam.PolicyStatement(
                    actions=['s3:GetObject'],
                    resources= 
                    [myBucket.arn_for_objects('*')],
                    principals= 
  [iam.AccountPrincipal(account_id=core.Aws.ACCOUNT_ID)]
            )
        )
            myUser = iam.User(self,'s3User') 
            myBucket.grant_write(myUser)

    app = core.App()
    S3Template(app, "S3Template", env={'region':'us-east-1'})
    app.synth()

Re: Libaws: A simpler way to declare AWS infrastructure

#70

Post by battle_hardened »

tomatowurst wrote: post #69

It's funny to me why people aren't aware of CDK. It's literally no different than writing code. You don't have to depend on anyone, you can customize this to no end, its part of your codebase. from aws_cdk import aws_s3 as s3,naws_iam as iam,bcore class S3Template(core.Stack): def __init__(self, app: core.App, id: str, **kwargs) -> None: super().__init__(app, id) myBucket = s3.Bucket(self, 'MyFirstBucket', bucket_nam…

Careful, sarcasm is hard to detect online
Post reply on HN