Converting Full Terraform Programs to Pulumi • Hacker News Board

Re: Converting Full Terraform Programs to Pulumi

#21

Post by irjustin » Tue, Jun 13, 2023, 3:35 AM UTC

Honest question, if you're locked in to AWS why not just use the CDK?

I've used Terraform and Pulumi in the past and the "platform agnostic" is only true for trivial builds. Once you get into medium sized infra you're writing so much AWS specific code it stands better to go 1st party.

I can't speak for GCP, but this is what happened to us. We kept fighting 3rd party code we ended up going CDK. While there are still issues, there were less issues. Cloudformation manages the state under the hood anyway so we're all just stuck wrapping that sucker, even in the CDK.

Re: Converting Full Terraform Programs to Pulumi

#22

Post by jryan49 » Tue, Jun 13, 2023, 3:56 AM UTC

Isn't pulumi aws just terraform under the hood still? Why wouldn't I just use terraform's CDKTF instead?

Re: Converting Full Terraform Programs to Pulumi

#23

Post by x3n0ph3n3 » Tue, Jun 13, 2023, 4:01 AM UTC

__jem wrote: post #20

x3n0ph3n3 wrote: post #12

My biggest issue with Pulumi is that it doesn't have all of the guardrails/tools (like tfsec, tflint, and pavlov) to prevent newbies from doing bad things. I highly considered it for my team's project, but they are not mature enough developers for me to trust them not to massively screw things up or spend forever doing fairly simple things.

One of the big benefits of using an actual programming language rather than HCL is that it allows you to define abstractions / use the type system to enforce these invariants. I find this massively preferable to something like a linter in terms of being able to provide a nice API to engineers who might need a little more help defining their resources.

Oh, I agree! I use pulumi for my own personal projects, but I don't have the time to build a DSL on top of it pulumi just to establish the same guardrails my team needs that I'd otherwise get for free.

Re: Converting Full Terraform Programs to Pulumi

#24

Post by Volundr » Tue, Jun 13, 2023, 4:17 AM UTC

__jem wrote: post #18

preommr wrote: post #17

I really dislike pulumi's pricing structure. I absolutely hate the idea of paying credits for each resource that I create - it creates resistance to experiment and just create a bunch of resources that don't cost anything on the cloud provider yet somehow costs me money for the tool to manage them. It's funny because I believe they used to have a per seat licensing and then people complained that they were paying too…

You can use Pulumi for free. Also, the incremental cost for each resource is minuscule. Unless your experiments require thousands of resources, it really shouldn't be a big deterrent. Just clean up when you're done experimenting.

Sort of. You can use Pulumi for free as an individual. As a team you'd have to use a backend for your state other than Pulumi cloud, and your responsible for your own concurrency control [1]. In a team environment that's such a big foot gun I don't really consider it usable.

That limitation makes Pulumi a nonstarter for me. Even if I am paying for it, this is super sensitive data and I want control over how it's stored. I'm sure Pulumi cloud is excellent, but I hate it's the only real option.

[1] https://www.pulumi.com/docs/concepts/state/

Re: Converting Full Terraform Programs to Pulumi

#25

Post by rjbwork » Tue, Jun 13, 2023, 4:51 AM UTC

dronacharya wrote: post #19

s_Hogg wrote: post #6

Earlier quoted context omitted.

Yeah we have a 10Mb state file and it already takes 30-45 min to deploy. The main way it happens is if someone cancels a CI/CD job because they're realised something is wrong and don't want to wait that long to try again. That's not an issue if there isn't a state file, but yeah I can see how there would be a tradeoff possibly associated.

That does seem like quite a large state file. Would you be able to shed some light on the sorts of of resources that you're provisioning? I've found it to be useful to split dependencies between cloud resources, and to add links via data lookups. At any reasonable scale, I've found splitting Terraform state to be crucial. For example: network resources and Kubernetes clusters are created separately, and networking is…

How do you manage the meta DAG between all of that stuff in the event of, e.g. complete disaster recovery?

Re: Converting Full Terraform Programs to Pulumi

#26

Post by mzi » Tue, Jun 13, 2023, 4:52 AM UTC

jryan49 wrote: post #22

Isn't pulumi aws just terraform under the hood still? Why wouldn't I just use terraform's CDKTF instead?

Yes, pulumi is just wrapping terraform[1]. So you need to understand both the quirks of that and the quirks of pulumi. And I'm lazy so I just want to deal with one quirk at the time.

[1] https://github.com/pulumi/pulumi-terraform-bridge

Re: Converting Full Terraform Programs to Pulumi

#27

Post by renewiltord » Tue, Jun 13, 2023, 4:55 AM UTC

Volundr wrote: post #24

__jem wrote: post #18

Earlier quoted context omitted.

You can use Pulumi for free. Also, the incremental cost for each resource is minuscule. Unless your experiments require thousands of resources, it really shouldn't be a big deterrent. Just clean up when you're done experimenting.

Sort of. You can use Pulumi for free as an individual. As a team you'd have to use a backend for your state other than Pulumi cloud, and your responsible for your own concurrency control [1]. In a team environment that's such a big foot gun I don't really consider it usable. That limitation makes Pulumi a nonstarter for me. Even if I am paying for it, this is super sensitive data and I want control over how it's stor…

The Terraform equivalent is pretty good. You can just use DynamoDB for concurrency control, S3 for state control, and it's very easy to set up. I imagine Pulumi is not very different on this front. Very little boiler plate and you're off to the races.

Re: Converting Full Terraform Programs to Pulumi

#28

Post by latchkey » Tue, Jun 13, 2023, 4:58 AM UTC

irjustin wrote: post #21

Honest question, if you're locked in to AWS why not just use the CDK? I've used Terraform and Pulumi in the past and the "platform agnostic" is only true for trivial builds. Once you get into medium sized infra you're writing so much AWS specific code it stands better to go 1st party. I can't speak for GCP, but this is what happened to us. We kept fighting 3rd party code we ended up going CDK. While there are still i…

CDK is the correct answer.

Re: Converting Full Terraform Programs to Pulumi

#29

Post by fodkodrasz » Tue, Jun 13, 2023, 5:50 AM UTC

s_Hogg wrote: post #3

Do these guys need a state file like terraform does? It's great that you can use a normal programming language, but that's nowhere near as big a pain point as state files getting out of sync. Not even close.

The claimed pro of being able to use a normal programming language fades away quickly as you are forced into non-idiomatic boilerplate patterns to get pulumi to work.

State is a problem and have had just the same kind of hardships when had to manually modify it on some changes as I would have had on terraform. (transfer resources between state stores as a business function is handed over within company, with zero downtime, import docs are not always correct/up2date just like in case of terraform, as if the example codes were generated by chatgpt)

Overall pulumi couldn't convince me, it wasn't a bit more convenient than terraform, yet I could bump into its bugs for pretty basic aws features (around lambdas/apigw, but was more than a year ago, might be fixed already) which did work just fine with terraform ootb, and the pulumi code felt bad to look at despite being in a language I generally like. It did not bring the effect management wanted that developers with no terraform knowledge can be onboarded to infra, as of course infra knowledge is needed and is harder to pick up than learning terraform syntax.

Despite my dislike for terraform and its syntax's limitations in expressiveness, I'm still back to TF, and is more productive than pulumi was.

Re: Converting Full Terraform Programs to Pulumi

#30

Post by fcarletti » Tue, Jun 13, 2023, 5:52 AM UTC

irjustin wrote: post #21

Honest question, if you're locked in to AWS why not just use the CDK? I've used Terraform and Pulumi in the past and the "platform agnostic" is only true for trivial builds. Once you get into medium sized infra you're writing so much AWS specific code it stands better to go 1st party. I can't speak for GCP, but this is what happened to us. We kept fighting 3rd party code we ended up going CDK. While there are still i…

It's not about being vendor agnostic, but being able to manage resources outside of AWS. in my use cases, our platform integrated GitHub, Auth0, AWS, kubernetes and so on. Everything managed via Pulumi, what would not be feasible if we were just on Cloudformation.

When trying to achieve a cloud agnostic approach, there are some other tools focused on that (such as serverless framework).