Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Creating my personal cloud with HashiCorp

cgamesplay.com

Post reply on HN
Print view View original

81–90 of 90 posts

Re: Creating my personal cloud with HashiCorp

#81

Post by hnlmorg »

whazor wrote: post #29
hnlmorg wrote: post #12

Earlier quoted context omitted.

I'm not the biggest fan of Terraform either but it would be naive to deny that Terraform doesn't still offer a lot of advantages over CloudFormation even if you're not writing cloud agnostic code (even though I do actually agree with your point that Terraform doesn't make your infra cloud agnostic). Terraform offers far more constructs than CloudFormation and there is still a lot to be said for using the same languag…

Terraform Language server together with VSCode is working quite well.

That's what I use and it is better than nothing but nowhere near as finely tuned as other language servers:

1. It's slow to return results

2. sometimes the only way you can generate a list of available properties is to type the first character, which is really annoying if you don't even know what the first character might be

3. it offers suggestions for properties that aren't even valid for that resource

4. there is no description against any of the properties so you're still left guessing (or having the docs open in another window) anyway

Re: Creating my personal cloud with HashiCorp

#82

Post by hnlmorg »

andreineculau wrote: post #22
hnlmorg wrote: post #13

Earlier quoted context omitted.

For one, it's closer to a proper programming language as opposed to straight up data interchange format. Sure if you write it in YAML than you can take advantage of variables but YAML's syntax for variables is pretty gross. Comparing CloudFormation to Terraform is a little like comparing HTML and CSS to Javascript (though Terraform isn't nearly as nice to code in as Javascript -- and I'm not exactly a big fan of Java…

> For one, it's closer to a proper programming language as opposed to straight up data interchange format. Sure if you write it in YAML than you can take advantage of variables but YAML's syntax for variables is pretty gross. So what is stopping you from using "a proper programming language" to generate the json/yaml cloudformation template? This is what you see in GCP docs from day one. On AWS, they brainwashed ever…

> So what is stopping you from using "a proper programming language" to generate the json/yaml cloudformation template?

Nothing and there are already products out there that offer that. However I think the issue will always fall back to the problem that you're compiling from a functional or procedural language down to a dumb data interchange format. That can cause a variety of issues such as losing your carefully crafted order of execution.

> This is what you see in GCP docs from day one. On AWS, they brainwashed everyone in this corner of "static template with parameters", so that you can "reuse" a template to build your custom stack. It's great for "look what I can do, mom" (but I have no idea what it's doing) but nobody sane would ever trust a 1-km long yaml/json and deploy it. So if you anyway have to inspect it, why not make it easy to inspect? Split into modules, add docs, etc = code to run.

I don't think anyone was "brainwashed" by CloudFormation and the solution you describe is exactly the approach Terraform takes.

> I have no idea how we switched from random scripts to "reusable" random scripts (ansible &co) to random static configuration and then the cherry on top: "reusable" random static configuration. Insane. Abstractions on top of abstractions.

This I agree with. It's not just AWS though, you see YAML-based config all over the place from CI/CD pipelines to Kubernetes pods. And they all suffer from the same problems. It's easily my least favourite thing about the DevOps modernisation of what would have been random sysadmin shell scripts 10 years ago. Frankly I'm not convinced these YAML files are any more readable nor less brittle than the duct tape we wrote in #!/bin/sh before.

> CDK is on the right track, but even there it's a mess, again for the sake of hiding complexity: constructs and deployment. Where did One thing well and Keep it simple stupid go? :)

As I'd written elsewhere, I think CDK is aimed at a subtly different audience. CF, TF, etc are more focused around the sysadmin side of DevOps, whereas CDK are more focused around the application developers end of the DevOps tool chain. That's not to say sysadmin guys can't use CDK, but rather than CDK doesn't just aim to deploy infra, it embeds and deploys the serverless applications (like lambda code) as well. It's more akin to the "full stack" style of developers and not every team nor individual who's job it is to manage cloud infra is going to be an application developer. Particularly in larger organisations. So there definitely is a place for cloud infra stacks to be described in less sophisticated languages (even if it doesn't appeal to you and I personally).

Re: Creating my personal cloud with HashiCorp

#83

Post by hnlmorg »

throwaway894345 wrote: post #50
AstroDogCatcher wrote: post #20

Earlier quoted context omitted.

Out of interest, do you find yourself writing actual software with CDK stacks integrated, or is it more accurate to say the CDK is just a stand-alone bit of code purely for deploying infrastructure? I'm definitely in the latter camp, which is something I find frustrating. I get that for a developer the syntax familiarity might make CDK easier, but for me as a non-developer the pain of groping around the terrible docu…

I strongly suspect that these CDKs are not very well designed. In particular, what I want is something that lets me generate YAML/etc in a type-safe fashion. That YAML is then the input for an engine which reconciles the desired state with the actual state (a la Terraform or cloudFormation). The idea is that the “real programming language” layer just allows us to DRY our YAML. For a use case like this, we don’t need…

It's important to note that Terraform doesn't compile down to CloudFormation stacks like CDK does. Terraform providers instead call AWS APIs directly. This should allow Terraform much more composability than CDK for the very points you've listed unfortunately Terraform does still compile down to a fucking JSON state -- which is easily my biggest complaint about Terraform.

My issue with anything that ultimately compiles to a JSON or YAML state/config is that you lose the dependency tree (or your dependency tree becomes rigidly defined by the way the transpiler converts your code into JSON). It causes so many problems on any large project that ultimately the only solution is to break the project up into smaller distinct projects within the same git repository. Which is basically the same solution to working with JSON/YAML CF stacks directly.

If someone can create a language (or SDK for $PROGRAMMING_LANG) that then worked with AWS APIs directly, (like Terraform), didn't just transpile back to JSON, and isn't as verbose as using boto3 directly -- well I think that product might stand a real chance of displacing Terraform.

I've got a lot of strong thoughts on how this could be done right so I did consider having an attempt at it myself using the shell scripting language I'd designed as a rough foundation. But having a full time job already, two young kids, and a growing in popularity open source project (namely my aforementioned $SHELL), I realised any Terraform competitor I did create would be doomed to either never being maintained, or the project would literally burn me out.

Re: Creating my personal cloud with HashiCorp

#84

Post by throwaway894345 »

hnlmorg wrote: post #83
throwaway894345 wrote: post #50

Earlier quoted context omitted.

I strongly suspect that these CDKs are not very well designed. In particular, what I want is something that lets me generate YAML/etc in a type-safe fashion. That YAML is then the input for an engine which reconciles the desired state with the actual state (a la Terraform or cloudFormation). The idea is that the “real programming language” layer just allows us to DRY our YAML. For a use case like this, we don’t need…

It's important to note that Terraform doesn't compile down to CloudFormation stacks like CDK does. Terraform providers instead call AWS APIs directly. This should allow Terraform much more composability than CDK for the very points you've listed unfortunately Terraform does still compile down to a fucking JSON state -- which is easily my biggest complaint about Terraform. My issue with anything that ultimately compil…

Neither system deals well with large projects, but my CF criticisms weren’t predicated on the project size. So at the end of the day, CF has all of the problems TF has and then some.

But I do think that it would be really interesting to build an IaC project, and you’re right that it probably would be doomed. :)

Re: Creating my personal cloud with HashiCorp

#85

Post by vgb2k18 »

> they still feel usable after 6 months of use. By this time, most of the context that I learned to develop the project has faded away and I have to rely on the documentation that I left myself.

This was a much-needed reminder that the target audience for my documentation is my future-self.

Re: Creating my personal cloud with HashiCorp

#86

Post by [deleted] »

CGamesPlay wrote: post #24
reidjs wrote: post #4

What does it mean to host a personal cloud vs hosting a server? I thought the point of using cloud services was so you didn’t have to host it yourself.

I am using "personal cloud" to refer to the set of "cloud services" that I am hosting myself on my own server. The advantage of having the server is that my devices have a central syncing hub, and the advantage of hosting myself is no need to trust third parties. There are obviously disadvantages, like having to host it myself.

[deleted]

Re: Creating my personal cloud with HashiCorp

#87

Post by bradknowles »

shell0x wrote: post #5

As a DevOps guy, I'm not a huge fan of Terraform. Often I hear from enterprises that Terraform is cloud agnostic, but that's often very wrong. Terraform modules are still specific to the cloud platform and a rewrite is required to port an app running on AWS to GCP. If you use AWS, you're probably better off to use AWS Cloudformation and for GCP Google Cloud Deployment manager. A business reason is often that the engi…

I’ve used CloudFormation at a previous employer. I hate, loathe, and despise the thing.

YAML is not a programming language, and any attempt to turn it into a programming language is fraught with pain, woe, and suffering — at least on the part of the poor suckers who are tricked into using it.

Give me a real programming language, with interfaces to the appropriate constructs necessary to do the job, via an SDK. In AWS, that means using CDK, not CloudFormation.

Feel free to argue over whatever aspects of TerraForm that you don’t like, but please, for the love of ${DEITY}, please do not recommend CloudFormation as your suggested alternative.

Re: Creating my personal cloud with HashiCorp

#88

Post by dragonwriter »

bradknowles wrote: post #87
shell0x wrote: post #5

As a DevOps guy, I'm not a huge fan of Terraform. Often I hear from enterprises that Terraform is cloud agnostic, but that's often very wrong. Terraform modules are still specific to the cloud platform and a rewrite is required to port an app running on AWS to GCP. If you use AWS, you're probably better off to use AWS Cloudformation and for GCP Google Cloud Deployment manager. A business reason is often that the engi…

I’ve used CloudFormation at a previous employer. I hate, loathe, and despise the thing. YAML is not a programming language, and any attempt to turn it into a programming language is fraught with pain, woe, and suffering — at least on the part of the poor suckers who are tricked into using it. Give me a real programming language, with interfaces to the appropriate constructs necessary to do the job, via an SDK. In AWS…

> YAML is not a programming language

To bw fair, while CF can be written in YAML for less visual noise, its fundamentally JSON.

Also, its not a programming language, its a target-state description language.

> Give me a real programming language, with interfaces to the appropriate constructs necessary to do the job, via an SDK. In AWS, that means using CDK, not CloudFormation.

Technically,“CloudFormation via CDK”. You can't do “CDK, not CloudFormation” because CDK is just a tool for generating CloudFormation.

Re: Creating my personal cloud with HashiCorp

#89

Post by lupire »

chromatin wrote: post #43

Great article. I’ve also just started using nomad to manage personal services on my home loan. What I particularly like is that I have a mixture of Docker containers, VMs, and LXD containers all centrally managed. Overall I found nomad to be fairly intuitive, and the single binary/single job paradigm of the Hashicorp stack is very appealing.

home loan?

Re: Creating my personal cloud with HashiCorp

#90

Post by ddfx00 »

mitchellh wrote: post #71
shell0x wrote: post #5

As a DevOps guy, I'm not a huge fan of Terraform. Often I hear from enterprises that Terraform is cloud agnostic, but that's often very wrong. Terraform modules are still specific to the cloud platform and a rewrite is required to port an app running on AWS to GCP. If you use AWS, you're probably better off to use AWS Cloudformation and for GCP Google Cloud Deployment manager. A business reason is often that the engi…

(Creator of Terraform, Co-Founder of HashiCorp) I'm quite late to respond here, but just wanted to clarify one thing: Terraform is WORKFLOW agnostic, not TECHNOLOGY agnostic. This is a key part of our product philosophy that we make the 1st element of our Tao: https://www.hashicorp.com/tao-of-hashicorp I've talked about this more with more references in this tweet: https://twitter.com/mitchellh/status/107868276596335…

I also have to use Terraform and I try to avoid looking at it as much as possible. Hashicorp HCL is incredibly bad (not to use stronger words). I am trying to use an alternative like Pulumi whenever possible, but thankfully there is some hope with cdktf, since Terraform is today effectively technical debt. I can't see how anyone thought that HCL, a terrible language to use in any aspect, was a better choice than a general purpose language like python or js. Every one in our infra org utterly despises HCL. The workflow or Terraform is also very poor, it promotes bad practices and does not offer decent guarantees. I know I sound harsh, but I hope it is received as constructive criticism
Post reply on HN