Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Terraform Cloud is down for nearly five hours, and counting

status.hashicorp.com

Post reply on HN
Print view View original

11–16 of 16 posts

Re: Terraform Cloud is down for nearly five hours, and counting

#11

Post by pxc »

emptysongglass wrote: post #9
mardifoufs wrote: post #6

Earlier quoted context omitted.

I think the founder left the corporation. Not certain about it though, can't find definitive proof (a part from not being ceo anymore, but maybe he still has some role inside?)

He said he wanted to be an ICP again but now seems more concerned with writing his own terminal over saving his own company. https://twitter.com/mitchellh/status/1694785322318712960?s=2...

In June of this year (there are no snapshots for July), his Twitter bio included the phrase 'lover of open source'. Nowadays it says instead 'passionate about indie software' (as well as stressing that he's no longer in leadership at Hashicorp).

https://web.archive.org/web/20230614095444/https://twitter.c...

Re: Terraform Cloud is down for nearly five hours, and counting

#12

Post by oarmstrong »

erulabs wrote: post #7

I’ve done a lot of good work with Hashicorp tools for many many years. But I’m extremely excited to move off Terraform, Nomad, Consul, Vault and the rest. Actually quite happy with the future: CDK, CDK8s… it’s a bit of a mess currently but at least there’s a strong path forward and tons of innovation.

What are you looking at for a Vault replacement?

Re: Terraform Cloud is down for nearly five hours, and counting

#13

Post by mdaniel »

atonse wrote: post #4
rvz wrote: post #3

If this was Threads, Twitter / X, Instagram, etc the entire news world would be reporting in less than 10 mins if it was down for just 5 mins. It looks like you can get away with it and let your service(s) have downtime for more than 5 hours if little to no-one is paying for your service or there are almost no users using it. Their stock price being 66% down since IPO, also suggests that not only Hashicorp's valuatio…

I regret buying their stock. Their tech seemed good initially so i thought everyone going to need the multi cloud stuff. But after using it and encountering how brittle it was, and easily our nomad cluster would go down without any way to recover, even though the major point of a cluster is high availability, I can see that the tech is just very brittle and the value of the stock doesn’t surprise me. I’m relieved tha…

> our nomad cluster would go down without any way to recover,

for my curiosity, was in Nomad or Consul that fell over? My experience with etcd leads me to suspect it was actually a consul fire, but since I (thankfully) have never run Nomad I don't know first hand about its dragons

Re: Terraform Cloud is down for nearly five hours, and counting

#14

Post by erulabs »

oarmstrong wrote: post #12
erulabs wrote: post #7

I’ve done a lot of good work with Hashicorp tools for many many years. But I’m extremely excited to move off Terraform, Nomad, Consul, Vault and the rest. Actually quite happy with the future: CDK, CDK8s… it’s a bit of a mess currently but at least there’s a strong path forward and tons of innovation.

What are you looking at for a Vault replacement?

Vault will be last on my list to replace. It's boring and it works and it's integration with K8s is actually much better than it's integration with Nomad.

Eventually though, I'd like to move to AWS Secrets Manager.

Re: Terraform Cloud is down for nearly five hours, and counting

#15

Post by oarmstrong »

erulabs wrote: post #14
oarmstrong wrote: post #12

Earlier quoted context omitted.

What are you looking at for a Vault replacement?

Vault will be last on my list to replace. It's boring and it works and it's integration with K8s is actually much better than it's integration with Nomad. Eventually though, I'd like to move to AWS Secrets Manager.

I agree that it’s boring and works well. It’s also what I’m most worried about finding a reasonable alternative, we can’t just not update Vault.

Sadly Secrets Manager doesn’t hit all the features I need. Really hoping for an OpenTF-style fork of Vault.

Re: Terraform Cloud is down for nearly five hours, and counting

#16

Post by atonse »

mdaniel wrote: post #13
atonse wrote: post #4

Earlier quoted context omitted.

I regret buying their stock. Their tech seemed good initially so i thought everyone going to need the multi cloud stuff. But after using it and encountering how brittle it was, and easily our nomad cluster would go down without any way to recover, even though the major point of a cluster is high availability, I can see that the tech is just very brittle and the value of the stock doesn’t surprise me. I’m relieved tha…

> our nomad cluster would go down without any way to recover, for my curiosity, was in Nomad or Consul that fell over? My experience with etcd leads me to suspect it was actually a consul fire, but since I (thankfully) have never run Nomad I don't know first hand about its dragons

I’m trying to remember. I think it was nomad.

Essentially, about once a week the raft pings between the nodes would result in no response from the primary, so then the cluster would assume it lost the node and try to hold an election to pick the new leader and get stuck in a loop cuz it kept indefinitely trying to ask the leader for it’s vote.

I thought, surely this software isn’t that stupid. But it was.

The recovery documentation was to hand-generate a peers.json file. Seriously? In 2013 when you have a million ways to do auto discovery? Including in your own software? it couldn’t just auto heal?

I managed a MongoDB cluster ten years ago and never had a single issue like this. I could routinely take nodes down and bring them up and the cluster healed perfectly.

Post reply on HN