Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Bottlerocket, an open source Linux distribution built to run containers

aws.amazon.com

Post reply on HN
Print view View original

51–60 of 135 posts

Re: Bottlerocket, an open source Linux distribution built to run containers

#51

Post by NathanKP »

GauntletWizard wrote: post #48
NathanKP wrote: post #39

Earlier quoted context omitted.

I can provide a little background on this. In general yes I would recommend that you just use an ASG and roll out a new AMI. However that approach can be very expensive and time-consuming at truly massive scale (1000's or even 10's of thousands of machines). Bottlerocket is built in part based on our experiences operating AWS Fargate, which obviously has as one of its needs the ability to patch a colossal number of h…

It seems to me, not to be combative, that if Fargate can't afford the "noschedule: node is old" overhead and customers of Fargate can't handle their containers restarting on a regular basis, there's something wrong with your management engine or with their design and implementation. Much of the point of containerization is that you can roll containers often and run enough of them that you never have a single point of…

There are any number of reasons to avoid restarting things. Some customers are running code that has a cold start and needs some time to warm up its cache if it restarts. Some customers are running jobs (video rendering, machine learning training, etc) that might take literally days to complete. Interrupting these jobs and causing them to restart wastes the customer time and causes them to lose progress. Other containers may be hosting multiplayer game servers, and forcing them to restart would cause all people logged into the game instance to get disconnected or otherwise dropped from their game.

All of the above are use-cases that AWS Fargate is used for. Beyond this many folks simply don't like it when things happen unexpectedly outside of their control. We have Fargate Spot for workloads that can tolerate interruption, and we discount the price if you choose this launch strategy. However Fargate on-demand seeks to avoid interrupting your containers. You are in control of when your containers start and stop or autoscale.

Re: Bottlerocket, an open source Linux distribution built to run containers

#52

Post by paxys »

dpryden wrote: post #33

I'm confused about how the documentation recommends using a Kubernetes operator to manage OS updates. That seems weird and backwards to me. I would rather see an immutable OS AMI in an auto-scaled group, and just replace the node instance whenever there is an update. I can see a place for managing OS updates on an instance, but that seems more like "pets" than "cattle"... and I've always treated Kubernetes nodes like…

Constantly rotating nodes in and out of the cluster and restarting/relocating pods, even if mostly automated, causes a lot of needless infrastructure strain. It is IMO one of the most overlooked parts of Kubernetes, and I wish there was a better solution to maintain stable, long-running processes when needed.

Re: Bottlerocket, an open source Linux distribution built to run containers

#53

Post by john-shaffer »

GauntletWizard wrote: post #48
NathanKP wrote: post #39

Earlier quoted context omitted.

I can provide a little background on this. In general yes I would recommend that you just use an ASG and roll out a new AMI. However that approach can be very expensive and time-consuming at truly massive scale (1000's or even 10's of thousands of machines). Bottlerocket is built in part based on our experiences operating AWS Fargate, which obviously has as one of its needs the ability to patch a colossal number of h…

It seems to me, not to be combative, that if Fargate can't afford the "noschedule: node is old" overhead and customers of Fargate can't handle their containers restarting on a regular basis, there's something wrong with your management engine or with their design and implementation. Much of the point of containerization is that you can roll containers often and run enough of them that you never have a single point of…

Nothing is "broken" about it. It's just that when you have tens of thousands of machines that might need an urgent security update, it's very inefficient and costly to destroy all of them at once instead of patching. Destroying machines regularly is not the same thing as frequently destroying all of them at once.

Re: Bottlerocket, an open source Linux distribution built to run containers

#54

Post by jjtheblunt »

dpryden wrote: post #33

I'm confused about how the documentation recommends using a Kubernetes operator to manage OS updates. That seems weird and backwards to me. I would rather see an immutable OS AMI in an auto-scaled group, and just replace the node instance whenever there is an update. I can see a place for managing OS updates on an instance, but that seems more like "pets" than "cattle"... and I've always treated Kubernetes nodes like…

your description made me think of...

https://www.merriam-webster.com/dictionary/fungible

Re: Bottlerocket, an open source Linux distribution built to run containers

#55

Post by mchusma »

NathanKP wrote: post #12
solatic wrote: post #3

As strong as the engineering behind Bottlerocket seems to be, I'm not entirely sure who they built it for, except as a foundational component for AWS's managed offerings. If you, as an AWS customer, decide to fully embrace AWS lock-in, then why would you run this yourself on an EC2 instance instead of running ECS or EKS? If you're trying to avoid AWS lock-in, why would you choose an OS that's locking you into AWS Sys…

Hi I'm a developer advocate in the container engineering org at AWS. I think there are a few misunderstandings here that I may be able to explain better. First Bottlerocket is not Amazon Linux 2, it is its own minimal operating system, with most components built from the ground up in Rust. This is totally different than the Amazon Linux 2 you may be familiar with (and most other operating systems for that matter). Bo…

I tried to see any performance characteristics of running bottlerocket, just to understand what the expectations are there. I assume since it was not mentioned it is either similar to Amazon Linux 2 or worse (but with security advantages). Can you request a follow on post on the aws blog that describes the performance impact of Bottlerocket? If performance is better, would be nice to know that as well of course.

Re: Bottlerocket, an open source Linux distribution built to run containers

#56

Post by [deleted] »

solatic wrote: post #3

As strong as the engineering behind Bottlerocket seems to be, I'm not entirely sure who they built it for, except as a foundational component for AWS's managed offerings. If you, as an AWS customer, decide to fully embrace AWS lock-in, then why would you run this yourself on an EC2 instance instead of running ECS or EKS? If you're trying to avoid AWS lock-in, why would you choose an OS that's locking you into AWS Sys…

[deleted]

Re: Bottlerocket, an open source Linux distribution built to run containers

#57

Post by nuclearnice1 »

NathanKP wrote: post #51
GauntletWizard wrote: post #48

Earlier quoted context omitted.

It seems to me, not to be combative, that if Fargate can't afford the "noschedule: node is old" overhead and customers of Fargate can't handle their containers restarting on a regular basis, there's something wrong with your management engine or with their design and implementation. Much of the point of containerization is that you can roll containers often and run enough of them that you never have a single point of…

There are any number of reasons to avoid restarting things. Some customers are running code that has a cold start and needs some time to warm up its cache if it restarts. Some customers are running jobs (video rendering, machine learning training, etc) that might take literally days to complete. Interrupting these jobs and causing them to restart wastes the customer time and causes them to lose progress. Other contai…

Forgive me my hijack

Any plans to reduce the minimum bill time for Fargate to accommodate short tasks?

With 1 minute minimum billing you have to turn to lambda for very short tasks or have a long running Fargate consuming tasks from some message bus.

If you choose lambda, your containers don’t work so you need to rebuild your runtime with lambda layers or ebs or squeeze into the lambda env.

If you choose messaging, say SQS from a lambda called by API gateway you’ve complicated your architecture and your Fargate instance is potentially hanging out billing, idle, and waiting for messages.

Fargate spot removed the last reason to consider AWS Batch. Short tasks could largely replace lambda.

It would be nice to Fargate all the things.

Re: Bottlerocket, an open source Linux distribution built to run containers

#58

Post by dharmab »

paxys wrote: post #52
dpryden wrote: post #33

I'm confused about how the documentation recommends using a Kubernetes operator to manage OS updates. That seems weird and backwards to me. I would rather see an immutable OS AMI in an auto-scaled group, and just replace the node instance whenever there is an update. I can see a place for managing OS updates on an instance, but that seems more like "pets" than "cattle"... and I've always treated Kubernetes nodes like…

Constantly rotating nodes in and out of the cluster and restarting/relocating pods, even if mostly automated, causes a lot of needless infrastructure strain. It is IMO one of the most overlooked parts of Kubernetes, and I wish there was a better solution to maintain stable, long-running processes when needed.

If your pods need to be long running, you can annotate them as such and they will not be autoscaled.

https://github.com/kubernetes/autoscaler/blob/master/cluster...

Re: Bottlerocket, an open source Linux distribution built to run containers

#60

Post by IMTDb »

NathanKP wrote: post #51
GauntletWizard wrote: post #48

Earlier quoted context omitted.

It seems to me, not to be combative, that if Fargate can't afford the "noschedule: node is old" overhead and customers of Fargate can't handle their containers restarting on a regular basis, there's something wrong with your management engine or with their design and implementation. Much of the point of containerization is that you can roll containers often and run enough of them that you never have a single point of…

There are any number of reasons to avoid restarting things. Some customers are running code that has a cold start and needs some time to warm up its cache if it restarts. Some customers are running jobs (video rendering, machine learning training, etc) that might take literally days to complete. Interrupting these jobs and causing them to restart wastes the customer time and causes them to lose progress. Other contai…

This stuff is probably waaaay over my head, but isn't that why SIGTERM was made for ? To notify a running process that the host needs to be shutdown/restarted and to let the running process finish it's current task (current frame encoding / current multiplayer game / current request / ...) and that the state / cache / progress / ... needs to be saved.

The process on aws side would then be : send SIGTERM to all workloads. wait for [configurable] amount of time (maxed at xx hours) or until all workloads have exited (whichever comes first). Shutdown the node. Update the node. Start the node. Restart the workloads.

Post reply on HN