Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Who operates at scale without containers?

news.ycombinator.com

Post reply on HN
Print view View original

341–350 of 446 posts

Re: Ask HN: Who operates at scale without containers?

#341

Post by smilliken »

kazinator wrote: post #312
smilliken wrote: post #225

Earlier quoted context omitted.

Yeah, the environment is bit-for-bit identical in dev and prod. Any difference is an opportunity for bugs. OK, there's one concession, there's an env var that indicates if it's a dev and prod environment. We try to use it sparingly. Useful for stuff like not reporting exceptions that originate in a dev environment. Basically, there's a default.nix file in the repo, and you run nix-shell and it builds and launches you…

Does that mean you turn off security-related randomizations in everything, like address space randomization and hash table randomization?

No, we have address space randomization and hash table randomization since those happen at runtime. /dev/random works as you'd expect.

The immutability is just at build time. So chrome and firefox aren't able to seed a unique ID in the binaries like you might be accustomed to. Funny story, we had a python dependency that would try to update itself when you imported it. I noticed because it would raise an exception when it was on a read only mount.

Re: Ask HN: Who operates at scale without containers?

#342

Post by smilliken »

Gwypaas wrote: post #240
smilliken wrote: post #225

Earlier quoted context omitted.

Yeah, the environment is bit-for-bit identical in dev and prod. Any difference is an opportunity for bugs. OK, there's one concession, there's an env var that indicates if it's a dev and prod environment. We try to use it sparingly. Useful for stuff like not reporting exceptions that originate in a dev environment. Basically, there's a default.nix file in the repo, and you run nix-shell and it builds and launches you…

How do you manage quick iteration loops?

We use python. If we were writing in a compiled language, we'd use the same compiler toolchain as everyone else, but with the versions of all of our dependencies exactly the same from nix. We have some c extensions and compile Typescript and deploy those build artifacts. In the case of javascript, our node modules is built by nix, and our own code is built by webpack --watch in development.

Re: Ask HN: Who operates at scale without containers?

#343

Post by smilliken »

john-shaffer wrote: post #279
smilliken wrote: post #81

Earlier quoted context omitted.

In my case, adopting Nix was a response to having a poor onboarding process for new engineers. It was always fully automated, but it wasn't reliable before nix. So somebody would join the team, and it was embarrassing because the first day would be them troubleshooting a complex build process to get it to work on their machine. Not a great first impression. So I adopted Nix "in anger" and now new machines always buil…

Have you had problems with Nix on macOS? Nix works great for me, but I can't use it much to deal with installs or synchronize dependencies because the devs on macOS can't get Nix running.

A few people I've worked with have used our nix build successfully on MacOS, but I stick to Linux. They've told me it works fine after making some dependencies conditional. I would've expected it to be death by a million papercuts, so I'm delighted it works and a VM isn't needed.

Re: Ask HN: Who operates at scale without containers?

#344

Post by smilliken »

abecedarius wrote: post #287
claytonjy wrote: post #47

Earlier quoted context omitted.

is Nix a hurdle when onboarding engineers? do only a few people need to know the language? I've wanted to use Nix and home-manager for personal stuff but the learning curve seems big.

Just my experience with Nix, two or three years back: I went to the Nix website for the install instructions, saw something like "curl | sh", said "what? no way" and used the alternative non-recommended out-of-date instructions, which turned into a full-day rabbit hole until it would build. A year later I went through this again for another machine, and this time just gave up. So I would say curl|sh is probably the w…

I grabbed a copy of that shell script, reviewed it, and committed it to git. It's extra nice to save a specific cooy of it so you get the same version of the nix cli tools across the team. It works with different versions just fine, but still better to standardize.

Re: Ask HN: Who operates at scale without containers?

#345

Post by smilliken »

geoduck14 wrote: post #291
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

Well, I was about to ask if you were looking for a job - but then I saw you were the founder and CTO. So... are you hiring?

Yes, we're hiring! :)

Reach out to dev@mixrank.com. Hiring globally for pretty much all roles, including junior roles.

Re: Ask HN: Who operates at scale without containers?

#346

Post by mrslave »

john-shaffer wrote: post #279
smilliken wrote: post #81

Earlier quoted context omitted.

In my case, adopting Nix was a response to having a poor onboarding process for new engineers. It was always fully automated, but it wasn't reliable before nix. So somebody would join the team, and it was embarrassing because the first day would be them troubleshooting a complex build process to get it to work on their machine. Not a great first impression. So I adopted Nix "in anger" and now new machines always buil…

Have you had problems with Nix on macOS? Nix works great for me, but I can't use it much to deal with installs or synchronize dependencies because the devs on macOS can't get Nix running.

We have devs on MacOS with Nix working, but we do encounter inconsistencies. Production & other devs are on Linux and they are having a much better time.

Re: Ask HN: Who operates at scale without containers?

#347

Post by smilliken »

mamcx wrote: post #180
smilliken wrote: post #35

My company runs without containers. We process petabytes of data monthly, thousands of CPU cores, hundreds of different types of data pipelines running continously, etc etc. Definitely a distributed system with lots of applications and databases. We use Nix for reproducible builds and deployments. Containers only give reproducible deployments, not builds, so they would be a step down. The reason that's important is t…

Any hints in how use nix with DBs? I have a semi-docker setup where I stay with the base OS + PostgreSQL and the rest is docker. But that means that upgrade the OS/DB is not that simple. I tried before dockerize PG, but still the OS need management anyway and was harder to use diagnostics/psql/backups with docker....

Yeah, we built a tool for managing postgresql databases with nix. It's called schematic: https://gitlab.com/deltaex/schematic

We've been using it in prod for a couple years. There's a couple dozen production deployments outside the company as well. It's open source, MIT licensed. It doesn't have documentation yet, so it's currently only for people that don't mind reading the source or have talked to us directly about it.

It uses Nix because postgresql has C extensions, which can depend on anything in the software universe. Schema depends on extensions, so it's not technically possible to separate schema migrations from Nix without duct tape abd glue. So schematic is a sort of "distribution" of PostgreSQL that has a package manager (for extensions, schema, content, etc), and manages revisions.

If this is interesting to others here, I can do a "Show HN" post after getting the docs in order.

Re: Ask HN: Who operates at scale without containers?

#349

Post by kronin »

spion wrote: post #316
treis wrote: post #247

Earlier quoted context omitted.

>Getting repeatable docker images But there's no real need for repeatable build docker images. You copy the image and run it where ever you need to. The entire point of Docker is to not have to repeat the build.

Until there is a critical vulnerability in one of the components present on that image (system packages or application packages).

This is why I haven't adopted the practice of "Build your artifact along with the docker image that packages it".

Instead, build your artifact and publish it to an artifact repository, just like we used to.

_then_ wrap that artifact in a Docker image.

Vulnerability found in the docker image? No problem. Build a new image with the same artifact.

Re: Ask HN: Who operates at scale without containers?

#350

Post by mamcx »

smilliken wrote: post #347
mamcx wrote: post #180

Earlier quoted context omitted.

Any hints in how use nix with DBs? I have a semi-docker setup where I stay with the base OS + PostgreSQL and the rest is docker. But that means that upgrade the OS/DB is not that simple. I tried before dockerize PG, but still the OS need management anyway and was harder to use diagnostics/psql/backups with docker....

Yeah, we built a tool for managing postgresql databases with nix. It's called schematic: https://gitlab.com/deltaex/schematic We've been using it in prod for a couple years. There's a couple dozen production deployments outside the company as well. It's open source, MIT licensed. It doesn't have documentation yet, so it's currently only for people that don't mind reading the source or have talked to us directly about…

Yeah, managing the DBs is the major complications on doing this. A little discrepancy in binaries are not that always problematic, but a fail on the DB is catastrophic :)
Post reply on HN