Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Diskless infrastructure in beta (System Transparency: stboot) (2022)

mullvad.net

Post reply on HN
Print view View original

31–39 of 39 posts

Re: Diskless infrastructure in beta (System Transparency: stboot) (2022)

#31

Post by hpcjoe »

kfreds wrote: post #26

It's worth noting that System Transparency is a multi-year effort to bring transparency to running computer systems. We are aiming for what we call transparent servers. Just like there's open source software and open source hardware we think there should be open source running systems. That's the gist of it. If you think this is interesting I can highly recommend you check out Sigsum - our transparency log design for…

I love the concept. I created and ran a PXE/netbooted full OS on ramdisk[1] for my old companies servers for years. We were in the high performance computing and storage space. Stateless machines have so many advantages over stateful.

That said, solving a trusted boot problem was not something I could tackle alone. I didn't have a sense for how much/little I could trust the machine/bios/firmware. None of the tooling I considered (hashing firmware/boot data/etc.) seemed secure without a whole additional infrastructure.

I'm thrilled to see this implemented though.

[1] modern version here: https://github.com/joelandman/nyble

Re: Diskless infrastructure in beta (System Transparency: stboot) (2022)

#32

Post by ignoramous »

> Running the system in RAM does not prevent the possibility of logging. It does however minimise the risk of accidentally storing something that can later be retrieved.

I don't know what the threat model is, but if it involves nation states confiscating servers, then diskless is of limited help: https://en.wikipedia.org/wiki/Cold_boot_attack

> If the computer is powered off, moved or confiscated, there is no data to retrieve.

Oh wait...

Re: Diskless infrastructure in beta (System Transparency: stboot) (2022)

#34

Post by vbezhenar »

crazygringo wrote: post #2

Wow, I had no idea "diskless infrastructure" was even a thing. Easy to imagine in theory, but this is the first time I'm hearing about it in practice, and it makes total sense in this case. It makes me curious if there are any other real-world use cases for diskless. Are there any customers who would benefit from such a configuration from major cloud providers? E.g. a diskless EC2 instance type that ran off of a RAM…

Talos Linux is a Linux designed for Kubernetes which runs from memory. I'm not sure if it could run diskless, because I expect kubernetes workloads to require some local disk.

Re: Diskless infrastructure in beta (System Transparency: stboot) (2022)

#35

Post by yjftsjthsd-h »

vbezhenar wrote: post #34
crazygringo wrote: post #2

Wow, I had no idea "diskless infrastructure" was even a thing. Easy to imagine in theory, but this is the first time I'm hearing about it in practice, and it makes total sense in this case. It makes me curious if there are any other real-world use cases for diskless. Are there any customers who would benefit from such a configuration from major cloud providers? E.g. a diskless EC2 instance type that ran off of a RAM…

Talos Linux is a Linux designed for Kubernetes which runs from memory. I'm not sure if it could run diskless, because I expect kubernetes workloads to require some local disk.

> because I expect kubernetes workloads to require some local disk

For small uses, I would think tmpfs would be fine. And I would personally tend to think that non-small data probably shouldn't be local anyways (because most data should live in a central database and/or on a SAN or ceph or whatever). Obviously it depends on your exact usecase, but usually.

Re: Diskless infrastructure in beta (System Transparency: stboot) (2022)

#36

Post by yjftsjthsd-h »

latchkey wrote: post #24

I created a system that booted 12k+ diskless blades via PXE and running Ubuntu (it was built to scale to 30k+, but we never got there). This generally works well, but I'd say there are about 0-20 blades that crash a day due to some sort of memory corruption issues. Due to the fact that I was operating remotely from the hardware, I never really got a chance to resolve it... also... just a simple reboot would fix it (a…

Is that caused or exacerbated by being diskless, though? Or is it just inevitable that 12k+ machines are going to have a certain rate of memory errors regardless?

Re: Diskless infrastructure in beta (System Transparency: stboot) (2022)

#37

Post by generalizations »

Mave83 wrote: post #14

we at croit.io use PXE boot into RAM for more than 6 years on all our worldwide storage deployments. It provides so many benefits and eases the server management greatly.

> It provides so many benefits and eases the server management greatly.

Can you elaborate on this? I would have thought that needing local storage cache and the potential for network latency would make PXE untenable.

Re: Diskless infrastructure in beta (System Transparency: stboot) (2022)

#38

Post by kjs3 »

crazygringo wrote: post #2

Wow, I had no idea "diskless infrastructure" was even a thing. Easy to imagine in theory, but this is the first time I'm hearing about it in practice, and it makes total sense in this case. It makes me curious if there are any other real-world use cases for diskless. Are there any customers who would benefit from such a configuration from major cloud providers? E.g. a diskless EC2 instance type that ran off of a RAM…

Been a thing for a long time. I first ran into it back in the Novell days with RPL netbooting, and it's moved through BOOTP to PXE booting. Biggest benefits pitched are usually "lower cost" and "centralized management".

As an example, X Terminals all worked this way. Much of the old Sun Microsystems "The Network Is The Computer" pitch was having low-end, diskless SPARC machines (e.g. SLC, ELC) netbooting and mounting disk and doing heavy lift compute via NFS & X11 from large SPARC servers and storage arrays.

You can look at the Linux Terminal Server Project (ltsp.org) for some reasonably current ideas of what someone might do with this.

Re: Diskless infrastructure in beta (System Transparency: stboot) (2022)

#39

Post by latchkey »

yjftsjthsd-h wrote: post #36
latchkey wrote: post #24

I created a system that booted 12k+ diskless blades via PXE and running Ubuntu (it was built to scale to 30k+, but we never got there). This generally works well, but I'd say there are about 0-20 blades that crash a day due to some sort of memory corruption issues. Due to the fact that I was operating remotely from the hardware, I never really got a chance to resolve it... also... just a simple reboot would fix it (a…

Is that caused or exacerbated by being diskless, though? Or is it just inevitable that 12k+ machines are going to have a certain rate of memory errors regardless?

That's the thing, I don't know. It could be a whole bunch of issues, but I thought it was interesting to note what I see at scale while doing this.
Post reply on HN