Live data from Hacker News

Containers and Distributed Systems: Where They Came from and Where They’re Going

mesosphere.com

21–30 of 44 posts

Re: Containers and Distributed Systems: Where They Came from and Where They’re Going

#21
post #19

Earlier quoted context omitted.

>"back in the early 2000's when "clusters" and "NUMA SMP machines" were competing with each other" Did you mean Numa vs SMP? Or something else maybe? How can a machine be a NUMA SMP? NUMA and SMP are fundamentally different architectures.

Is this clear cut in the terminology? A plausible definition would also be that it's still symmetric MP if remote memory access has non-uniform performance - since the nodes and their memories are symmetrical. After all, you get that just with caches and 2 sockets plugged to the same DRAM. The historical opposite of SMP used to be asymmetric multiprocessors in the heterogenous sense - different kinds of processors, f…

>"Is this clear cut in the terminology?"

I think in the context that OP was referring to "early 2000s" which presumably means the introduction of Opteron and HyperTransport then yes I believe the NUMA vs SMP distinction would be pretty clear.

Re: Containers and Distributed Systems: Where They Came from and Where They’re Going

#22
post #20

"You basically came up with Docker before Docker was around, or at least with things like Solaris zones and C groups." Except jails already existed on FreeBSD, and CP/CMS on mainfraimes in the 70s existed long before this...

Docker won the UX game, which mattered the most here. :)

Re: Containers and Distributed Systems: Where They Came from and Where They’re Going

#23
post #22
post #20

"You basically came up with Docker before Docker was around, or at least with things like Solaris zones and C groups." Except jails already existed on FreeBSD, and CP/CMS on mainfraimes in the 70s existed long before this...

Docker won the UX game, which mattered the most here. :)

It only mattered the most because it was thought of last. If it was UX that increased impact by improving access, then it indeed mattered a great deal.

Re: Containers and Distributed Systems: Where They Came from and Where They’re Going

#24
post #20

"You basically came up with Docker before Docker was around, or at least with things like Solaris zones and C groups." Except jails already existed on FreeBSD, and CP/CMS on mainfraimes in the 70s existed long before this...

Minus CP/CMS and OpenVZ, here's an article on the concepts and tech Docker is built on. https://www.kentik.com/the-evolution-of-docker/

Re: Containers and Distributed Systems: Where They Came from and Where They’re Going

#26
post #2

This was a lot of fun, one of the things that doesn't get much air time is that back in the early 2000's when "clusters" and "NUMA SMP machines" were competing with each other the big argument for large SMP iron was ACID compliant SQL databases like Oracle. Now that Google has implemented an ACID compliant SQL database across clusters it puts the final nail in the argument (for me at least) that "Some things only wor…

>"back in the early 2000's when "clusters" and "NUMA SMP machines" were competing with each other" Did you mean Numa vs SMP? Or something else maybe? How can a machine be a NUMA SMP? NUMA and SMP are fundamentally different architectures.

No, I was thinking non-uniform memory architecture, which is to say an SMP machine where the "speed" at which you can access RAM is dependent on the core or 'thread' from which you accessed it. Lack of memory uniformity was the compromise to achieve larger effective address spaces and "simple" programming.

Today on a 24/48 core dual socket server you'll see the same sorts of thing, having a core using memory on the 'other' physical chip's memory bus will impact the overall performance significantly.

In the 2000's, before multi-core chips were a thing, there were two major camps, the 'super computer' camp, and the 'cluster' camp.

The 'super computer' camp insisted on cache coherent memory between all of the cores or threads. You got these very expensive fabrics from people like cray that would snoop access to memory from the cores and send coherency messages around to insure that if someone wrote something in to memory somewhere, everyone else's L1 or L2 cache got the message to invalidate what they were holding (shoot downs). These machines are very expensive and take months to build.

The 'cluster' camp said, "We can use a network fabric and just parameterize shared memory usage." So they put together independent machines connected by a network fabric and no cache coherency protocol. If you wanted to use shared memory you could build something like memcached and wrap your access with network calls. With that architecture even if it took twice as many cores to do what you wanted to do, the price of the machine was one tenth what it was for the big SMP machine.

For something that was trivially parallellizable like internet search or serving up web sites, that was a much more cost effective way to go. When people started doing stuff that they previously used 'super computers' and big SMP machines for on these Linux clusters it became a sort of race to pull apart these problems into "shared nothing" clusters.

Re: Containers and Distributed Systems: Where They Came from and Where They’re Going

#27
post #14
post #6

Earlier quoted context omitted.

Which has still horrible join performance, and several other tradeoffs. (see https://www.cockroachlabs.com/blog/cockroachdbs-first-join/ ) Maybe in a year, or two. But not today.

Spanner has a lot of tradeoffs too, I'm not sure what do you see as problematic in Cockroach, joins are good enough. The biggest tradeoffs are inherent to strongly consistent distributed systems. Even precise clocks and fast networks won't help as much as you might think. You still have to accept vastly different latencies and performance than in traditional single-node RDBMSs.

If I run several servers in the same rack, with a local private 10Gbps network, with CockroachDB, running on spinning HDDs, I expect to get the same (or better) throughput as with a single PGSQL instance, and a similar latency. (When accessing it from another server, via the public internet).

That’s not always the case, though.

Re: Containers and Distributed Systems: Where They Came from and Where They’re Going

#28
post #4
post #2

This was a lot of fun, one of the things that doesn't get much air time is that back in the early 2000's when "clusters" and "NUMA SMP machines" were competing with each other the big argument for large SMP iron was ACID compliant SQL databases like Oracle. Now that Google has implemented an ACID compliant SQL database across clusters it puts the final nail in the argument (for me at least) that "Some things only wor…

I actually thing "NUMA SMP machines" are about due for a renaissance with the advent of extreme multicore CPUs. Next year we will see CPUs that run 32 cores (64 hyperthreads) per socket, and don't cost an arm and a leg. Current max is 28 cores per socket, at a staggering $13K per chip. With relatively economical 64 core / 128 thread options with nearly unlimited RAM capacity appearing, a lot more workloads will "fit"…

The irony of clusters of many core NUMA SMP machines is not lost on me :-)

If things follow the previous patterns that will open up the market to a single/dual core, large memory, machine with a high speed networking and storage ports that is lower cost and lower power.

Re: Containers and Distributed Systems: Where They Came from and Where They’re Going

#29
post #20

"You basically came up with Docker before Docker was around, or at least with things like Solaris zones and C groups." Except jails already existed on FreeBSD, and CP/CMS on mainfraimes in the 70s existed long before this...

The article says " It turned out that about twenty years earlier IBM did sort of solve that problem with partition isolation, but they did that on custom mainframe hardware and architecture."

Re: Containers and Distributed Systems: Where They Came from and Where They’re Going

#30
post #2

This was a lot of fun, one of the things that doesn't get much air time is that back in the early 2000's when "clusters" and "NUMA SMP machines" were competing with each other the big argument for large SMP iron was ACID compliant SQL databases like Oracle. Now that Google has implemented an ACID compliant SQL database across clusters it puts the final nail in the argument (for me at least) that "Some things only wor…

Some things do only work on SMP, because for some workloads no network fabric (that I know of) is as fast as the system bus. It's the classic Beowulf vs Mainframe problem: some things can be split up into discrete parts, and some things have to share so much that bandwidth is the bottleneck. Then you go from SMP to NUMA because your bus is too slow. But both are still faster than most (all?) networks. A lot of this a…

It is exactly the Beowulf vs Mainframe problem. And it is this: "Some things do only work on SMP, because for some workloads no network fabric (that I know of) is as fast as the system bus." is under siege.

From a systems architecture point of view it is a really interesting exploration of Amdahl's law. So many things that people said "You'll never do that on a networked cluster of machines." have fallen (data bases being one of the larger ones). And while it used to be mainframes won on I/O channel capacity, Google and others have shown that when you can parallelize the I/O channels effectively that advantage goes away as well.

Post reply on HN