21–30 of 94 posts
Perhaps the biggest "fake-out" in 21st century computing: Google publicly released its MapReduce paper -- directing most of the rest of the industry toward loosely coupled, overly complex distributed data processing systems like Hadoop for the following decade -- but internally they just bought a bunch of atomic clocks and built a distributed RDBMS. I know this is a somewhat simplified story, but it does make me chuc…
Earlier quoted context omitted.
Yes, which is why it's amusing in hindsight that for a decade everyone* outside Google was forcing all* their distributed data tasks into the MapReduce paradigm, without considering alternative approaches like the one used by Spanner. * slight exaggerations, I know
Spanner didn’t exist in 2012.
https://cloud.google.com/files/storage_architecture_and_chal...
Perhaps the biggest "fake-out" in 21st century computing: Google publicly released its MapReduce paper -- directing most of the rest of the industry toward loosely coupled, overly complex distributed data processing systems like Hadoop for the following decade -- but internally they just bought a bunch of atomic clocks and built a distributed RDBMS. I know this is a somewhat simplified story, but it does make me chuc…
However, no one is looking at Map Reduce type jobs as a replacement for a database and vice versa. That's like saying "wow linkedin made kafka why do we need a webserver too". Those two technologies are only related in the loosest sense.
Earlier quoted context omitted.
Yes, which is why it's amusing in hindsight that for a decade everyone* outside Google was forcing all* their distributed data tasks into the MapReduce paradigm, without considering alternative approaches like the one used by Spanner. * slight exaggerations, I know
Spanner didn’t exist in 2012.
Perhaps the biggest "fake-out" in 21st century computing: Google publicly released its MapReduce paper -- directing most of the rest of the industry toward loosely coupled, overly complex distributed data processing systems like Hadoop for the following decade -- but internally they just bought a bunch of atomic clocks and built a distributed RDBMS. I know this is a somewhat simplified story, but it does make me chuc…
MapReduce predates TrueTime by a decade or more. MR was critical to scaling internet systems at the time it was released.
However, Flume + Spanner was a much nicer system to work with than MR + GFS, I'll give you that.
I'm curious about the title on this submission. I thought TrueTime had unusually strong guarantees about accuracy that don't seem to be called out in what I'm reading on the linked article.
Sorry if it's a bit sensationalized. I wanted to give some color as to why this release might be interesting and took this tweet I saw at face value https://twitter.com/rbranson/status/1455923426359578631
"Please use the original title, unless it is misleading or linkbait; don't editorialize."
Perhaps the biggest "fake-out" in 21st century computing: Google publicly released its MapReduce paper -- directing most of the rest of the industry toward loosely coupled, overly complex distributed data processing systems like Hadoop for the following decade -- but internally they just bought a bunch of atomic clocks and built a distributed RDBMS. I know this is a somewhat simplified story, but it does make me chuc…
Google uses map reduce extensively... where it's appropriate. True time helps with things like spanner transactions. It's just a totally different use case.
MapReduce was used at Google for highly inappropriate things. For example, the machine learning system I worked on, Sibyl https://www.datanami.com/2014/07/17/inside-sibyl-googles-mas... was implemented using mapreduce but there was no real technical justification for that- it's just that there was no other system that could scale to the volumes required or handle the constant failures endemic to GOogle's internal systems. It ended up requiring all sorts of heroic work to make MR scale, for example map-side combiners (which "reduced" items with common keys in the map output before it gets flushed to the shuffle files). All of this got replaced with TensorFlow and only the good bits of Sibyl were extracted to TFX.
Earlier quoted context omitted.
Google uses map reduce extensively... where it's appropriate. True time helps with things like spanner transactions. It's just a totally different use case.
The tech lead of the Google MapReduce team (which no longer exists) just received their award for turning down mapreduce. IIRC it was officially done 5 years ago. However I believe the code to delete MR was never checked in and I'm not sure if there are still users. MapReduce was used at Google for highly inappropriate things. For example, the machine learning system I worked on, Sibyl https://www.datanami.com/2014/0…
Earlier quoted context omitted.
Spanner didn’t exist in 2012.
Yes it did. Google published a paper about it in 2012, and claimed that at that point it had been in development for 5 years and in production for more than 1.
I’ve never heard of Spanner internally. Maybe it was in development, but it was not in use.
Edit: went and read more. Looks like Spanner existed but didn’t have sql, so it wasn’t what it is today. And looks like I don’t remember things any more.