History of massive-scale sorting experiments at Google
cloud.google.com
History of massive-scale sorting experiments at Google
1–10 of 82 posts
Re: History of massive-scale sorting experiments at Google
#2Only at Google
Re: History of massive-scale sorting experiments at Google
#3Only at Google
I wonder how fast this could do it:
http://www.lanl.gov/projects/trinity/specifications.php
They're claiming 87.0 TB/min on an 80PB filesystem, relative to Google's 36.2 TB/min.
Re: History of massive-scale sorting experiments at Google
#4 Note that this sort [in 2012] was 500 times larger than the
GraySort large-scale requirement and twice as fast in
throughput as the **current 2015** official GraySort winner.
Re: History of massive-scale sorting experiments at Google
#5Only at Google
I wonder how fast this could do it: http://www.lanl.gov/projects/trinity/specifications.php They're claiming 87.0 TB/min on an 80PB filesystem, relative to Google's 36.2 TB/min.
That just their I/O bandwidth. There is computing overhead in actually doing the sort. Also google was using redundant persistence.
Re: History of massive-scale sorting experiments at Google
#6"To reduce the impact of stragglers, we used a dynamic sharding technique called reduce subsharding. This is the precursor to fully dynamic sharding used in Dataflow."
Fun ways in which internal Google tech makes its way to Google Cloud services.
Re: History of massive-scale sorting experiments at Google
#7I'm sure they're asking about this in interviews now. My experience with Google interviews was that the interviewers were very keen on proving that they knew more theoretical CS than I did vs talking about what the actual work would require or entail.
Re: History of massive-scale sorting experiments at Google
#8>> Nobody really wants a huge globally sorted output. We haven’t found a single use case for the problem as stated.
Do anyone has a real world use case of global sorting other than top-k?
Re: History of massive-scale sorting experiments at Google
#9 We haven’t found a single use case for the problem as stated.
Re: History of massive-scale sorting experiments at Google
#10I'm sure they're asking about this in interviews now. My experience with Google interviews was that the interviewers were very keen on proving that they knew more theoretical CS than I did vs talking about what the actual work would require or entail.
You may well be asked how a distributed sort would work in principle, and be asked to code a small part of it. Any reasonable answer that shows you can think on your feet would be evidence in favor of hiring.
It sounds like you didn't get hired, and I'm sorry about that. Interviewers' preferences are of course diverse, but for at least the last 5 years, probably 10, Google interviews favor practical solutions to problems we face every day. Some of those are theoretical CS, many are not.