I'm sure they're asking about this in interviews now. My experience with Google interviews was that the interviewers were very keen on proving that they knew more theoretical CS than I did vs talking about what the actual work would require or entail.
You may well be asked how a distributed sort would work in principle, and be asked to code a small part of it. Any reasonable answer that shows you can think on your feet would be evidence in favor of hiring. It sounds like you didn't get hired, and I'm sorry about that. Interviewers' preferences are of course diverse, but for at least the last 5 years, probably 10, Google interviews favor practical solutions to prob…
History of massive-scale sorting experiments at Google
11–20 of 82 posts
Re: History of massive-scale sorting experiments at Google
#12"To reduce the impact of stragglers, we used a dynamic sharding technique called reduce subsharding. This is the precursor to fully dynamic sharding used in Dataflow." Fun ways in which internal Google tech makes its way to Google Cloud services.
Re: History of massive-scale sorting experiments at Google
#13Earlier quoted context omitted.
I wonder how fast this could do it: http://www.lanl.gov/projects/trinity/specifications.php They're claiming 87.0 TB/min on an 80PB filesystem, relative to Google's 36.2 TB/min.
That just their I/O bandwidth. There is computing overhead in actually doing the sort. Also google was using redundant persistence.
Re: History of massive-scale sorting experiments at Google
#14I'm sure they're asking about this in interviews now. My experience with Google interviews was that the interviewers were very keen on proving that they knew more theoretical CS than I did vs talking about what the actual work would require or entail.
You may well be asked how a distributed sort would work in principle, and be asked to code a small part of it. Any reasonable answer that shows you can think on your feet would be evidence in favor of hiring. It sounds like you didn't get hired, and I'm sorry about that. Interviewers' preferences are of course diverse, but for at least the last 5 years, probably 10, Google interviews favor practical solutions to prob…
I mention this instance largely because my friends at the big G tell me such a question is generally frowned upon, and not part of standard practice, (So I don't feel bad speaking openly about it) but more importantly it gives me a way to see how people can get a skewed perspective on a company given a... unique interview question when the overall guidance within the corp wouldn't align to asking that.
Re: History of massive-scale sorting experiments at Google
#15Earlier quoted context omitted.
You may well be asked how a distributed sort would work in principle, and be asked to code a small part of it. Any reasonable answer that shows you can think on your feet would be evidence in favor of hiring. It sounds like you didn't get hired, and I'm sorry about that. Interviewers' preferences are of course diverse, but for at least the last 5 years, probably 10, Google interviews favor practical solutions to prob…
Lol normal Google arrogance. Yea they only seem to ask theoretical questions basically out of that famous book
Re: History of massive-scale sorting experiments at Google
#16We haven’t found a single use case for the problem as stated. Impressive nonetheless.
Re: History of massive-scale sorting experiments at Google
#17Note that this sort [in 2012] was 500 times larger than the GraySort large-scale requirement and twice as fast in throughput as the **current 2015** official GraySort winner. (emphasis mine)
Re: History of massive-scale sorting experiments at Google
#18Earlier quoted context omitted.
You may well be asked how a distributed sort would work in principle, and be asked to code a small part of it. Any reasonable answer that shows you can think on your feet would be evidence in favor of hiring. It sounds like you didn't get hired, and I'm sorry about that. Interviewers' preferences are of course diverse, but for at least the last 5 years, probably 10, Google interviews favor practical solutions to prob…
Lol normal Google arrogance. Yea they only seem to ask theoretical questions basically out of that famous book
Re: History of massive-scale sorting experiments at Google
#19I'm sure they're asking about this in interviews now. My experience with Google interviews was that the interviewers were very keen on proving that they knew more theoretical CS than I did vs talking about what the actual work would require or entail.
Also, part of the point of a Google style interview is to keep pushing you to the point where your ability fails, then push in another direction to the point your ability fails. That way, they get an idea of the dimensions of your skillset. The vast majority of the people I recommended hiring didn't answer the questions perfectly.
Or, someone could not quite put enough thought into the integer encoding for Protocol Buffers. The encoding actually used is that the first bit of every byte is a flag for if there's a next byte, for a maximum of 10 bytes. The only advantage to this over encoding the length of the integer as the number of leading ones in the first byte (like UTF-8) is if they want the option to have the option later of a forward-compatible encoding for longer integer types, without just supporting multi-precision integers as byte arrays. If you slide all of those flag bits to the first byte, you don't have any more or any fewer flag bits, so it takes as much space, but you can use a jump table and the ability to use native 8-byte and 4-byte simple loads instead of many more masking operations and conditional branches. Giving up on anything longer than int64_t means that the maximum encoding length becomes 9 bytes instead of 10.
One day, the speed of the indexing system just dropped in half. It turns out that someone wrote a job that used machine learning to generate a bunch of regexes for some signal. They were recompiling the regexes for every page in the index. Google really needs most of their engineers that will naturally spot this kind of thing in code reviews, because if every engineer on the indexing team made a mistake like that once a year, the indexing system would always run at half speed, sometimes even 1/3 or 1/4 speed.
The indexing system uses as much electricity as a small town. The tiniest of improvements can mean thousands of dollars per year in savings.
Re: History of massive-scale sorting experiments at Google
#20We haven’t found a single use case for the problem as stated. Impressive nonetheless.