There is no mention of SPARK like in-memory compute engine(s)
Did i miss it in my read ?
thanks in advance
What does it take to make Google work at scale?
81–90 of 134 posts
Re: What does it take to make Google work at scale?
#82Earlier quoted context omitted.
Can you elaborate on the need? What possible reason, short of a very contrived one, is there for having to keep a large number of machines' clocks in-sync? And for that matter, why would anyone build any process/system/software that requires a distributed system's machines to all have their clocks in-sync. I am baffled.
Many security protocols use credentials that are only valid for a brief time. Here is a simple example of a problem caused by my server's time being more than 10 minutes off from the time on S3: http://illuminatedcomputing.com/posts/2015/04/paperclip_expi...
Yep, security is a big one. Authentication & encryption can behave very unpredictably if your times are not in sync. So, you might not be able to login to a server, or connect via HTTPS etc
Re: What does it take to make Google work at scale?
#83Nice overview. I particularly like how they visualized BigTable and MapReduce. Will make it easier for some to understand. For those liking Spanner, Google has exceeded that with their awesome F1 RDBMS (built on Spanner/GFS) below: https://static.googleusercontent.com/media/research.google.c... Can't wait for someone to clone this. Come to think of it, has anyone cloned Spanner? I recall some calling GPS a hack but I…
The people who are working to it are the Cockroach Labs folks: http://www.cockroachlabs.com/
Re: What does it take to make Google work at scale?
#84Kind of ironic: "Wow, this file is really popular! Some tools might be unavailable until the crowd clears"
This is actually a really good example of the CAP theorem. Because of the way Docs merges updates to a single document they had to either tie an editable document to a single machine or risk inconsistent updates. The choice was made to ensure consistency. If a doc gets too popular to be served off a single machine then it goes into read-only mode and gets distributed to multiple boxes. When the traffic dies down it can fall back to a single box and re-enter read/write mode.
Re: What does it take to make Google work at scale?
#85Kind of ironic: "Wow, this file is really popular! Some tools might be unavailable until the crowd clears"
Incidentally, that's part of scalability: knowing when you can drop functionality and the user won't care. For Google Docs, it's highly unlikely that a file with 1000s of viewers is actively being edited by 1000s of users, and more likely the link just got shared on the Internet. So you can drop edit functionality for everyone but the original author and they won't care. Another example - one that I've heard second-h…
That's really smart. Never noticed a thing. Awesome :)
Re: What does it take to make Google work at scale?
#86Earlier quoted context omitted.
"Every review I hear about working at Google makes me want to stay away" I loved working there. Sure, who your boss is has a huge impact on your happiness. I ended up with a boss I enjoyed working for. Does that review make you want to stay away, too?
It does for me, at least. I'd absolutely hate having my job satisfaction determined so much by a manager and team that I was assigned . (Ideally I wouldn't want it impacting much at all , but truly flat companies are still hard to find, despite the rhetoric.) It works for some people, but it won't work for me.
> It does for me, at least. I'd absolutely hate having my job satisfaction determined so much by a manager and team that I was assigned.
Well put. There were a lot of reasons I turned down my Google offer out of college and have been unenthusiastic to re-apply, but the process of saying "We'll find something for you" was a huge part of it. (The recruiter wouldn't even listen to something as simple as wanting SWE over SRE.) I ended up at a much smaller company where I could know my job and product and meet my boss before I signed.
Re: What does it take to make Google work at scale?
#87Earlier quoted context omitted.
It does for me, at least. I'd absolutely hate having my job satisfaction determined so much by a manager and team that I was assigned . (Ideally I wouldn't want it impacting much at all , but truly flat companies are still hard to find, despite the rhetoric.) It works for some people, but it won't work for me.
> It does for me, at least. I'd absolutely hate having my job satisfaction determined so much by a manager and team that I was assigned . Well put. There were a lot of reasons I turned down my Google offer out of college and have been unenthusiastic to re-apply, but the process of saying "We'll find something for you" was a huge part of it. (The recruiter wouldn't even listen to something as simple as wanting SWE ove…
I think the best analogy for Google is that it's like being accepted at a University. And they think it's going to take you a while to declare your major. You may not like your weeder classes. You may not like your first few professors. But pretty soon, you'll find a home for yourself, if you want to make the best of it.
Re: What does it take to make Google work at scale?
#88"NTP isn't accurate enough so we use a hardware clock assist that uses GPS and atomic clocks" - Google I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale. Every review I hear about working at Google makes me want to stay away, just like the recent conversations about working at Amazon, but crazy stuff like this always piques my interest.
> I think it is crazy that something as simple as keeping time can have such a complicated solution at that sort of scale.
Timekeeping is such a basic problem that Albert Einstein invented the theory of relativity because of it.
Re: What does it take to make Google work at scale?
#89Earlier quoted context omitted.
Google Chicago is almost all salespeople. Not much technology going on there.
It's true the sales people do out number us here in Chicago, but we do have O(100) engineers in the office. We are working on lots of cool stuff: Ads, Search, and Privacy to name a couple.
And you guys have a super-cool and mildly famous engineering site lead!
Re: What does it take to make Google work at scale?
#90The thing that might be scary working at Google is that you work with tech than no-one else uses.
The one thing that worries me about leaving Google is how much time I'd have to spend building hacked-together imitations of the infrastructure I'm used to using. It's not because I don't know how other things work, it's because I know how these things work and I like them better.
At least the build system was recently open-sourced, so I won't have to build that from scratch. But things like Borg and D are both elegant and easy to use, and I would hate to have to go back and care about deploying software or configuring RAID arrays again. Unfun and uninteresting. Totally solved at Google for the kinds of problems I work on.