Why is every measure of storage always "demystified" with a paper stack analogy? I find it funny that when the paper stack would not provide the right scale, the used a stack of storage devices...
Our new search index: Caffeine
31–40 of 82 posts
Re: Our new search index: Caffeine
#32huh? 40 miles of iPods. +5 for an intuitive analogy. Priceless!
hopefully iPod-miles (iPm?) will become a new ISO standard for measuring storage capacity.
Re: Our new search index: Caffeine
#33Earlier quoted context omitted.
You think Google's mainline storage of their index is on disk ? It is to laugh. I don't think that's been true since around the time they stopped building server shelves out of lego or cardboard. They've been keeping the full text of the web in RAM (for the snippets), with indexes, several times over. With independent live siblings in multiple datacenters.
Well, the serving part of their index is in memory (or largely so) as has been publicly announced. But are you really suggesting they might do the same for indexing? Let's suppose for a second that they can get 8GB sticks of RAM for 1/3 the retail price of around $600, and for the sake of the back of the envelope calculation, let's round it up to ten GB. So let's call it $20/GB of RAM. Four gigabyte sticks might seem…
Your retail RAM prices are inflated about 3x, but I think your point is correct.
You know how Google has always had that little blurb about how long it took to process your query on the results page? Back when they were for sure running everything out of RAM, it was usually something comically small like 0.00025 seconds. I just checked and it's now more like 0.25 seconds.
Perhaps they've done testing and found that absurdly fast results don't matter (or no longer matter) as much as they thought?
Re: Our new search index: Caffeine
#34I'm waiting for Google to open up an API or dashboard on top of Caffeine to compete with some of the monitoring tools like Radian6 and Jive. They could own that space and it's growing like crazy right now. Plus, we don't need any more companies coming out with "social web" search engines.
Re: Our new search index: Caffeine
#35Earlier quoted context omitted.
Well, the serving part of their index is in memory (or largely so) as has been publicly announced. But are you really suggesting they might do the same for indexing? Let's suppose for a second that they can get 8GB sticks of RAM for 1/3 the retail price of around $600, and for the sake of the back of the envelope calculation, let's round it up to ten GB. So let's call it $20/GB of RAM. Four gigabyte sticks might seem…
Your retail RAM prices are inflated about 3x, but I think your point is correct. You know how Google has always had that little blurb about how long it took to process your query on the results page? Back when they were for sure running everything out of RAM, it was usually something comically small like 0.00025 seconds. I just checked and it's now more like 0.25 seconds. Perhaps they've done testing and found that a…
Your memory might've been wrong. 0.00025 is just too small a number for a system like this (Multi tier, RPC every where distributed systems).
Re: Our new search index: Caffeine
#36Earlier quoted context omitted.
Well, the serving part of their index is in memory (or largely so) as has been publicly announced. But are you really suggesting they might do the same for indexing? Let's suppose for a second that they can get 8GB sticks of RAM for 1/3 the retail price of around $600, and for the sake of the back of the envelope calculation, let's round it up to ten GB. So let's call it $20/GB of RAM. Four gigabyte sticks might seem…
Your retail RAM prices are inflated about 3x, but I think your point is correct. You know how Google has always had that little blurb about how long it took to process your query on the results page? Back when they were for sure running everything out of RAM, it was usually something comically small like 0.00025 seconds. I just checked and it's now more like 0.25 seconds. Perhaps they've done testing and found that a…
Sorry, that's just what I saw on Newegg for 8GB sticks. I did not intend to mislead.
I am under the impression that most of Google's index is still served out of ram. Certainly I never saw any announcements to the contrary. The other poster also pointed out that sub-millisecond latencies are pretty unlikely. Considering all the diverse sets of data they need to pull from, it would be similarly difficult to believe that tens or hundreds of disk seeks could be carried out in a timely manner for each query to yield a 200 ms total calculation time. If I had to guess they have probably just started taking factors into account that they did not before. For example, perhaps they are measuring total internal latency rather than just the latency of a particular sub-system. Or maybe there are components that go to disk while the posting list lookups are done in RAM.
Re: Our new search index: Caffeine
#37[deleted]
Re: Our new search index: Caffeine
#38Java or C++
Google uses only four languages in their codebase: C++, Java, Python and Javascript. Yegge has covered this pretty extensively in the past. Go Google him.
Re: Our new search index: Caffeine
#39Earlier quoted context omitted.
Considering they had that page with pigeons to explain how pagerank works, I would assume it's not supposed to be very serious.
The pigeon ranking page was an April 1st joke. It's exactly funny in one day of the year. Saying how you'll rank 85% of web searches is a little bit more serious business.
This shouldn't affect the ranking; updated data will just get to the ranking algorithm faster, is all.
Re: Our new search index: Caffeine
#40Earlier quoted context omitted.
You think Google's mainline storage of their index is on disk ? It is to laugh. I don't think that's been true since around the time they stopped building server shelves out of lego or cardboard. They've been keeping the full text of the web in RAM (for the snippets), with indexes, several times over. With independent live siblings in multiple datacenters.
Well, the serving part of their index is in memory (or largely so) as has been publicly announced. But are you really suggesting they might do the same for indexing? Let's suppose for a second that they can get 8GB sticks of RAM for 1/3 the retail price of around $600, and for the sake of the back of the envelope calculation, let's round it up to ten GB. So let's call it $20/GB of RAM. Four gigabyte sticks might seem…
That link has over 7 billion of capex for 2006-2009 there without accounting for their earlier pre-2006 hardware. It's eminently feasible.