Earlier quoted context omitted.
>What do you reckon the cost of doing this would be? That is indeed the key question. I tried to find out before commenting but the information I found is very vague. Internet Archive spends millions per year, but their index is updated far too slowly for a search engine. I have no idea what it costs to create a Google-size index. Do you think the Mojeek index is good enough to compete with Google?
Internet archive snapshot the whole page while google only index the first few kb of text data on every page (forgot the number, was it 100kb max per page?), so maybe it'll cost lower to build a search engine index compared to an internet preservation project? Assuming they'll need to index 820 billion pages (the number of pages preserved in the internet archive), at 100kb each, and assuming they use a database with…
So you're looking at maybe 20 bn docs, 4 Kb each. 100 Tb, before compression.