321–330 of 346 posts
Earlier quoted context omitted.
implementing a toy datalog should take no more than a week.
maybe less
Earlier quoted context omitted.
Nowadays (and increasingly, going forward) it's possible to be a very productive programmer without knowing much low-level stuff. This may seem unfair to those who spent years wrestling assembly and then C pointers but that's just today's reality. It's not possible to be a "productive" musician on a traditional instrument (i.e. excluding iPads) without knowing how to play scales, chords, etc.
Tell that to most guitarists. Vast majority are self taught and can’t read music; a number are excellent players.
While writing a text editor, a compiler, an operating system, or a raytracer might make you a better programmer, it won't make you a better software engineer. In fact, it might make you worse at software engineering, because it embodies the disastrous "Not Invented Here" doctrine. Hackers like to obsess about Big-O, data structures, HoTT, and other high-theory stuff, yet the following skills, essential for software e…
Earlier quoted context omitted.
A simple search engine is certainly doable from scratch in a matter of weeks, complete with most things expected from a search engine. Similar to compilers or tiny OSes, one can go as hardcore as necessary, or just stick to basic stuff. Of all the typical personal challenge style projects, databases are probable the hardest to build, and even that is not impossible.
making a database is easy. making a fast database is what's hard.
Earlier quoted context omitted.
That's not enough. It needs to be sharded and handle things like 10 search terms, each of which match a million documents and you're trying to find the intersection. Across shared results from 20 servers. Quickly. That's not a textbook algorithm.
Intersecting postingslists is a solved problem. You can do it in sublinear time with standard search engine algorithms. The problem space is embarassingly parallel, so sharding is no problem. Although, realistically, you probably only need 1 server to cope with the load and storage needs. This isn't 2004. Servers are big and fast now, as long as you don't try to use cloud compute.
And of course all of this is based on "standard" algorithms. But it doesn't change the fact that implementing them all to build a functional performant web crawler+database+search is a project that is larger than any one person could build from scratch.
It would only be feasible to build on top of existing libraries, which is the whole point here.
Earlier quoted context omitted.
Intersecting postingslists is a solved problem. You can do it in sublinear time with standard search engine algorithms. The problem space is embarassingly parallel, so sharding is no problem. Although, realistically, you probably only need 1 server to cope with the load and storage needs. This isn't 2004. Servers are big and fast now, as long as you don't try to use cloud compute.
1 server to index the whole internet? I don't care how big your server is -- that's going not to happen. And of course all of this is based on "standard" algorithms. But it doesn't change the fact that implementing them all to build a functional performant web crawler+database+search is a project that is larger than any one person could build from scratch . It would only be feasible to build on top of existing librar…
Well let's do the math!
1 billion bytes is 1 GB. The average text-size of a document is ~5 KB, that is the HTML, uncompressed. 1 billion documents is thus about 5 TB raw HTML. In practice the index would be smaller, say 1 TB, because you're not indexing the raw HTML, but a fixed width representation of the words therein, typically with some form of compression as well. (In general, the index is significantly smaller than the raw data unless you're doing something strange)
You could buy a thumbdrive that would a search index for 1 billion documents. A real server uses enterprise SSDs for this stuff, but even so, you don't even have to dip into storage server specs to outfit a single server with hundreds of terabytes of SSD storage.
> And of course all of this is based on "standard" algorithms. But it doesn't change the fact that implementing them all to build a functional performant web crawler+database+search is a project that is larger than any one person could build from scratch.
I'm still not seeing why this is impossible.
> It would only be feasible to build on top of existing libraries, which is the whole point here.
I don't think there are existing libraries for most of this stuff. Most off-the-shelf stuff in the search space is for indexing smaller corpuses, they generally aren't built to scale up to corpus sizes in the billions (e.g. using 32 bit keys etc.)
Earlier quoted context omitted.
Genuine question: what value was added by getting rid of the frameworks?
20,000 fewer lines of code, faster compile / deploy times, significantly less interfacing / translation between "their" types and "their" apis, and a much better control over types and structure across our codebase b/c we didn't need "their" types and structure anywhere. It had crept everywhere. It's just faster, cleaner, and easier in a few cases to do precisely what you need right now, rather than anticipate a mill…
Earlier quoted context omitted.
Simple methods of search like exact matching are very fast using textbook algorithms. There are well known algorithm like suffix tree which could search in millions of documents in milliseconds.
That's not enough. It needs to be sharded and handle things like 10 search terms, each of which match a million documents and you're trying to find the intersection. Across shared results from 20 servers. Quickly. That's not a textbook algorithm.
Earlier quoted context omitted.
Intersecting postingslists is a solved problem. You can do it in sublinear time with standard search engine algorithms. The problem space is embarassingly parallel, so sharding is no problem. Although, realistically, you probably only need 1 server to cope with the load and storage needs. This isn't 2004. Servers are big and fast now, as long as you don't try to use cloud compute.
1 server to index the whole internet? I don't care how big your server is -- that's going not to happen. And of course all of this is based on "standard" algorithms. But it doesn't change the fact that implementing them all to build a functional performant web crawler+database+search is a project that is larger than any one person could build from scratch . It would only be feasible to build on top of existing librar…
I don't think you've made a single point that substantiates that claim and neither has anyone else, really. The person you're arguing with has built (as per their own estimation) more from scratch when it comes to the actual search engine core than anything else in their own project (https://marginalia.nu/).
Honestly, it seems more like the people arguing a search engine is somehow "off limits" from scratch are doing so because they imagine it's less feasible, probably because they simply don't know or are in general pretty bad at implementing things from scratch, period.
To a large degree there is also a wide difference in actual project scope. Remember that we are comparing this to implementing Space Invaders, one of the simplest games you can make, and one that is doable in an evening even if you write your own renderer.
To talk about "webscale" (a term that is so deliciously dissonant with most of the web world which runs at a scale several 100 times slower and less efficient than things should be) is like suddenly talking about implementing deferred shading for the Space Invaders game. Even if this was something you'd want to explore later, it's just not something you're doing right now because this is something you're using to learn as you go.
Earlier quoted context omitted.
> You'd be foolish to write a console emulator and not import someone else's CPU code. Why? It's part of the learning and it's not particularly complex.
That part has been done to death and optimized to hell, that was my main thought. Plus it's a ton of research on the opcodes etc.
Also, I don't get the point about research: It's a learning project... Doing research and learning is the actual point. Sure, maybe you slice off one part right now if you're not interested in learning how the CPU works, but to say that you'd be foolish to not use someone else's implementation seems to me to be missing the point by quite a margin.