From the screenshot, it's very funny that one of the indexed sites is www.llresearch.org, which looks like it's run by a crackpot.
I built a 500k-domain search engine for makers in a weekend for $10
41–50 of 102 posts
Re: I built a 500k-domain search engine for makers in a weekend for $10
#42Domains are way more than just 40M though.
From what I understand the aim was not to collect all the domains on the web but focus on personal website, etc. and avoid corporate web sites.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#43Here's my impressions of your algorithm: 1. read each site 2. rent a 4090 with https://vast.ai to run vllm 3. let llm model invent its own category and tag names freely 4. save 1KB of metadata each a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags. 5. `code is going up as open source` soon (TM)
Code appears to already be up: https://github.com/alexmorleyfinch/marlin
Re: I built a 500k-domain search engine for makers in a weekend for $10
#44How do you build a list of domains you want to index ? I see there is a fetcher and a spider in the code but so for I haven't found how to build that list.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#45[dead]
Re: I built a 500k-domain search engine for makers in a weekend for $10
#46[dead]
Re: I built a 500k-domain search engine for makers in a weekend for $10
#47Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea. I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
It feels like we've hit a point where search engines can become what "todo list apps" were for devs 10 years ago. What a homebrewed solution lacks in coverage it excels in indexing and serving a small slice of the internet really really well.
To be fair they are a supremely interesting problem to hack away at, and one that will meet you where you are.
Almost anyone can put together a basic search engine in a few thousand lines of code, it's just not very hard to make a program that will index a few million documents better than Confluence.
Then, between that first ansatz and a working scalable internet search engine, you have a pile of interesting problems touching every aspect of computer science and computer hardware and networking, enough so that hundreds of people will have gotten PhDs in narrow sub-problems of those problems you'll be facing.
It's great because you can just tackle the stuff you feel comfortable approaching and leave the rest for later.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#48Earlier quoted context omitted.
I'm not going to spend an hour trying to distill the AI-slop to find out what potential golden nugget may lie in there. It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."
[flagged]
It’s not my problem because I can safely skip ai slop without fomo.
It’s extremely rare that someone smart with something smart to say produces this crap.
Like someone said upthread if the author isn’t willing to spend time writing cogently then I don’t think it’s worth my time to try to parse it.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#49Earlier quoted context omitted.
[flagged]
> because that’s how things will look like from now on. No it's not. People are increasingly getting fed up with slops like this -- you can see comments in HN discussions. Many people just completely skip those articles.
> People are increasingly getting fed up with slops like this -- you can see comments in HN discussions.
Why claim 'people' (plenty of people get value from satisficing) and even try to use 'see comments in HN discussions' as if there is something that actually documents that ie 'we all think it'.
I skimmed it and even skimming it I got value from (what some others here) call 'slop'. (The value was basically it gave me an idea of something I could implement and again I didn't have to study it or even understand everything (or agree) just seeing someone did that made me think about something I might want to do.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#50Earlier quoted context omitted.
[flagged]
> because that’s how things will look like from now on. No it's not. People are increasingly getting fed up with slops like this -- you can see comments in HN discussions. Many people just completely skip those articles.
More importantly, I’ve started my own little blacklist of people and site I just ignore.
There’s people at work whose messages get completely ignored after too many times of posting verbose, useless, ai slop.