I built a 500k-domain search engine for makers in a weekend for $10
alexmorleyfinch.github.io
I built a 500k-domain search engine for makers in a weekend for $10
1–10 of 102 posts
Re: I built a 500k-domain search engine for makers in a weekend for $10
#2Check out my latest project! You can fork it, tweak the policy manually or with AI, run the system and watch the data come in! It's engineered to keep a low data footprint, so 500k domains fits into 1GB on disk. If you have local models it's free! You just might not get the best throughput depending on your GPU. My production data is not exposed anywhere yet, and I may never expose it. The point is for you to fork and make your own policy, and thus your own personal search engine! The article covers basic analysis on my data, so it's worth a read if you're interested! A deeper analysis may arrive with V2 if I ever do it
Re: I built a 500k-domain search engine for makers in a weekend for $10
#3[dead]
Re: I built a 500k-domain search engine for makers in a weekend for $10
#4[dead]
Re: I built a 500k-domain search engine for makers in a weekend for $10
#5Here's my impressions of your algorithm:
1. read each site
2. rent a 4090 with https://vast.ai to run vllm
3. let llm model invent its own category and tag names freely
4. save 1KB of metadata each
a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#6TS;DR: Too Sloppy; Didn't Read.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#7I think Kagi Small Web filter would give you very similar results.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#8TS;DR: Too Sloppy; Didn't Read.
[flagged]
Re: I built a 500k-domain search engine for makers in a weekend for $10
#9Re: I built a 500k-domain search engine for makers in a weekend for $10
#10TS;DR: Too Sloppy; Didn't Read.
[flagged]
I'm not going to spend an hour trying to distill the AI-slop to find out what potential golden nugget may lie in there.
It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."