Earlier quoted context omitted.
Thanks for this. My point is proven.
[flagged]
How did web3 and crypto work out for you buddy?
I built a 500k-domain search engine for makers in a weekend for $10
31–40 of 102 posts
Re: I built a 500k-domain search engine for makers in a weekend for $10
#32Re: I built a 500k-domain search engine for makers in a weekend for $10
#33Here's my impressions of your algorithm: 1. read each site 2. rent a 4090 with https://vast.ai to run vllm 3. let llm model invent its own category and tag names freely 4. save 1KB of metadata each a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags. 5. `code is going up as open source` soon (TM)
They really needn't have specified "in a weekend" cause yeah we can tell.
Since when has low effort become a selling point anyhow?
Re: I built a 500k-domain search engine for makers in a weekend for $10
#34Domains are way more than just 40M though.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#35Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#36Earlier quoted context omitted.
I'm not going to spend an hour trying to distill the AI-slop to find out what potential golden nugget may lie in there. It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."
[flagged]
> because that’s how things will look like from now on.
No it's not.
People are increasingly getting fed up with slops like this -- you can see comments in HN discussions.
Many people just completely skip those articles.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#37Re: I built a 500k-domain search engine for makers in a weekend for $10
#38Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea. I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
It feels like we've hit a point where search engines can become what "todo list apps" were for devs 10 years ago.
What a homebrewed solution lacks in coverage it excels in indexing and serving a small slice of the internet really really well.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#39Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea. I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
I have been wanting do do this. The biggest source of domains is certificate transparency logs. Also ICANN zone files. According to some scientific papers these cover 88% of all registered domains. You could crawl dns for CNAME records with all ipv4 IPs by distributing requests across dozens of DNS servers, the internet archive or the common crawl but doing it for the internet archive is a dick move without giving them money
There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages
Re: I built a 500k-domain search engine for makers in a weekend for $10
#40Earlier quoted context omitted.
I'm not going to spend an hour trying to distill the AI-slop to find out what potential golden nugget may lie in there. It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."
[flagged]
If I want to read AI output, I’ll just ask my own AI. I’m not going to bother reading someone else’s AI.