This is a damn good project. Makes me want to make headway into an idea I've had for quite some time P2P search...we'll see.
I built a 500k-domain search engine for makers in a weekend for $10
71–80 of 102 posts
Re: I built a 500k-domain search engine for makers in a weekend for $10
#72Built a free CLI for this as well: https://github.com/solozerolabs/Namera
Checks socials and trademark too
Re: I built a 500k-domain search engine for makers in a weekend for $10
#73Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea. I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.
I have been wanting do do this. The biggest source of domains is certificate transparency logs. Also ICANN zone files. According to some scientific papers these cover 88% of all registered domains. You could crawl dns for CNAME records with all ipv4 IPs by distributing requests across dozens of DNS servers, the internet archive or the common crawl but doing it for the internet archive is a dick move without giving th…
> There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages
Where do you get stats on this info? Not refuting your claims but it's interesting data. I would've thought a wider gap in all biz to active websites since some countries run nearly all their biz on whatsapp/telegram/wechat.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#74I was thinking that a search engine that ignores anything with advertising on it would be useful. This has inspired me to give it a go (on the weekend even)
Re: I built a 500k-domain search engine for makers in a weekend for $10
#75It takes a bit of setup and a huge download, but every time I need a good domain I follow this old post from Derek Sivers. I have Claude de-dupe it and turn it into a searchable database (on my machine), then have it search genres and terms I'm looking for. It's a task Claude is very well-suited to, from the technical implementation to back-and-forth about selections. [link]: https://sive.rs/com
[deleted]
Re: I built a 500k-domain search engine for makers in a weekend for $10
#76FYI for those needing a list of domains
Subject: I want all domains and subdomains https://groups.google.com/g/common-crawl/c/XC2QmOE-sdI?pli=1
or google for COMMON CRAWL
Re: I built a 500k-domain search engine for makers in a weekend for $10
#77Earlier quoted context omitted.
- you are going to regret offloading this much of your brain's critical thinking to an external source in the long run - every night a good number of neuron connections in your brain are automatically severed because you did not use that ability - over long periods of time, it ll take you to a point where you wont be able to write a simple factorial program on your own without gpt telling you - i am not saying this t…
I am not offloading anything. I need way more critical thinking than I ever needed before to review solutions proposed by my digital slaves to evaluate whether they are garbage or good. If you think that having slaves means you don’t need to be a good administrator, then you are sorely mistaken. It requires more brainpower than it ever did to manage a project the size what used to do a 100 people team. All on your ow…
> I need way more critical thinking than I ever needed before to review
Review does not require more critical thinking than designing or building.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#78Earlier quoted context omitted.
I am not offloading anything. I need way more critical thinking than I ever needed before to review solutions proposed by my digital slaves to evaluate whether they are garbage or good. If you think that having slaves means you don’t need to be a good administrator, then you are sorely mistaken. It requires more brainpower than it ever did to manage a project the size what used to do a 100 people team. All on your ow…
> I need way more critical thinking than I ever needed before to review Review does not require more critical thinking than designing or building.
Designing is all I do basically. AI just a calculator but I tell it what to do and guide it.
Guys you cannot win this argument. We will soon be in a new era. It’s like the old math heads complaining about calculators.
Does anyone complain about calculators making us dumb nowadays?
You can of course avoid this tech like a plague and miss out and that’s fine. No one cares
Re: I built a 500k-domain search engine for makers in a weekend for $10
#79Earlier quoted context omitted.
> I need way more critical thinking than I ever needed before to review Review does not require more critical thinking than designing or building.
Designing is all I do basically. AI just a calculator but I tell it what to do and guide it. Guys you cannot win this argument. We will soon be in a new era. It’s like the old math heads complaining about calculators. Does anyone complain about calculators making us dumb nowadays? You can of course avoid this tech like a plague and miss out and that’s fine. No one cares
> Does anyone complain about calculators making us dumb nowadays
It's a fact that people who rely on calculators for everything lose their skills at mental math
What do you think will happen when people rely on computers for everything?
Your new era is going to be full of idiots
And I wouldn't care about that except you all seem to be insisting on dragging everyone down with you
Re: I built a 500k-domain search engine for makers in a weekend for $10
#80Reminds me that AltaVista's servers ran in 4G of RAM (there were famous, at the time, pics of the circuit boards - DEC was rightfully proud of this, 30 years ago) and that a modern AltaVista should run on a decent laptop :-)
During AltaVista's prime, 32MB would have been a lot of memory for a typical computer.
I still can't find pics of the boards (other than on ebay :-) but https://seltzerbooks.com/alta.html has quotes like "To get a sense of the relative size of the RAM in AltaVista Search, consider that a typical personal computer today comes with 8 to 16MB of RAM, and the AltaVista Index Servers each have 6GB of RAM—about 500 times more." and "AltaVista shows the practical value of machines that have more than 4 gigabytes of physical main memory." (Dick Sites, DEC SRL) The book is from 1997.