Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a 500k-domain search engine for makers in a weekend for $10

alexmorleyfinch.github.io

Post reply on HN
Print view View original

91–100 of 102 posts

Re: I built a 500k-domain search engine for makers in a weekend for $10

#91

Post by vivzkestrel »

juleiie wrote: post #78
bluefirebrand wrote: post #77

Earlier quoted context omitted.

> I need way more critical thinking than I ever needed before to review Review does not require more critical thinking than designing or building.

Designing is all I do basically. AI just a calculator but I tell it what to do and guide it. Guys you cannot win this argument. We will soon be in a new era. It’s like the old math heads complaining about calculators. Does anyone complain about calculators making us dumb nowadays? You can of course avoid this tech like a plague and miss out and that’s fine. No one cares

- also no gloating but i wanna mention something

- i dont use calculators anymore

- it was impossibly hard at first

- but my mind is adjusting to it slowly

- i can add and subtract 7 digit numbers effortlessly now

- i am still working on multiplication and division though but we ll get there slowly

- let me give you an example live

2949745 - 2947648

- my thought process to do this goes like this

- 1st 3 digits are same ignore

9745 - 7648

- 2nd step is to round off 9700 - 7700 = 2000, add 45 to it and it makes 2045 , we are still 52 short, add 50 to it and make it 2095, now add 2 and it becomes 2097

- see? no calculator needed

- i can do this faster than you can type the numbers above inside your beloved calculator

Re: I built a 500k-domain search engine for makers in a weekend for $10

#92

Post by RadiozRadioz »

moffkalast wrote: post #33
iFire wrote: post #5

Here's my impressions of your algorithm: 1. read each site 2. rent a 4090 with https://vast.ai to run vllm 3. let llm model invent its own category and tag names freely 4. save 1KB of metadata each a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags. 5. `code is going up as open source` soon (TM)

They really needn't have specified "in a weekend" cause yeah we can tell. Since when has low effort become a selling point anyhow?

It's an interesting new type of brag. It roughly means: "I want you to know how advanced I am at using LLMs, and how AI-first I am. So here is how little time I spent, to prove that I am using LLMs as much as possible, demonstrating that I am ahead of the curve on this new trend"

Re: I built a 500k-domain search engine for makers in a weekend for $10

#93

Post by vivzkestrel »

deadbabe wrote: post #81

The author struggled with categorization simply because they did not truly understand k-means clustering, a fundamental concept in this kind of computing science. You cannot just let a model run wild.

- since you know what k-means clustering is

- why dont you tell me how you ll categorize this list with k-means?

Re: I built a 500k-domain search engine for makers in a weekend for $10

#96

Post by ccgreg »

alightsoul wrote: post #39
marginalia_nu wrote: post #12

Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea. I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.

I have been wanting do do this. The biggest source of domains is certificate transparency logs. Also ICANN zone files. According to some scientific papers these cover 88% of all registered domains. You could crawl dns for CNAME records with all ipv4 IPs by distributing requests across dozens of DNS servers, the internet archive or the common crawl but doing it for the internet archive is a dick move without giving th…

Both IA and CCF are private foundations and accept donations.

Re: I built a 500k-domain search engine for makers in a weekend for $10

#97

Post by deadbabe »

vivzkestrel wrote: post #93
deadbabe wrote: post #81

The author struggled with categorization simply because they did not truly understand k-means clustering, a fundamental concept in this kind of computing science. You cannot just let a model run wild.

- since you know what k-means clustering is - why dont you tell me how you ll categorize this list with k-means?

Run k-means over embeddings

Re: I built a 500k-domain search engine for makers in a weekend for $10

#98

Post by [deleted] »

prepend wrote: post #51
fg137 wrote: post #35

Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing.

I was wondering the same thing. I’ve wished for just a big blob of the web to grep and regex through, but I don’t think this is that much easier than using duckduckgo or even google.

[deleted]

Re: I built a 500k-domain search engine for makers in a weekend for $10

#99

Post by Xyra »

prepend wrote: post #51
fg137 wrote: post #35

Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing.

I was wondering the same thing. I’ve wished for just a big blob of the web to grep and regex through, but I don’t think this is that much easier than using duckduckgo or even google.

I've been working on grep.it.com. can currently grep over 200 TB of the internet
Post reply on HN