Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a 500k-domain search engine for makers in a weekend for $10

alexmorleyfinch.github.io

Post reply on HN
Print view View original

51–60 of 102 posts

Re: I built a 500k-domain search engine for makers in a weekend for $10

#51

Post by prepend »

fg137 wrote: post #35

Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing.

I was wondering the same thing. I’ve wished for just a big blob of the web to grep and regex through, but I don’t think this is that much easier than using duckduckgo or even google.

Re: I built a 500k-domain search engine for makers in a weekend for $10

#53

Post by tpowell »

It takes a bit of setup and a huge download, but every time I need a good domain I follow this old post from Derek Sivers. I have Claude de-dupe it and turn it into a searchable database (on my machine), then have it search genres and terms I'm looking for. It's a task Claude is very well-suited to, from the technical implementation to back-and-forth about selections. [link]: https://sive.rs/com

Re: I built a 500k-domain search engine for makers in a weekend for $10

#54

Post by smokel »

moffkalast wrote: post #33
iFire wrote: post #5

Here's my impressions of your algorithm: 1. read each site 2. rent a 4090 with https://vast.ai to run vllm 3. let llm model invent its own category and tag names freely 4. save 1KB of metadata each a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags. 5. `code is going up as open source` soon (TM)

They really needn't have specified "in a weekend" cause yeah we can tell. Since when has low effort become a selling point anyhow?

I typically interpret it as an excuse, not a selling point.

Re: I built a 500k-domain search engine for makers in a weekend for $10

#55

Post by dreamforever »

fg137 wrote: post #35

Sorry I have a lot of trouble understanding what this is useful for. Like, I am never going to replace it with Google, DuckDuckGo, ChatGPT or even Bing.

It's not for that, sorry, I should have been more specific. It's for people who wanna put in the effort and steer their own crawl to surface their own slice of the web. The article is just a little story of the journey

Re: I built a 500k-domain search engine for makers in a weekend for $10

#56

Post by order-matters »

juleiie wrote: post #8
headz wrote: post #6

TS;DR: Too Sloppy; Didn't Read.

[flagged]

did you edit it? did you fully read everything yourself and take out anything extra that didnt need to be in there like unnecessary comparisons and typical AI idioms? tweak the language to be less dramatic or emphatic about things that do not need emphasis?

AI is a great tool, but it is somewhere between a 3D printer and a CNC machine. it can make something smooth and easy to handle and maybe good enough for personal use but you would want to run it through some Finishing steps before giving its output to someone else. In some cases thats using the output as a mold for a full recreation and other times maybe its just sanding it a bit, smoothing out some rough spots and putting paint on it.

youre right not to care about if something was made by a human or a tool, but the full presentation of the content including editing is part of the content when it comes to a write-up.

Re: I built a 500k-domain search engine for makers in a weekend for $10

#57

Post by ModernMech »

ImPostingOnHN wrote: post #17
juleiie wrote: post #8

Earlier quoted context omitted.

[flagged]

"slop" is a perfectly valid judgement of content surely if we're expected to read this ourselves, the author can write it themselves? indeed, it's helpful to the author, too: writing helps you learn

I dunno, you'll notice several people here saying that got value from it, despite others not engaging with it at all. If your filter is that you don't engage with anything you don't expect a priori to get value from (based on weak signals), you'll miss a lot.

Re: I built a 500k-domain search engine for makers in a weekend for $10

#58

Post by dreamforever »

BaudouinVH wrote: post #44

How do you build a list of domains you want to index ? I see there is a fetcher and a spider in the code but so for I haven't found how to build that list.

Ah, I forgot to mention that anywhere. You have to provide your own. You can start from a small set, like 10 websites you like that have a bit of character, and it will also add any domains it finds from those 10

Re: I built a 500k-domain search engine for makers in a weekend for $10

#59

Post by dreamforever »

pavel_lishin wrote: post #41

From the screenshot, it's very funny that one of the indexed sites is www.llresearch.org, which looks like it's run by a crackpot.

There are all kinds of websites in here lol. There is some gold in here and I'm determined to surface it all. I had to wrap this up without full analysis cos it was dragging on

Re: I built a 500k-domain search engine for makers in a weekend for $10

#60

Post by dreamforever »

orliesaurus wrote: post #13

Like a personal Google? How do you bypass all the captcha, ip bans, cloudflare turnstile antibot stuff etc?

despite this limitation, there is still some good stuff out there, and with the priority steering, you can focus compute on what you actually want, fast and cheap.
Post reply on HN