Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a 500k-domain search engine for makers in a weekend for $10

alexmorleyfinch.github.io

Post reply on HN
Print view View original

11–20 of 102 posts

Re: I built a 500k-domain search engine for makers in a weekend for $10

#12

Post by marginalia_nu »

Interesting project. Website discovery is indeed in a pretty dire spot, definitely a space that needs innovation. An auto-labeled website directory isn't that silly of an idea.

I have a 400 GB sqlite database with samples of rendered root document DOMs I use for ad detection in Marginalia Search I've been meaning to explore similar ideas using.

Re: I built a 500k-domain search engine for makers in a weekend for $10

#15

Post by juleiie »

uean wrote: post #10
juleiie wrote: post #8

Earlier quoted context omitted.

[flagged]

I'm not going to spend an hour trying to distill the AI-slop to find out what potential golden nugget may lie in there. It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."

[flagged]

Re: I built a 500k-domain search engine for makers in a weekend for $10

#16

Post by eggbrain »

This is actually where I see software going in the short term -- cloud moving to local.

A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search.

But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations for websites, and build search exactly as you want it.

The main limiter now is hard drive space (and to an extent, local compute) -- but right now it feels like the 70s again where the terminal into a remote server turned into building applications locally.

Re: I built a 500k-domain search engine for makers in a weekend for $10

#17

Post by ImPostingOnHN »

juleiie wrote: post #8
headz wrote: post #6

TS;DR: Too Sloppy; Didn't Read.

[flagged]

"slop" is a perfectly valid judgement of content

surely if we're expected to read this ourselves, the author can write it themselves?

indeed, it's helpful to the author, too: writing helps you learn

Re: I built a 500k-domain search engine for makers in a weekend for $10

#18

Post by jeroenhd »

iFire wrote: post #5

Here's my impressions of your algorithm: 1. read each site 2. rent a 4090 with https://vast.ai to run vllm 3. let llm model invent its own category and tag names freely 4. save 1KB of metadata each a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags. 5. `code is going up as open source` soon (TM)

The technical details are on another page: https://alexmorleyfinch.github.io/marlin/history/v1/article/...

Your impressions seem about right, but there are a few control steps it seems.

Re: I built a 500k-domain search engine for makers in a weekend for $10

#19

Post by uean »

juleiie wrote: post #15
uean wrote: post #10

Earlier quoted context omitted.

I'm not going to spend an hour trying to distill the AI-slop to find out what potential golden nugget may lie in there. It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."

[flagged]

Thanks for this. My point is proven.
Post reply on HN