Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I built a 500k-domain search engine for makers in a weekend for $10

alexmorleyfinch.github.io

Post reply on HN
Print view View original

21–30 of 102 posts

Re: I built a 500k-domain search engine for makers in a weekend for $10

#21

Post by dylan604 »

eggbrain wrote: post #16

This is actually where I see software going in the short term -- cloud moving to local. A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search. But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations…

The number of times we've gone from cloud/server access via terminal to local compute back and forth is something that always makes me laugh a bit.

Re: I built a 500k-domain search engine for makers in a weekend for $10

#22

Post by marginalia_nu »

pimlottc wrote: post #20

Sometimes I think people forget how capable computers are. 500k is not much. You can just slap that in a Lucene instance. This is a solved problem.

Approaching search by just tossing the data in Lucene is how you end up with Confluence's search box though.

Re: I built a 500k-domain search engine for makers in a weekend for $10

#25

Post by vivzkestrel »

juleiie wrote: post #15
uean wrote: post #10

Earlier quoted context omitted.

I'm not going to spend an hour trying to distill the AI-slop to find out what potential golden nugget may lie in there. It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."

[flagged]

- you are going to regret offloading this much of your brain's critical thinking to an external source in the long run

- every night a good number of neuron connections in your brain are automatically severed because you did not use that ability

- over long periods of time, it ll take you to a point where you wont be able to write a simple factorial program on your own without gpt telling you

- i am not saying this to incite you or mock you or anything. i am just concerned about how much of your thinking you are offloading

Re: I built a 500k-domain search engine for makers in a weekend for $10

#26

Post by yieldcrv »

juleiie wrote: post #15
uean wrote: post #10

Earlier quoted context omitted.

I'm not going to spend an hour trying to distill the AI-slop to find out what potential golden nugget may lie in there. It's impossible to judge the content if it's buried under a landfill. "If you won't take the time to write it, I won't take the time to read it."

[flagged]

yes but tell your slopywriter to be less verbose, get to the point

Re: I built a 500k-domain search engine for makers in a weekend for $10

#28

Post by sandeepkd »

orliesaurus wrote: post #13

Like a personal Google? How do you bypass all the captcha, ip bans, cloudflare turnstile antibot stuff etc?

Thats the fun part, the user just went with happy path. Javascript, captchas, cloudflare protected content did not made to the catalogue. This sort of use case exists in LLM training data a lot which makes it easier. The data gathered by the user is not really practically useful cause there are way too many gotchas when it comes to web scraping and building a catalogue (source: I have done scraping for a particular domain data and had to do at least 10+ iterations to get it >90 right)

Re: I built a 500k-domain search engine for makers in a weekend for $10

#29

Post by danggggg »

vivzkestrel wrote: post #25
juleiie wrote: post #15

Earlier quoted context omitted.

[flagged]

- you are going to regret offloading this much of your brain's critical thinking to an external source in the long run - every night a good number of neuron connections in your brain are automatically severed because you did not use that ability - over long periods of time, it ll take you to a point where you wont be able to write a simple factorial program on your own without gpt telling you - i am not saying this t…

[dead]

Re: I built a 500k-domain search engine for makers in a weekend for $10

#30

Post by an0malous »

dylan604 wrote: post #21
eggbrain wrote: post #16

This is actually where I see software going in the short term -- cloud moving to local. A few years ago, if you wanted translation, you'd use Google Translate. If you wanted to search the web, you'd use Google search. But for a few gigabytes, you can now install nllb-200-distilled-600M, and get translations for almost any language locally. You can have your computer crawl the web, create abstracts and categorizations…

The number of times we've gone from cloud/server access via terminal to local compute back and forth is something that always makes me laugh a bit.

It's more of a pipeline than a back-and-forth. New abilities happen in the cloud first because they require specialized, higher capacity resources and then move towards being local as the resource usage gets optimized.
Post reply on HN