The author struggled with categorization simply because they did not truly understand k-means clustering, a fundamental concept in this kind of computing science.
You cannot just let a model run wild.
I built a 500k-domain search engine for makers in a weekend for $10
81–90 of 102 posts
Re: I built a 500k-domain search engine for makers in a weekend for $10
#82Earlier quoted context omitted.
They really needn't have specified "in a weekend" cause yeah we can tell. Since when has low effort become a selling point anyhow?
it was by no means low effort
Isn't 60 hours by definition low effort? Setting an LLMs effort value to high doesn't count
Re: I built a 500k-domain search engine for makers in a weekend for $10
#83Here's my impressions of your algorithm: 1. read each site 2. rent a 4090 with https://vast.ai to run vllm 3. let llm model invent its own category and tag names freely 4. save 1KB of metadata each a. a small local language model that reads each one and writes a name, two or three sentences, a category, and a handful of tags. 5. `code is going up as open source` soon (TM)
don't forget:
6. let llm write a blog post about this conversation
Re: I built a 500k-domain search engine for makers in a weekend for $10
#84Here's an idea: Each day you could summarize all the new sites into an email. Call it NCSA "What's New" or something like that. ;)
Re: I built a 500k-domain search engine for makers in a weekend for $10
#85Earlier quoted context omitted.
I have been wanting do do this. The biggest source of domains is certificate transparency logs. Also ICANN zone files. According to some scientific papers these cover 88% of all registered domains. You could crawl dns for CNAME records with all ipv4 IPs by distributing requests across dozens of DNS servers, the internet archive or the common crawl but doing it for the internet archive is a dick move without giving th…
> There's about 200 million active domains currently. That's about 66% of all businesses worldwide of which there are around 300 million. Around 100 to 150 million have active webpages Where do you get stats on this info? Not refuting your claims but it's interesting data. I would've thought a wider gap in all biz to active websites since some countries run nearly all their biz on whatsapp/telegram/wechat.
I remember 300 million is the number of businesses on Google maps. I forgot how I got that number though. I didn't crawl Google maps. A lot or even most of those businesses run via chat often have no physical presence besides the houses of their owners so houses on Google maps are not counted. There are a lot of businesses on shopping malls and B2B businesses that aren't on Google maps often because they don't know they can add themselves to Google maps.
I did count the number of ICANN domains on their zone files on czds and there are a few services selling pre compiled domain lists that do crawling for marketers (it is forbidden by ICANN to use zone files for marketing) I don't remember how I got the active website count of 100 to 150 million, maybe I interpolated by checking some ICANN domains or there's a domain list service that does it. Domain list services usually say how many domains are active regardless of whether they have a website or not
Of course parked domains add up to the active website count even though they have no relevant content
Re: I built a 500k-domain search engine for makers in a weekend for $10
#86Earlier quoted context omitted.
"slop" is a perfectly valid judgement of content surely if we're expected to read this ourselves, the author can write it themselves? indeed, it's helpful to the author, too: writing helps you learn
I dunno, you'll notice several people here saying that got value from it, despite others not engaging with it at all. If your filter is that you don't engage with anything you don't expect a priori to get value from (based on weak signals), you'll miss a lot.
it's not that it might help or not, it's more like,
"if an AI wrote this, that means I can generate the exact same text (or better) myself, and be _even more_ helped, whenever I'm interested in this topic, by using AI"
having a blog full of ai content is like having a web page that's a list of links to searches for random keywords on google: what's your contribution? picking a topic? is that anything?
Re: I built a 500k-domain search engine for makers in a weekend for $10
#87Earlier quoted context omitted.
I dunno, you'll notice several people here saying that got value from it, despite others not engaging with it at all. If your filter is that you don't engage with anything you don't expect a priori to get value from (based on weak signals), you'll miss a lot.
it's not that it might help or not, it's more like, "if an AI wrote this, that means I can generate the exact same text (or better) myself, and be _even more_ helped, whenever I'm interested in this topic, by using AI" having a blog full of ai content is like having a web page that's a list of links to searches for random keywords on google: what's your contribution? picking a topic? is that anything?
That's only if you assume the human didn't have any control over what the AI wrote, and I think that's more and more a very wrong assumption. It's easy to have a lot of control over LLM output, and to use expertise to steer it (non experts steer the LLM in very different directions compared to experts). And the output being raw text, it's also very easy to edit and refactor it to say what needs to be said. Some people are even (unconsciously or not) adopting AI colloquialisms in their own writing. I think it's becoming very unclear where human and AI content begins and ends.
> having a blog full of ai content is like having a web page that's a list of links to searches for random keywords on google
What's any blog? A person just presses a bunch of buttons on a keyboard, is that anything?
Re: I built a 500k-domain search engine for makers in a weekend for $10
#88Earlier quoted context omitted.
it's not that it might help or not, it's more like, "if an AI wrote this, that means I can generate the exact same text (or better) myself, and be _even more_ helped, whenever I'm interested in this topic, by using AI" having a blog full of ai content is like having a web page that's a list of links to searches for random keywords on google: what's your contribution? picking a topic? is that anything?
That's only if you assume the human didn't have any control over what the AI wrote, and I think that's more and more a very wrong assumption. It's easy to have a lot of control over LLM output, and to use expertise to steer it (non experts steer the LLM in very different directions compared to experts). And the output being raw text, it's also very easy to edit and refactor it to say what needs to be said. Some peopl…
I think you dramatically overstate the human contribution, but if it exists, it would be seen by the author sharing their prompts. I can see how that might prove embarrassing, though, if their contribution is minimal, so I can see why so few people do it.
> What's any blog? A person just presses a bunch of buttons on a keyboard, is that anything?
If it's a human writing it, sure: it's a unique contribution by somebody who put forth effort to help others.
If it's just LLM output, then you might as well have a blog posting the results of random math equations from your calculator. Was any significant human effort involved? Odds are no, but "authors" using LLMs are free to prove otherwise by providing their prompts.
Re: I built a 500k-domain search engine for makers in a weekend for $10
#89Earlier quoted context omitted.
> I need way more critical thinking than I ever needed before to review Review does not require more critical thinking than designing or building.
Designing is all I do basically. AI just a calculator but I tell it what to do and guide it. Guys you cannot win this argument. We will soon be in a new era. It’s like the old math heads complaining about calculators. Does anyone complain about calculators making us dumb nowadays? You can of course avoid this tech like a plague and miss out and that’s fine. No one cares
https://news.ycombinator.com/item?id=48789042 bad idea bro
Re: I built a 500k-domain search engine for makers in a weekend for $10
#90> pages classified as “portfolio” or “zine” or “software” push their outbound links way up the priority list, pages classified as “corporate” or “docs” push theirs down.
And just like that, you recreated the internet of the 90s - early 00s. Brilliant!