131–140 of 156 posts
Earlier quoted context omitted.
Cant you just request the ICANN’s zone files and have the canonical list of the day?
Avoiding GIGO (Garbage In, Garbage Out). This is why we have computer-variants of Library Science and Archeology, Forensic Science and a bunch of other advanced knowledge (not AI, mind you).
The photo of the power socket right next to the sink looks safe
Well, I created my own domain index. I have not crawled every page inside domains, but it is not my goal. I have 1542766 domains. Might not be much, but it is an honest work. It is available as a github repo, so anybody that wants to start crawling has some initial data to kick off. Links https://github.com/rumca-js/Internet-Places-Database
Cant you just request the ICANN’s zone files and have the canonical list of the day?
I know that you cannot count and visit every domain, so the list will never be finished, but I am happy with the results.
Well, I created my own domain index. I have not crawled every page inside domains, but it is not my goal. I have 1542766 domains. Might not be much, but it is an honest work. It is available as a github repo, so anybody that wants to start crawling has some initial data to kick off. Links https://github.com/rumca-js/Internet-Places-Database
What a nice project. What inspired this initially? FYI there's a broken link in your readme: https://rumca-js.github.io/internet full internet search
Earlier quoted context omitted.
> The effect is most likely to occur when there are no obvious reasons for performing the task. Because expending effort to perform a useless or unenjoyable task, or experiencing unpleasant consequences in doing so, is cognitively inconsistent (see cognitive dissonance), people are assumed to shift their evaluations of the task in a positive direction to restore consistency. I’m not following you. https://dictionary.…
It's not limited to physical effort. Wikipedia's example has embarassment in place of effort; presumably, money could also work.
Granted I’ve been annoyed by similar occurrences with other services, but not to the point of suspecting collusion between the service and the public like the GP comment did.
Searching on the web takes effort. I don’t think this sentiment is controversial. Especially not on HN.
But do you think that because/if searching on the web takes effort and because people have to pay for Kagi, they are compelled to exaggerate its usefulness in public to justify the cost?
Thank you for those who tried it, and I'm sorry if you were one of the people it didn't perform for. As far as load goes this was the first day it truly had a "trial by fire".
I was trying to do this in 2023! The hardest part about building a search engine is not the actual searching though, it is (like others here have pointed out), building your index and crawling the (extremely adversarial) internet, especially when you're running the thing from a single server in your own home without fancy rotating IPs. I hope this guy succeeds and becomes another reference in the community like the m…
Is the common crawl usable for something like this? https://commoncrawl.org
I was trying to do this in 2023! The hardest part about building a search engine is not the actual searching though, it is (like others here have pointed out), building your index and crawling the (extremely adversarial) internet, especially when you're running the thing from a single server in your own home without fancy rotating IPs. I hope this guy succeeds and becomes another reference in the community like the m…
Why can't crawling be crowd sourced? It would solve ip rotation and spread the load