41–45 of 45 posts
Duckduckgo is now particularly riddled by AI spam. The top 5 summaries for lots of searches now begin with some variation of "In the fast-moving world of...". Utter shit. It'd be so easy to filter I wonder why they/Microsoft don't bother. Oh, wait...
How much is DDG Bing based?
Earlier quoted context omitted.
startpage does have ads, it's just somebody else than google tracking you.
You have just claimed that startpage is lying in their privacy policy. Please provide evidence for such a claim. https://www.startpage.com/en/privacy-policy?t=device "Our search result pages may include a small number of clearly labeled "sponsored links," which generate revenue and cover our operational costs. Those links are retrieved from platforms such as Google AdSense. In order to enable the prevention of click…
Still, there are ads.
Creating a thin, non-working wrapper around DuckDuckGo as a "hire me" ad, claiming it's a search engine and mentioning Google in the headline because it'll get more clicks.
Yes, I don’t know how that reached the frontpage. A failed 15 minute experiment from someone who has no clue what they are doing
I'm convinced that's a lot of what passes for software development in the 2020s: Make a splash screen and display some branding on top of older, robust, actually-complex projects.
Earlier quoted context omitted.
Yes, I don’t know how that reached the frontpage. A failed 15 minute experiment from someone who has no clue what they are doing
To be fair, a lot of resumes say "I created X" and "I developed Y" when, if you look even a little past the surface, the claim boils down to "Made a thin wrapper around one or more existing services." I interviewed a guy who was the sole developer of a web app that converted image files from one format to another. OK, cool project, bro, let's look at it. A little probing revealed it was just a small generic page that…
I have been thinking about this. I’d like to index the blogosphere and some knowledge base sites and stuff and exclude most of the Top1000 websites. I imagine I could do it on consumer hardware for less than $10-20k. Perhaps common crawl has done much of the heavy lifting already and I just have an indexing task.
The sales pitch is simple, you download the crawler, point it at your own blog index it and build an index of pages that your blog links to. Then index the pages those pages link to. etc You simply crank up the depth whenever you like it.
If you are a half decent blogger you have articles that link to most of the important websites that fit the subject of your blog.
You put a search box/page on your website that connects to your desktop client, your visitors can search with options:
- articles on this blog,
- related pages you've linked to,
- 1-5 depth to broaden the topical search (but less related articles)
- search other instances
It scales so well because searching your own blog is the most important, linked pages is pretty nice to have, deeper crawls are still useful but much less important and searching other instances, the anti climax if you like, is great but the least important.
The most crappy hardware can do 50 000 per day, if you run it slowly in the background [say] 100 pages on average per day is still 36 500 every year.
More usual is to be excited about the new found tool and run it for a few hours the first day. You are initially shocked how useful it is. The next day you crawl a few more pages until you get bored with it. You look again after a while and do one more good crawl. Few years later and you have an oddly large index.
You might want to run it automatically when your rss updates.
If you use it once in a while it is easy to ban some instances full of spam.
YaCy checks all results returned by other nodes by fetching the html and looking for the keywords on the page. This worked well. A very stale index may reflect poorly on the node but it may also be full of material that is important to you.
You would get crusty results at times but this is a feature not a bug. There is no man behind the curtain who is the big decider what you may and may not look at.
If your client is not running the search box/page on your blog only does p2p but it is likely able to still search your domain. What is a lot of posts for a blog is not a lot for a crawler. You can glue all kinds of products onto this. Besides a db YaCy keeps the full text of all pages crawled but only the text. If users want a feature that cant be done for free you can sell it to them. If someone has a website that is hard to index they can customize their crawler themselves or pay to have it done.
If you want to throw money at it and have a blog search engine you can limit the results by things that have an rss or atom feed.