Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

281–290 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#281

Post by marginalia_nu »

lukev wrote: post #195
Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

I second this. Don't get me wrong, I applaud the concept and the effort, but this implementation isn't quite there. I searched for "document management system comparison" since I am currently in the process of selecting one for our legal team at work. Some on-the-ground reports from real users would be hugely valuable. But this is the classic example of where Google utterly fails; document management is a 100 billion…

Is a comparison of document management systems something you expect actually find, as something written by humans? I wouldn't write such an article, I don't know who would.

The only people who seem to be writing these types of comparison articles are spammers.

I typed this reply without checking, but I checked now, and yeah -- if you google "document management system comparison", you get ads for document management systems, and search engine spam. That's hardly helpful.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#282

Post by haswell »

m-i-l wrote: post #267
soheil wrote: post #231

Earlier quoted context omitted.

How exactly is a "general purpose search engine" different than a "search engine to try and find some of the fun and interesting content"?

The general purpose search engines search the whole internet, and as a result claim that you can search for anything on the whole internet, even going beyond that to answer questions which aren't on the internet as such, e.g. "What is my IP?" and "What time is it?". However, niche search engines only search specific parts of the internet, and only claim to be able to deliver results relating to their specific topic,…

> general purpose search engines search the whole internet, and as a result claim that you can search for anything on the whole internet, even going beyond that to answer questions which aren't on the internet as such, e.g. "What is my IP?"

I think there are two distinct things here:

1) Searching the whole internet

2) Returning results that aren't necessarily from the Internet, but instead are convenience features of the engine

I understand that you're not trying to replicate things like "What's the weather today", but when I want results about , how can you return meaningful results without searching the whole Internet?

Put another way, if you don't search the whole Internet, the results are going to be limited to only the curated list of sources you do search. This can be useful in its own way - i.e. if you are positioning this as "search this list of curated sources", but also means the site will only be as useful as the curation you provide.

For example, I dabble with Software Defined Radio. If I search your site for "rtlsdr", a very popular package, I get three results. Those results are somewhat interesting, but I know there's a whole world of content out there related to rtlsdr that I'm not seeing here.

So adding a bit to what the parent commenter was saying - if I'm using your site to look for my particular niche, and I only see three results when I know there are many more, I'm not likely to continue using your site to search for rtlsdr.

It then leads me to wonder what I can search for, or if there's much utility to searching at all.

Please take these comments in the spirit they are intended - I think a search engine that helps find things on the "old" web, or just helps me cut through all of the SEO optimized crap is a great idea. It's something I want to use. But I can also understand why someone might try a search and move on.

Just an idea, but maybe providing a way for independent creators to submit their site for indexing (or for an interested user like me to submit a site) would help increase your reach.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#283

Post by ColinHayhurst »

Domenic_S wrote: post #216
ColinHayhurst wrote: post #137

Earlier quoted context omitted.

Agreed. > If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just legacy content and spam. The "general" Internet is not dead. Though if you just want to participate in just Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord you might well think that. Users of marginalia (aut…

> The "general" Internet is not dead. For some things it is. Good luck getting a non-sponsored/SEO-gamed review of a kitchen appliance or particular vacation mode such as a cruise. It's flabbergasting. Most times I just stick "inurl:reddit.com" in my search and try to get discussion threads about the thing I'm researching, but even that's getting filled up with shills.

Result #1 & #2 for kitchen appliance review (your personalised/local results might vary):

Google:

https://www.expertreviews.co.uk/home-garden/home-appliances

https://www.goodhousekeeping.com/appliances/

Bing:

https://www.which.co.uk/reviews/fitted-kitchens/article/plan...

https://www.goodhousekeeping.com/appliances/

DDG:

https://www.goodhousekeeping.com/appliances/

https://www.which.co.uk/reviews/fitted-kitchens/article/plan...

Marginalia:

https://www.infiniteeureka.com/shop-markdowns-on-small-kitch...

http://www.fullyramblomatic.com/essays/sarah.htm

Mojeek:

https://www.appliancesreviewed.net/

https://busybakers.co.uk/category/kitchen-appliance-reviews/

Re: Almost all searches on my independent search engine are now from SEO spam bots

#284

Post by not2b »

The conclusion isn't that there's nobody out there, but that the billion-odd people who use search engines every day have no idea what searchmysite.net is. They use Google, often without even knowing it because they just type some words into their browser and take what they get.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#285

Post by soheil »

m-i-l wrote: post #267
soheil wrote: post #231

Earlier quoted context omitted.

How exactly is a "general purpose search engine" different than a "search engine to try and find some of the fun and interesting content"?

The general purpose search engines search the whole internet, and as a result claim that you can search for anything on the whole internet, even going beyond that to answer questions which aren't on the internet as such, e.g. "What is my IP?" and "What time is it?". However, niche search engines only search specific parts of the internet, and only claim to be able to deliver results relating to their specific topic,…

Ok, but answering questions like "what time is it?" doesn't subtract from the usefulness of a search engine. Seems like you're saying it makes your search engine better somehow because it can't do the above.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#286

Post by pde3 »

nonrandomstring wrote: post #226
dreen wrote: post #68

Earlier quoted context omitted.

I think this was inevitable all along, something similar happened to radio if I'm not mistaken. However, the good news is that we will never stop reinventing everything. The real value of the old internet was showing us what is possible.

> The real value of the old internet was showing us what is possible. Of equal value is that it showed us what not to do. We have 30 years of documentation for research on exactly what a successful intra-planetary network needs to be immune to. A successful future network must build-in resistance all forms of human pyschopathology from the ground up.

This is a nice fantasy, but it's a fantasy. The tech stack and network we have is too dense a forest to be replaced by clean slate designs. But maybe some of the problems could be improved with some new platforms and APIs. Mind you, ML is making so much progress so quickly that what happened over the last thirty years is at best a partial model of the problem we have to solve now, and the tools we have to do it with...

Re: Almost all searches on my independent search engine are now from SEO spam bots

#287

Post by rightbyte »

Jleagle wrote: post #78
Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

I'm getting lots of `No results found for query = xxx.`

That sounds like a feature actually, being honest about no hits.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#288

Post by oneeyedpigeon »

marginalia_nu wrote: post #281
lukev wrote: post #195

Earlier quoted context omitted.

I second this. Don't get me wrong, I applaud the concept and the effort, but this implementation isn't quite there. I searched for "document management system comparison" since I am currently in the process of selecting one for our legal team at work. Some on-the-ground reports from real users would be hugely valuable. But this is the classic example of where Google utterly fails; document management is a 100 billion…

Is a comparison of document management systems something you expect actually find, as something written by humans? I wouldn't write such an article, I don't know who would. The only people who seem to be writing these types of comparison articles are spammers. I typed this reply without checking, but I checked now, and yeah -- if you google "document management system comparison", you get ads for document management…

2nd result I got from that exact search is an article from techradar:

https://www.techradar.com/uk/best/best-document-management-s...

Do you consider that search engine spam?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#289

Post by _tom_ »

I think many people in the comments here, and most users, are missing that you index a SMALL subset of the web. This leads to people running a default test search, finding no results, and concluding your search engine is bad, and leaving.

While you imply that in the search page, obviously it's not clear enough.

Maybe add "this search engine only searches a small set of user submitted sites. Click for the list. Or to add your site."

Re: Almost all searches on my independent search engine are now from SEO spam bots

#290

Post by ipaddr »

norman784 wrote: post #269
weird-eye-issue wrote: post #157

Earlier quoted context omitted.

Ironically the Kagi search engine is not in the first few results in Google when you search Kagi (at least in Thailand) And when I did make it to the site, it looks like I have to sign up to use it? I'm not sure putting a locked gate in front of a search engine in 2022 makes sense but okay

The whole concept of kagi is to be a paid service (is still in beta and for now it's free AFAIK), so you pay money instead of having ads or the search engine selling your data, use the service that suits best to your purposes and philosophy.

The concept in 2022 sounds doomed to fail on many fronts. A service that claims to offer privacy but requires identifying payment information. A required email signup so followup sales emails can happen when the service is ready.

Ddg was popular on here until they censored certain websites. Does this search service censor?

Sounds like they are trying to tackle privacy but in reality users of this service will have less privacy.

Post reply on HN