Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Almost all searches on my independent search engine are now from SEO spam bots

blog.searchmysite.net

Post reply on HN
Print view View original

231–240 of 391 posts

Re: Almost all searches on my independent search engine are now from SEO spam bots

#231

Post by soheil »

m-i-l wrote: post #199
Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

Hi, "this guy" here:-) If people come to a site but don't come back then it is reasonable to conclude that "it's a bad website", but as the blog entry put it "without any real users in the first place it is hard to gauge whether people like it or not". Note also that it isn't intended to be a general purpose search engine, but a niche search engine to try and find some of the fun and interesting content, e.g. relatin…

How exactly is a "general purpose search engine" different than a "search engine to try and find some of the fun and interesting content"?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#232

Post by Pelam »

Gigachad wrote: post #34
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

Interesting thought. I just went though my browser history and realised that almost every time I use google search, I already know what website I want, I just don’t know the exact link/page. I’ll use google because the search on stack overflow or reddit sucks but I know I’m looking for a page on one particular site.

I realized this too. I disabled search from address bar and started bookmarking everything even remotely sane I see. I often add a few personal keywords to the bookmark bar.

It is starting to pay dividends. Instead of weird stuff thrown up by google when I type in something, I get the "oh yeah, that was the page" from a short list of bookmarks shown to match the words.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#233

Post by MockObject »

stackbutterflow wrote: post #142
FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

I think you're generalizing your own behavior. I regularly use google to search for topics that cross my mind and I end up on many websites that are not one the giants in your list. It's a fun activity. If people stick to the same 10 websites that's on them. Nothing prevents you from exploring the web.

> Nothing prevents you from exploring the web.

What prevents you from exploring the web is you can't find but the same 10 sites through search engines.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#234

Post by melenaboija »

I am a total ignorant about search engines and I have a question after seeing all types of comments and projects popping up lately and criticizing Google results which is if it is realistic to think that something similar to Google could exist.

It seems to me that there are all sort of tools out there to do so such as all the public NLP implementations, vector search engines, ... and I wonder if it is that not everything that is needed is truly available, it is a matter of the needed resources to have something working or is just a matter of the products already existing and not getting traction (and I am not talking about the other big search engines).

Re: Almost all searches on my independent search engine are now from SEO spam bots

#235

Post by ouid »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

Google is still pretty good at searching reddit. Maybe reddit can acquire them.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#236

Post by CWuestefeld »

nl wrote: post #127
baxtr wrote: post #114

Earlier quoted context omitted.

I am not so sure... I think what happened is this: the WWW was everything back in the days. But in the "old days," only 10% of all people were online, the web elite. Then, AOL came, and the rest came online slowly but surely. The so-called "mainstream" people were no geeks, and these people were "just" ordinary people. Almost all were captured by what you call "big websites". Now, we see the 100% being dominated by t…

Eternal September was Sep 1993. AOL hit the internet in March 1994. Netscape didn't launch until December 1994 (and the WWW was nothing before that. I subscribed to a mailing list with new sites that were released and I'd visit most new websites on the internet on most days with the Cello browser in my uni labs most days). AOL users have been there since the beginning of the WWW. https://en.m.wikipedia.org/wiki/Etern…

My recollection is that the AOL event you reference was only making usenet accessible - a point that makes good sense in the context of the eternal September.

But when talking about the WWW, that's a very different story. I think that AOL didn't incorporate a web browser until quite some time after that.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#237

Post by derefr »

FargaColora wrote: post #27

You mention the "Dead Internet Theory" (not heard that phrase before!). I agree: the WWW Internet is dead, that is your problem. No-one visits websites anymore, everyone has moved to the 10 biggest websites and all data is now siloed there. If I want to search for something topical and relevant, I go to Facebook, Twitter, Reddit, HackerNews, Instagram, Google Maps, Discord etc. The general Internet is dead: it's just…

> Make a search engine which indexes the fresh relevant data from the big siloed websites, and ignores the general dead Internet

I don’t understand why Google themselves don’t do this. LinkedIn v. hiQ demonstrated that they won’t get in trouble for scraping users’ subjective views of data within these silos and then stitching them together to form a cohesive whole. So where’s the effort to do so? It seems like the obvious step.

Re: Almost all searches on my independent search engine are now from SEO spam bots

#238

Post by arbuge »

It is my experience that SEO bots are increasingly ignoring robots.txt entries disallowing them from crawling our sites. Last week we noticed several doing this. I don't mind naming names - semrush, something called grapeshot crawler, something else called blex bot, and moz dotbot. Anyone else having the same experience?

Re: Almost all searches on my independent search engine are now from SEO spam bots

#240

Post by [deleted] »

Teandw wrote: post #62

This guy throws multiple reasons/conspiracies out there on why the website is really struggling to gain literally any sort of traction. Web is all bots, search engines not promoting competitors and being drowned out by SEO spam, yet he's failing to see the most obvious reason... the reason nearly all websites don't gain traction... Because it's a bad website. It provides no value to the user. I put in a few search te…

[deleted]
Post reply on HN