Search engines and SEO spam • Hacker News Board

Re: Search engines and SEO spam

#141

Post by nickff » Mon, Jan 03, 2022, 5:19 PM UTC

the_other wrote: post #104

coldpie wrote: post #94

Earlier quoted context omitted.

You just have to look at Google's profit motive here. Their motive isn't to provide quality search results. Their motive is to show users ads, either in the search results themselves or on the destination sites via their ad network. The SEO spam sites aren't a bug, they are a feature of Google's profit algorithm. Google's search quality will never improve so long as their motivation is to show you ads. Why should it?…

The core problem, I guess, is that search engines view all their results as ads. That’s why they got into the ad business in the first place.

>" The core problem, I guess, is that search engines view all their results as ads. That’s why they got into the ad business in the first place. "

This seems a bit overly cynical. Some search engines only served ads, but they're long gone. The survivors are those who dedicated themselves to finding links which were responsive to people's search intent. They seem to have gotten into ads because it was the best business model in this market.

Re: Search engines and SEO spam

#142

Post by magicalist » Mon, Jan 03, 2022, 5:19 PM UTC

hardtke wrote: post #105

birken wrote: post #5

The funny thing is that if the people who worked on spam at Google were free to talk about it, I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence. It's a ridiculously hard problem, especially when people are targeting you directly. But they aren't free to talk about it, because if they did it would just give more assistance to the spammers, and make th…

The curated search results business model doesn't work. Google gives "aggregators" and other search engines the death sentence for organic search traffic from economically meaningful queries, so you'd get no free traffic. This is one of the major antitrust complaints against Google in the EU. Since you get no organic search traffic, you need to build a brand using advertising, and once you start down that road you ne…

> This is one of the major antitrust complaints against Google in the EU.

The complaints I've read are from exactly the kind of generated content farms people are complaining about in this thread.

Re: Search engines and SEO spam

#143

Post by roughly » Mon, Jan 03, 2022, 5:19 PM UTC

endisneigh wrote: post #126

numbsafari wrote: post #117

Earlier quoted context omitted.

The problem isn't that Google doesn't employ these people or invest in their activities. It's that Google has destroyed their own search results in order to continue to expand their revenue opportunities. If Google: - Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT) - Allowed you to block certain domains from your search results, like YT videos. (If they added some kind of "coo…

> That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable. Why would they do this? Google's customers are the advertisers, not the end-users. And no one is going to pay for a search engine, it's been tried and has failed.

> And no one is going to pay for a search engine, it's been tried and has failed.

Always curious about things like this. I certainly would pay for this; it sounds like many other people here as well would. I'm curious if the constraint is that there aren't enough people to actually pay for the investment required for the service, or if there aren't enough people willing to pay to meet the standard VC notions of success. We seem to have a problem with building and supplying services for niche (read: "not expressable as an integer percent of the world's population") customer bases, and I'm never sure if that's a business problem or a cultural problem.

Re: Search engines and SEO spam

#144

Post by loceng » Mon, Jan 03, 2022, 5:19 PM UTC

jrockway wrote: post #91

To some extent, I worry that the problem with search engines is that there isn't any data worth returning. Yesterday's thread talked a lot about reviews. Writing a review is hard work that requires deep domain expertise, experience with similar products, and months of testing. If you want a review for something that came out today, there is no way that work could have been done, so there simply isn't anything to find…

Developing common standards/protocols for everything required for a quality review vs. a "candy" or shallow hype review would be a good place to start, making it culture that everyone educated knows about to follow - and then they can only go to or support reviewers who list what testing protocols they follow.

Industry has already done this with the "food pyramid" - influencing, capturing governments to make the food pyramid more based on economic reasons and much less on science - with the government putting it out and distributing it into schooling of different levels, giving it an unearned or undeserved authority which then people blindly trust/follow - not understanding that or when systems and their output or oversight have been captured; why the pandemic bringing the classroom home via Zoom, so parents could see/hear the learning material has outraged many parents - an example I've heard, where white children are being taught to feel guilty about their 'white privilege', or parents being upset their children are being taught at a very young age that they can decide what gender they are; I'm not stating what I believe here, just giving examples I've heard of.

This capturing of the government is why I think ultimately the government should be developing and maintaining such platforms, as per law, and requiring individuals and organizations to in real-time add and update their data (simply example being restaurants, their menu's ingredients, their open hours) - in part to de-risk the government having an unnatural power as "the single arbiter" of truth, perhaps instead to de-risk capture that the government funds multiple independent organizations at the federal level - that States can decide which ones they follow, if necessary, part of why States exist - to de-risk the potential capture of the Federal umbrella; however the system is in an imbalanced, broken, captured state - with the duopoly evolving to be more extreme lead or formed by the establishment, with a broken voting system in arguably most countries of the world, and mainstream media being captured by for-profit industrial complexes that fund MSM through ad revenues - which further develop or mould our culture and narratives/talking points and beliefs, whether truthful or not; without fixing these the other platforms/systems excelling won't be possible.

Re: Search engines and SEO spam

#145

Post by frenchyatwork » Mon, Jan 03, 2022, 5:20 PM UTC

jrockway wrote: post #91

To some extent, I worry that the problem with search engines is that there isn't any data worth returning. Yesterday's thread talked a lot about reviews. Writing a review is hard work that requires deep domain expertise, experience with similar products, and months of testing. If you want a review for something that came out today, there is no way that work could have been done, so there simply isn't anything to find…

I think one of the fundamental things that make search work well about 1-2 decades ago was that web sites would link to each other, and that those links could vaguely correlate with reputation. There were link spammers, but there was actually a some decent organic content as well.

What's happened since then is that almost all the normal "people linking to things they like" has gone behind walled gardens (chiefly Facebook), and vast majority of what remains on the open web are SEO spammers.

Re: Search engines and SEO spam

#146

Post by wakiza33 » Mon, Jan 03, 2022, 5:21 PM UTC

awillen wrote: post #31

judge2020 wrote: post #15

Earlier quoted context omitted.

Also, i'd be very surprised if they didn't have tens of thousands of workers aiding in spam review already. The hard part in all of this isn't finding and stopping spam - it's defining what spam is. Are all the pie recipes where there's a 2000 word essay about their grandma at the top 'spam'? They still have the recipe, and Google Home devices pick up the recipe instructions just fine so people end up not reading it,…

AFAIK the 2000 word essays in recipes are Google's fault - it prioritizes pages with a lot of content, so you have to add that junk to the top in order to rank highly. While I'm sure there's more going on behind the scenes than I'm aware of, it does seem like the rules could be altered on a category-specific basis where a lot of text isn't necessarily a positive.

prioritizes is correct, but in some ways it's not the best descriptor.

Google's algos, while advanced, still rely a ton on text to actually tell what the page is about. They need it.

If they just relied on other factors (title, links, website, etc.) they would end up with worse results for users. Im sure they've tested it.

Google's core algo in a lot of ways is much simpler than people think (in other ways of course it's very complex).

Re: Search engines and SEO spam

#147

Post by freediver » Mon, Jan 03, 2022, 5:21 PM UTC

Google’s job is to serve its customers, and it does that really, really well.

The problems being discussed today (and yesterday in the similar thread) come from the fact that for Google user != customer.

When you have incentives that are misaligned like this, you can only go so far! We seem to have reached that point with Google, where there is not much more that can be done on the search experience front without jeopardizing customer experience (ad revenue).

Disclosure: I’m working on a paid search engine to solve this problem on a fundamental level, by aligning the incentives and making user also the customer so we can best serve them and their needs. It is called Kagi and is currently in closed beta accepting beta-testers.

https://kagi.com

Re: Search engines and SEO spam

#148

Post by stoicjumbotron » Mon, Jan 03, 2022, 5:23 PM UTC

birken wrote: post #5

The funny thing is that if the people who worked on spam at Google were free to talk about it, I'm sure it would become evident that they know more about spam and anti-spam efforts than anybody else in existence. It's a ridiculously hard problem, especially when people are targeting you directly. But they aren't free to talk about it, because if they did it would just give more assistance to the spammers, and make th…

Highly OT, but if a technical person (not at a managerial level) involved in tackling spam at Google were to leave the team, are they allowed to work on the similar problem space at a different company?

Re: Search engines and SEO spam

#149

Post by neoneye2 » Mon, Jan 03, 2022, 5:23 PM UTC

aronpye wrote: post #51

A lot of the spam results just seem to be copy pasted content. I wonder how difficult it is to compare the main body of text in search results, then say if it is over a 95% match with another site (I.e. it has been copy-pasted), demote it in the search results. If a site generates too many of these demotions then it gets blacklisted from the index.

I have experimented using LSH (Locality Sensitive Hashing) for identifying similar documents, among 50k documents in total.

My LSH implementation is here: https://github.com/loda-lang/loda-rust/blob/develop/script/t...

Example of the 100 most similar documents: https://github.com/neoneye/loda-identify-similar-programs/bl...

There can be false positives, so after LSH then do a more in-depth comparison.

Re: Search engines and SEO spam

#150

Post by netcan » Mon, Jan 03, 2022, 5:23 PM UTC

I'm almost certain that pg gave "compete with Google by competing in some niche" advice 10+ years ago.

In any case, I'm not sure that competing in search is a very attractive notion. AdWords is the only meaningfully profitable search and business. Even if you steal 10% of Google's market, that absolutely doesn't translate into 10% of the revenue.

That said, recipes. Someone make a search engine where the top results don't start with 500 words on the history & etymology of butter, because that's what Google want.