Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kogan.com mysteriously disappears from Microsoft Search Engines

kogan.com

Post reply on HN
Print view View original

41–50 of 51 posts

Re: Kogan.com mysteriously disappears from Microsoft Search Engines

#41

Post by blauwbilgorgel »

UnoriginalGuy wrote: post #38
blauwbilgorgel wrote: post #31

They serve their robots.txt as a "application/octet-stream" not "text/plain". Might be confusing some search engines? Also according to Internet Archive their robots.txt has changed in the last months. This was a version that was archived: http://web.archive.org/web/20080104044407/http://www.kogan.c... At a glance, they are whitelisting some bots, including Googlebot, but not including Bingbot , and disallowing the r…

Your link is pointing at 2008. That is four years ago. Hardly proof of anything. You are right about the content type however.

Correct. It doesn't proof much, but that they previously had a buggy robots.txt.

Usually these posts are accompanied by a declining traffic graph for Bing referrers. That would give us more proof and a timeframe to work with.

The post gives two options for their absence in Bing results: a fault at Microsoft or as a punishment. With their history I think a fault on their own part is a more likely option. I can't rule that out.

Re: Kogan.com mysteriously disappears from Microsoft Search Engines

#43

Post by sreyaNotfilc »

This "tax" is elitist internet bullshit.

I'm all for using any browser that you'd want, but why go through elaborate lengths to put Microsoft down? Especially on a version of IE that's 3 iterations in the past.

I understand that IE7 may not be as great as the other browsers. I'm fine with being annoyed by that. But what about the other versions of IE? Why isn't there a link for IE9 or IE10?

Re: Kogan.com mysteriously disappears from Microsoft Search Engines

#44

Post by sreyaNotfilc »

rplnt wrote: post #16
mtgx wrote: post #7

I think Microsoft themselves admitted that they do a great deal of manual editing for their search engine rankings.

Where? By quick search I did not find anything.

I think mtgx is referring to where Google claimed that Microsoft stole their search results...

http://googleblog.blogspot.com/2011/02/microsofts-bing-uses-...

Re: Kogan.com mysteriously disappears from Microsoft Search Engines

#46

Post by reinhardt »

michaelt wrote: post #32
chrislomax wrote: post #27

Anyone else thinking they added a robots.txt entry for bing to disallow, waited to get de-listed then removed the offending robots.txt entry, posted this news article knowing bing would pick them back up in a couple of days?

The HTTP last modified header for www.kogan.com/robots.txt has Last-Modified: Wed, 06 Jun 2012 06:14:35 GMT It's possible they did this and they were cunning enough to turn back the last-modified clock.

How did you get that? Curl says May 4 instead:

  $ curl -I http://www.kogan.com/robots.txt
  HTTP/1.1 200 OK
  Server: nginx/1.0.14
  Date: Mon, 09 Jul 2012 18:48:34 GMT
  Content-Type: application/octet-stream
  Content-Length: 79
  Last-Modified: Fri, 04 May 2012 18:58:02 GMT
  Connection: keep-alive
  Accept-Ranges: bytes

Re: Kogan.com mysteriously disappears from Microsoft Search Engines

#47

Post by michaelt »

reinhardt wrote: post #46
michaelt wrote: post #32

Earlier quoted context omitted.

The HTTP last modified header for www.kogan.com/robots.txt has Last-Modified: Wed, 06 Jun 2012 06:14:35 GMT It's possible they did this and they were cunning enough to turn back the last-modified clock.

How did you get that? Curl says May 4 instead: $ curl -I http://www.kogan.com/robots.txt HTTP/1.1 200 OK Server: nginx/1.0.14 Date: Mon, 09 Jul 2012 18:48:34 GMT Content-Type: application/octet-stream Content-Length: 79 Last-Modified: Fri, 04 May 2012 18:58:02 GMT Connection: keep-alive Accept-Ranges: bytes

I get Last-Modified: Fri, 04 May 2012 18:58:02 GMT from my home machine (and you obviously do too) where I connect to 74.118.68.224 - but I got Wed, 06 Jun 2012 06:14:37 GMT from work (and via http://web-sniffer.net/ and http://www.rexswain.com/cgi-bin/httpview.cgi) which was looking at 108.166.39.66. Perhaps we get DNS load balanced or something similar?

Re: Kogan.com mysteriously disappears from Microsoft Search Engines

#48

Post by X-Istence »

UnoriginalGuy wrote: post #35

Their robots.txt file is being delivered as the wrong content type for me. It is being served as a "application/octet-stream" instead of a "text/html" I have no idea what consequences this might have since I've never seen another site which does it. Some quick Googling turned up nothing. Maybe Bing choked on the wrong content type and decided to blackball their entire domain in order to avoid indexing the unindexable…

Why would you want to serve up a robot.txt as "text/html" when in reality it is "text/plain"?

Re: Kogan.com mysteriously disappears from Microsoft Search Engines

#49

Post by Yver »

blauwbilgorgel wrote: post #41
UnoriginalGuy wrote: post #38

Earlier quoted context omitted.

Your link is pointing at 2008. That is four years ago. Hardly proof of anything. You are right about the content type however.

Correct. It doesn't proof much, but that they previously had a buggy robots.txt. Usually these posts are accompanied by a declining traffic graph for Bing referrers. That would give us more proof and a timeframe to work with. The post gives two options for their absence in Bing results: a fault at Microsoft or as a punishment. With their history I think a fault on their own part is a more likely option. I can't rule…

You are arguing in bad faith. The robots.txt from 2008 is a whitelist of 190+ known crawlers. It doesn't include Bing, which was launched in 2009.

In 2008, kogan.com was a parked domain. It has nothing to do with the current owner(s).

Re: Kogan.com mysteriously disappears from Microsoft Search Engines

#50

Post by whatusername »

sreyaNotfilc wrote: post #43

This "tax" is elitist internet bullshit. I'm all for using any browser that you'd want, but why go through elaborate lengths to put Microsoft down? Especially on a version of IE that's 3 iterations in the past. I understand that IE7 may not be as great as the other browsers. I'm fine with being annoyed by that. But what about the other versions of IE? Why isn't there a link for IE9 or IE10?

This "tax" is internet marketing linkbait. Pure and Simple.
Post reply on HN