31–40 of 93 posts
Start with a robots.txt? https://www.codeium.com/robots.txt Update: I also see that you serve your site off GitHub... so that might be part of it as well? Bing might not index those sites?
You can go through the procedure of registering with their webmaster tools equivalent, submit a ticket and wait.
Or you can simply just wait.
In my limited experience (6 sites) the result was the same. Eventually it will resolve itself and you'll be back in the index, albeit with some very weird links showing up (an effect of the original attack). This does, at least, alert you to some vulnerabilities you may have previously been unaware of.
Start with a robots.txt? https://www.codeium.com/robots.txt Update: I also see that you serve your site off GitHub... so that might be part of it as well? Bing might not index those sites?
Why would Bing require a robots.txt when all the other major search engines are fine with a sitemap.xml? Or are these separate somehow?
# Allowing all web crawlers access to all content
User-agent: *
Disallow:
At least you'll be able to look in your logs and see if BingBot is grabbing the file or not.
There's loads of stuff that Bing doesn't index very well. If you turn of JS there's almost no actual text on the site; only the FAQ has any serious amount of text. This probably doesn't help. Having the page open also really slows down my laptop and constantly uses ~50%-75% CPU and ~130M memory. No idea if that's a factor, but it takes quite a lot of computing resources to actually make the homepage render any text a…
4chan also does not show on Bing for me, but it does for some people it seems.
honestly if you look at the incident history for github, coupled with the ham-fisted tantrum Bing throws if you ask for Chrome ("There's no need to download a new web browser.") , it feels pretty on-point to say Redmond is throwing knuckle sandwiches to stay in the running for any relevance on the internet these days https://www.githubstatus.com/history
Start with a robots.txt? https://www.codeium.com/robots.txt Update: I also see that you serve your site off GitHub... so that might be part of it as well? Bing might not index those sites?
Why would Bing require a robots.txt when all the other major search engines are fine with a sitemap.xml? Or are these separate somehow?
For all the hate Google gets, it's crawlers are like 10x smarter than any other search engine. Wouldn't surprise me at all if there was some type of crawling issue that Google figured out and Bing didn't. This is a case where I'm not ready to attribute to malice what is very likely incompetence.
Operating 8.8.8.8 likely gives Google a near-complete list of every active domain on the internet, and new domains show up there the second any 8.8.8.8 user queries it - which is likely well before anyone links to it. I have no idea if Google use this data for hints to their crawler and indexer, but if I worked there I sure would.
At the time, there was no "Safety Report" to indicate why Bing thought it was dangerous. The report page linked to https://www.bing.com/toolbox/bing-site-safety?url=https%3a%2... and it said "That web page doesn't exist"
To fix it, we had to register with "Bing Webmaster Tools" (https://www.bing.com/webmasters/about) and raise a support ticket.
Within a few days, the issue "resolved itself". It's possible that raising a ticket forced some automatic refresh of the indexed data for the domain.
