I would add not just rely on robots.txt - use no index and/or password protect.
Robots stops crawling not indexing
SEO for Web Engineers
41–50 of 57 posts
Re: SEO for Web Engineers
#42Don't waste your time reading this article, it just mentions random points about how to have a functional web app, plus in a perfect world SEO should not exist, it's the Search Engine's job to find the content and show the user the content he is looking for, not the developers. I shouldn't have to add "hacks" in my code for the Google Crawler to understand my site the same way users already do.
Get off your high hirse mate
yes comes across as a lazy developer and not in a good way.
Re: SEO for Web Engineers
#43there is no such thing as "SEO". Also, everything on the list has nothing to do with Search Engine Optimization. For example "SERVER STABILITY AND DOWNTIME". How come taking case of your server stability is SEO???
I'm writing a little more about that because it's sad to read an article like that that makes no sense. SEO doesn't exist because the ONLY way to optimize your website is to write good content and get quality backlinks. To be precise, 99% of "SEO" is backlinks and 1% is your HTML quality (in-page SEO). To give you an example, you can have the worst HTML, no internal linking, to Sitemap, no nothing, but have 10-20 qua…
Bollocks links are still important yes but get the basics wrong and you wont rank on page 1.
Re: SEO for Web Engineers
#44Why does the name 'Web Engineer' bother me in this context? It had taken me a while just getting used to software engineer, even having a degree. Software dev, especially application building is mostly combining pre-built libraries. A web engineer is someone I would think drafted and refined the protocols and technologies used by app devs. Anyone else have thoughts on this? Is engineer == dev, STEM == STE*M now? UX engineer, marketing/growth engineer?
Re: SEO for Web Engineers
#45Re: SEO for Web Engineers
#46Earlier quoted context omitted.
Both a domain and an IP can get blacklisted, however it's typically IPs blacklists that you find yourself dealing with if you're sending emails from your webserver(s). You can sidestep these issues by using something like Mailgun or Sendgrid as your delivery mechanism for server-generated emails (password resets, account registration confirmations, etc). And always set up your SPF records to include these services as…
> it's typically IPs blacklists that you find yourself dealing with if you're sending emails from your webserver(s). Why wouldn't they blacklist the domain if they know that IP is associated with your domain? Would the domain blacklist only happen after you've had several IP addresses blacklisted?
All I can do is speculate as I don't know the answer to either question conclusively.
So...to speculate: I know that spam email is coming from an IP; I don't necessarily know that the IP is officially associated with a given domain. Sure, I could do a dig to see if the A record matches the IP, but blacklisting an entire domain is a pretty draconian step. I need my blacklist to be accurate, with a minimum of false positives, or its market value diminishes. Adding domains based on IP association is going to be more likely to induce false positives and increase the administrative overhead involved with my blacklist, even if that overhead is mostly automated through "get me off this blacklist" forms and the like. /speculation
As an aside, I've noticed that if a cloud provider recycles IPs (Digital Ocean) the chances of pulling a blacklisted IP off the heap is pretty good. At this point, even if all the server is doing is sending the occasional password reset email to a handful of internal staffers, we run the email through Mailgun. It just isn't worth dealing with the hassle of trying to get an IP off a blacklist.
Re: SEO for Web Engineers
#47> A complex web page might call the server 150 times as it loads to completion.
That irked me a little. I'm assuming this includes fonts & images (not a bunch of ajax requests), but it still seems high.
Re: SEO for Web Engineers
#48Earlier quoted context omitted.
The spirit of the idea is that Google will see the 304 status and move onto the next page more quickly than if it received a 200 status and reconciled that version of the page with the version that was previously crawled.
Thanks for clarifying. I guess that makes sense from a crawl efficient/budget standpoint, and in helping preserve server resources. For context, I've only come across a HTTP 304 status once in 9 years of SEO and crawling websites on a daily basis. I've no first hand experience of them being deployed in this way at scale on a live website and so haven't seen any server log analysis that demonstrates the efficacy of th…
304 has a specific use case, which is that if the crawler says, I have version X of the page already, give me a 304 if that’s still current or a 200 with content otherwise.
With dynamically generated content you’re more likely to just see 200s, but I think Nginx sets Etags automatically on static content so it’s common to see 304s there.
I’m pretty surprised you haven’t seen it often, but I’d guess it’s more to do with whatever crawlers you’re using (they’d need to be caching content and headers), rather than the scarcity of the status code.
Re: SEO for Web Engineers
#49The million dollar question is how to get good rankings for good keywords without having to spend too much money. This article does a good job describing how to keep a successful site optimized but actually getting the rankings is not covered much. Never, ever, let marketers send newsletters and promotional e-mails from the same IPs that the websites are hosted on. A rogue employee who violates the CAN-SPAM act may r…
I don’t understand, a mail server sends an email and it has an ip address. There’s no website associated with that. Sure, there’s an email address, and a load of rules about how to help avoid spam by setting up SPF and DKIM and DMARC to prove that the mail server is allowed to send on behalf of the domain. And if the specific IP address of the mail server ends up on a spam list, you can try to get it removed or start sending from another ip.
I don’t get what it means for the entire website to be blocked from an SEO perspective. Google don’t remove you from the index because an IP address ends up on a spam list and that’s also where you website is hosted, right?
Also, you’re only hosting on s3 if you’re hosting a totally static website, which very few people do.
Re: SEO for Web Engineers
#50Don't waste your time reading this article, it just mentions random points about how to have a functional web app, plus in a perfect world SEO should not exist, it's the Search Engine's job to find the content and show the user the content he is looking for, not the developers. I shouldn't have to add "hacks" in my code for the Google Crawler to understand my site the same way users already do.
Eh, not sure I'd say SEO shouldn't exist. At least, the techniques used for it are all things that should be the expected default on every site, like having a logical HTML structure with heading tags, having a useful unique title for every page, having a description shown in listings, logically linking to relevant content, rendering the page on the server rather than the client, etc. Maybe black hat/spammy SEO should…
Agreed. A lot of the "good" stuff you can do for SEO nowadays is pretty close to "stuff you should be doing anyway" for accessibility, usability, etc.