Live data from Hacker News

Google App Engine Silently Stopped Sending Email 5 Weeks Ago

code.google.com

71–80 of 119 posts

Re: Google App Engine Silently Stopped Sending Email 5 Weeks Ago

#71

Google Cloud Support here. To correct the misleading title, this is not a generalized outage of any kind, but an issue with some applications sending mail that trigger spam/malware/etc filters. Since spammers generally do not comply with RFC 3514, filtering is unfortunately not an exact science, so false positives happen and we're working with the customers in question on a case-by-case basis. Also, App Engine is int…

5 weeks ago your infrastructure stopped sending out mail. Our apps didn't change, your infrastructure did. Whether it's your outgoing spam filter that's overzealous, or if a datacenter is on fire is irrelevant. Your infrastructure stopped doing what your docs said it would. Paying customers like myself have apps running on your infrastructure. Your infrastructure stopped delivering the exact same emails that it had b…

Why do you think services like sendgrid, mailgun etc exist? It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area.

At the VERY least you could have easily set up a gmail account and sent them from smtp. Choose the right tool for the job.

Re: Google App Engine Silently Stopped Sending Email 5 Weeks Ago

#72

Hey Folks, The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a…

Long time App Engine user here. If you're sending email as part of your app, I'd recommend using Mailgun (or similar services). You can set up DKIM, inbound routes, etc, and generally have more control over how your mail is processed. I never got the impression App Engine really wanted to be in the email delivery business, so if that's important to you, there are tools that are focused on that (and free up to a monthly quota). My $0.02.

Re: Google App Engine Silently Stopped Sending Email 5 Weeks Ago

#73

Google Cloud Support here. To correct the misleading title, this is not a generalized outage of any kind, but an issue with some applications sending mail that trigger spam/malware/etc filters. Since spammers generally do not comply with RFC 3514, filtering is unfortunately not an exact science, so false positives happen and we're working with the customers in question on a case-by-case basis. Also, App Engine is int…

5 weeks ago your infrastructure stopped sending out mail. Our apps didn't change, your infrastructure did. Whether it's your outgoing spam filter that's overzealous, or if a datacenter is on fire is irrelevant. Your infrastructure stopped doing what your docs said it would. Paying customers like myself have apps running on your infrastructure. Your infrastructure stopped delivering the exact same emails that it had b…

[deleted]

Re: Google App Engine Silently Stopped Sending Email 5 Weeks Ago

#74
post #71

Earlier quoted context omitted.

5 weeks ago your infrastructure stopped sending out mail. Our apps didn't change, your infrastructure did. Whether it's your outgoing spam filter that's overzealous, or if a datacenter is on fire is irrelevant. Your infrastructure stopped doing what your docs said it would. Paying customers like myself have apps running on your infrastructure. Your infrastructure stopped delivering the exact same emails that it had b…

Why do you think services like sendgrid, mailgun etc exist? It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area. At the VERY least you could have easily set up a gmail account and sent them from smtp. Choose the right tool for the job.

> It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area.

I partially agree with this - dedicated email services have a higher deliverability rate than a random web server, especially a cloud-based server that might be using an external IP that was previously used to send spam. However, I can understand the parent poster being annoyed that their emails were being delivered one day, and not the next, without any changes on their end.

> At the VERY least you could have easily set up a gmail account and sent them from smtp. Choose the right tool for the job.

This is absolutely the wrong tool for the job. You should not be using a Gmail account to send out automated emails that you care about.

Re: Google App Engine Silently Stopped Sending Email 5 Weeks Ago

#75

Hey Folks, The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a…

Possible reason that email service providers and other email sending services might see an increase in new signups and free tier use recently: http://blog.mandrill.com/important-changes-to-mandrill.html Mandrill in essence raised their price, eliminated free tier, and required prepurchased blocks of sending capacity IIRC

The numbers Mandrill released about that business suggest that they had a large number of low volume senders, who may now be looking for a new home.

Re: Google App Engine Silently Stopped Sending Email 5 Weeks Ago

#76
post #74
post #71

Earlier quoted context omitted.

Why do you think services like sendgrid, mailgun etc exist? It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area. At the VERY least you could have easily set up a gmail account and sent them from smtp. Choose the right tool for the job.

> It's pretty common knowledge that if you need to send mission critical email, you should use a service specifically designed for that. Sending email programmatically directly from a web server has ALWAYS been a gray area. I partially agree with this - dedicated email services have a higher deliverability rate than a random web server, especially a cloud-based server that might be using an external IP that was previ…

> This is absolutely the wrong tool for the job. You should not be using a Gmail account to send out automated emails that you care about.

Agreed, but better chance of delivery than directly from the server.

Re: Google App Engine Silently Stopped Sending Email 5 Weeks Ago

#77
post #67

Hey Folks, The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a…

Isn't there an API to find out why a message was SPAM filtered ? Or are you afraid that "rogue" applications will use it to produce messages that are SPAM but yet not trigger the SPAM filter ?

The recent post about Candy Japan's attempt to combat credit card fraud gives a good sense of what fighting fraud and abuse is like: https://news.ycombinator.com/item?id=11431881

As an email service provider it's like that and more, since (1) once an abuser uses your service, they've gotten the benefit immediately and keep it even if their account is discovered as fraudulent later, e.g. stolen CC number & chargeback. (2) Abusive users can directly harm good users such as by harming the deliverability of the overall platform. It's not just bad debt, it's bad experience too. (3) Unlike Candy Japan where fraudsters mostly just wanted to check CC numbers and not actually buy product, email abusers really want to send emails (4) It can be hard to tell good and bad senders apart because some companies with an internet presence aren't email savvy and might make mistakes or might get hacked.

Spam filters are always tough because if you give someone transparency into which actions of theirs that you consider abuse, then they will quickly detect and route around your attempt to block them. (See Candy Japan article) It's pretty easy for a human to guess what might be the sign of their fraud and run a few experiments to see what gets flagged e.g. By comparison a machine learning system might be hard to outsmart, but then it's also challenging to explain and troubleshoot false positives. Hence what's effective is often a combination of machine-learned filters and heuristics along with manual overrides by human judgment.

All other things equal, new users are a lot more likely to engage in fraud than existing ones, and so tend to be under more suspicion. Aside from B2B fraud where companies take out lines of credit and then go bankrupt intentionally, it's uncommon for existing established customers to turn fraudulent - they're already vetted. (Consider: who is more likely to be fraudulent. The first time subscriber to Candy Japan, or a subscriber who has been using it for 12 months and is about to buy their 13th month?) It's not a great experience as a new user to be under suspicion, but if it's temporary and easily overridden by a human it can be a decent trade-off - the need to reach out acts a deterrent to spammers but does not deter legitimate users as much (speaking generally).

Re: Google App Engine Silently Stopped Sending Email 5 Weeks Ago

#78

Hey Folks, The issue reported here is linked to App Engine and Gmail tightening up their spam filters. The root cause was an increase in organizations sharding out their spam systems to utilize App Engine’s free tier in such a way that is (a) in direct violation of our ToS and (b) making all of our lives suck a bit more (raise your hand if you want spam). It’s unfortunate that while App Engine is trying to provide a…

Long time App Engine user here. If you're sending email as part of your app, I'd recommend using Mailgun (or similar services). You can set up DKIM, inbound routes, etc, and generally have more control over how your mail is processed. I never got the impression App Engine really wanted to be in the email delivery business, so if that's important to you, there are tools that are focused on that (and free up to a month…

I completely agree with your recommendation on Mailgun. Using it for past 4 months, never had a problem with their service.

Re: Google App Engine Silently Stopped Sending Email 5 Weeks Ago

#80
post #31

Earlier quoted context omitted.

Appengine's whack pricing says: we'd like to kick you off appengine onto GCE but that would scare away all the enterprise users from GCE. If we announce regular Moore's Law following price cuts for GCE and nothing for appengine, well hey, we can't stop you leaving. We're totally not twisting you're arm! (aside: we are) "A commitment to Moore's Law: https://cloud.google.com/pricing/philosophy/ "

I'm sorry, this simply does not make any sense. If you'd take the time to articulate a well-formed case, I will take the time to dig in. Thanks... -- Chris

I think he is saying that GCE being less expensive than App Engine will fore people to migrate.
Post reply on HN