Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Startup Fuck-ups: How we lost 25% of our monthly revenue overnight

medium.com

Post reply on HN
Print view View original

51–60 of 60 posts

Re: Startup Fuck-ups: How we lost 25% of our monthly revenue overnight

#51

Post by mbesto »

spacefight wrote: post #3
shizcakes wrote: post #2

This could be solved an even more fundamental way: Don't run your own mailer as a startup. There are lots of companies that will be responsible for email deliverability on your behalf, via an API. If it took them 3 months to notice no mail was being sent at all, imagine how long it's going to take them to figure out that their IP is blacklisted in Spamhaus or any number of other deliverability issues?

Not everyone wants to outsource transactional mail handling to a third party.

But clearly if an error in it is going to cause 40% of your business to drop off "overnight", you might want to outsource it to someone who can actually maintain it properly.

Re: Startup Fuck-ups: How we lost 25% of our monthly revenue overnight

#52

Post by protonfish »

meritt wrote: post #33
protonfish wrote: post #20

Earlier quoted context omitted.

I don't care what language they use, personally. However I am at work and having the F-word in 72pt font blaring from the top of my browser made scroll down very quickly. I doubt my bosses would be thrilled to see that.

Man, I'm sorry, but what sort of job do you have where seeing the word "Fuck" is like an actual job risk?

It pays awesome and is easy work. If the handbook prohibited saying the word "zucchini" under penalty of dismissal I'd happily strike the word from my vocabulary.

Re: Startup Fuck-ups: How we lost 25% of our monthly revenue overnight

#54

Post by ibebrett »

protonfish wrote: post #28
sjs382 wrote: post #25

Earlier quoted context omitted.

>> The email providers I have used don't make this easy so it is usually better to run the server yourself. Mandrill, Mailgun, Sendgrid all make this easy.

It has been a few years since I have been saddled with email responsibilities so it's not surprising I am a little behind the times. But it still doesn't mean you shouldn't be checking that customers are receiving your emails.

with alot of these services you can actually watch on a google maps app as customers open your emails.

Re: Startup Fuck-ups: How we lost 25% of our monthly revenue overnight

#55

Post by Bluecobra »

wmt wrote: post #40

What we found was that a number of failed jobs were being kept by the system, which meant that these were taking up a ton of space that they shouldn’t have been. To fix the issue, we put together a script to delete the failed items, since any retries to send them didn’t appear to work. At this point my head was screaming "NOOOOooooo!" and made me feel bad for author for the whatever disaster would soon follow. Not on…

Well said. This is the end result of letting developers perform system administration tasks. I doubt that they were able to Google the root cause on Stack Overflow. :)

Re: Startup Fuck-ups: How we lost 25% of our monthly revenue overnight

#57

Post by specialist »

Someone1234 wrote: post #16

As someone who has created quite a few of these mailers, the queue getting stuck on a single piece of mail and hanging indefinitely is incredibly common. As time has gone on my solutions have become simpler and more pragmatic, since additional complexity breads additional problems. For example, if I was going to design an emailer today: -Grab the email from a database save it to a file (likely one or several XML file…

I strongly approve. That's how I implemented the backend of my medical records exchange stack.

File system based queues. Point-to-point data interchange, so no concurrency; your notion of imprinted work tasks with PIDs is a good idea.

I used a "pull" model. A thread would take work from one directory and drop into another. Poor man's workflow. Worked great. Super easy to monitor and troubleshoot.

Using Java, implementing the cross platform file locking (so a downstream process wouldn't pull a task before it was ready) took some finesse, a small caveat.

Re: Startup Fuck-ups: How we lost 25% of our monthly revenue overnight

#58

Post by specialist »

brusch64 wrote: post #37
Someone1234 wrote: post #16

As someone who has created quite a few of these mailers, the queue getting stuck on a single piece of mail and hanging indefinitely is incredibly common. As time has gone on my solutions have become simpler and more pragmatic, since additional complexity breads additional problems. For example, if I was going to design an emailer today: -Grab the email from a database save it to a file (likely one or several XML file…

Sounds like you just recreated Microsoft Biztalk. But to sound less like a dick - communicating with a low probability of errors is hard !

Having implemented, deployed, and supported "workflows" using BizTalk, ICAN/JCAPS, Orion, homegrown JMS-based stacks, Cache & Ensemble, etc...

Someone1234's solution is the antithesis of BizTalk, the complete refutation of the ultimate futility of using workflow engines.

Re: Startup Fuck-ups: How we lost 25% of our monthly revenue overnight

#59

Post by nasalgoat »

I have a script that sets up 90% of a full nagios/icinga server automatically in about 5 minutes.

Why, in 2014, are people still not monitoring everything as job #1?

Why isn't this being taught in schools? How do people with tech jobs not know this?

Re: Startup Fuck-ups: How we lost 25% of our monthly revenue overnight

#60

Post by velocitypsycho »

NDizzle wrote: post #42
shizcakes wrote: post #2

This could be solved an even more fundamental way: Don't run your own mailer as a startup. There are lots of companies that will be responsible for email deliverability on your behalf, via an API. If it took them 3 months to notice no mail was being sent at all, imagine how long it's going to take them to figure out that their IP is blacklisted in Spamhaus or any number of other deliverability issues?

I don't entirely agree with this. I don't use my own mailer currently. I DO, however, use postfix to queue and relay email to rackspace, who actually sends my email. I don't think the API method is appropriate because then you need to run some other queue system so your app has an instant response time for the user. Their action would create a queue entry (with whatever data) that will eventually be fired off as an A…

Every email service I've seen offers an API and SMTP. For many applications, SMTP will work just fine.
Post reply on HN