Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A serverless email server on AWS using S3 and SES

github.com

Post reply on HN
Print view View original

181–190 of 266 posts

Re: A serverless email server on AWS using S3 and SES

#181

Post by tuldia »

Cyph0n wrote: post #159
tuldia wrote: post #153

Earlier quoted context omitted.

> I’ve been working on a small side project that involves processing incoming email. In particular, it’s an app that needs to do something for each email it receives from (hopefully paying!) users. I wish you all the best! Mind if I ask for the link? > With (1), there is no need to worry about overages, but scaling the mail server might be challenging. Honestly, quite the opposite. 1. Duplicate your MX box. 2. Duplic…

Thanks! No link yet, but there is a messy GH repo: https://github.com/aksiksi/vaulty Yes, I only learned about MX record priorities last night haha. With Postfix, the most straightforward way to run code on receiving an email seems to be through a pipe filter. Running multiple filter processes probably requires a beefy server. Thanks for that link! I might just use a similar approach to allow users to configure how t…

Thanks for the link!

Seems like a really fun project.

> I only learned about MX record priorities last night haha.

Nice!

> With Postfix, the most straightforward way to run code on receiving an email seems to be through a pipe filter.

True.

Take a look into LMTP, you may be able to "read the email" directly from your rust daemon (via unix socket or tcp) :)

Re: A serverless email server on AWS using S3 and SES

#182

Post by tyingq »

scarface74 wrote: post #141
tyingq wrote: post #140

Earlier quoted context omitted.

"Can a web hook automatically instantiate enough VMs on demand without reducing throughput?" Yes, there are various cgi-bin implementations with horizontal auto-scaling.

Across multiple servers or do you still have to provision your server to handle peak load?

Yes, you can do that the same way Amazon does it. Like I said, Amazon packaged existing tech in an easy to use way.

Re: A serverless email server on AWS using S3 and SES

#183

Post by cddotdotslash »

tuldia wrote: post #175
cddotdotslash wrote: post #129

Earlier quoted context omitted.

There are absolutely valid criticisms of serverless (as you called out, vendor lock in and complexity for large applications being a big ones). But it's still an important and interesting technology that many individuals and companies are invested in, sharing side projects, open source tools (like the OP's), etc. In many cases, it can be cost efficient, or even (human) resource efficient to use a serverless setup. So…

You are missing the point. (the "tired responses") No, there only very few cases, such as shaving video encoding time (the one that started all this mess around "serverless") For majority of other cases, using "(human) resource efficient" or "time saved managing servers" to justify the use of "serverless" is a plain bogus argument, since you end up shifting (best case scenario) OR spending more (worst case) time "man…

The poster above me does a better job of articulating the points I had, but at the end of the day, it seems to me like you are discounting an entire model of compute, part of an industry, the work that tons of developers are doing, not to mention new and interesting ways of developing new things, simply because you either don't want to learn something new or because you're attached to an existing way.

Note that I've never once said that running your own mail server is the wrong thing to do, because I don't claim to know everyone's complex environments. Instead, I'm pushing back against this habit of immediately discounting anything new simply because it replaces an older way of operating.

Fortunately the industry as a whole seems much more receptive to new things, otherwise we'd still be on mainframes everywhere, right?

Re: A serverless email server on AWS using S3 and SES

#184

Post by fulafel »

scarface74 wrote: post #73
tyingq wrote: post #49

Earlier quoted context omitted.

There's a firecracker VM and a Linux OS under the Lambda. Both of which need to be maintained by Amazon. Edit: "managed cgi-bin" is perhaps more apt.

cgi-bin didn’t have any type of process isolation. Also, it didn’t tie in to other none web based events.

cgi-bin had pretty strong process isolation for the day (Unix user account and process boundaries/limits/quotas). Enough to have remarkably robust multitenant setups on single servers.

Re: A serverless email server on AWS using S3 and SES

#185

Post by scarface74 »

tyingq wrote: post #182
scarface74 wrote: post #141

Earlier quoted context omitted.

Across multiple servers or do you still have to provision your server to handle peak load?

Yes, you can do that the same way Amazon does it. Like I said, Amazon packaged existing tech in an easy to use way.

That still begs the question, why am I as a developer wasting time fiddling with infrastructure that I could just throw a little money and a yaml file at AWS and let them do it? How am I adding a business value that will make my company money doing grunt work?

Re: A serverless email server on AWS using S3 and SES

#186

Post by unilynx »

Cyph0n wrote: post #159
tuldia wrote: post #153

Earlier quoted context omitted.

> I’ve been working on a small side project that involves processing incoming email. In particular, it’s an app that needs to do something for each email it receives from (hopefully paying!) users. I wish you all the best! Mind if I ask for the link? > With (1), there is no need to worry about overages, but scaling the mail server might be challenging. Honestly, quite the opposite. 1. Duplicate your MX box. 2. Duplic…

Thanks! No link yet, but there is a messy GH repo: https://github.com/aksiksi/vaulty Yes, I only learned about MX record priorities last night haha. With Postfix, the most straightforward way to run code on receiving an email seems to be through a pipe filter. Running multiple filter processes probably requires a beefy server. Thanks for that link! I might just use a similar approach to allow users to configure how t…

Then limit the number of filters... you can have postfix run a fixed number of smtpd processes, and each process handles only one message at a time.

When they're all full, your server will just stop handling messages, but SMTP will retry anyway, giving you plenty of time to scale up if the load is consistently too high

Re: A serverless email server on AWS using S3 and SES

#187

Post by [deleted] »

scarface74 wrote: post #178
wodny wrote: post #123

Earlier quoted context omitted.

On the contrary, I've worked with 8 and {32,64}bit x {LE,BE} chips many times and wrote quite a lot of network code in C. But you're mixing syntax with semantics.

Which byte is the high order byte is just as much semantics defined by the chip vendor that would be confusing to the uninitiated as “Serverless” would be to an old school net ops person who doesn’t know anything about cloud computing.

[deleted]

Re: A serverless email server on AWS using S3 and SES

#188

Post by tuldia »

scarface74 wrote: post #180
tuldia wrote: post #175

Earlier quoted context omitted.

You are missing the point. (the "tired responses") No, there only very few cases, such as shaving video encoding time (the one that started all this mess around "serverless") For majority of other cases, using "(human) resource efficient" or "time saved managing servers" to justify the use of "serverless" is a plain bogus argument, since you end up shifting (best case scenario) OR spending more (worst case) time "man…

Again you only spend more time if you don’t know what you’re doing. I’m not a front end developer by any stretch of the imagination. Would it be a valid argument if I said that React isn’t a good solution because I have years of experienced with server side rendering? Again, don’t blame the tools just because you didn’t take the time to learn how to use them efficiently. If my name doesn’t give you a clue,I was aroun…

> If my name doesn’t give you a clue,I was around and developing way before AWS or hosted solutions was a thing. Being able to provision resources by writing yaml and not having to deal with the infrastructure gatekeepers is a godsend.

Take some time to learn some soft skills, just enough to be able to convince people to do stuff for you, if you are the "throw over the wall ones" I recommend to get out of your sillo and interact well with the rest of your team :)

The problem with "serverless" is this kind of attitude around it.

Re: A serverless email server on AWS using S3 and SES

#189

Post by idclip »

arno1 wrote: post #15
coder1001 wrote: post #14

Earlier quoted context omitted.

Do you mind writing a blog post somewhere explaining to noobs how this can be done? This will be a great post with lots of traffic I imagine! Thanks, great work!

I am actually thinking on writing it since long ago... But not motivated enough... :/

You sort of almost already did with your reply. Just copy it, add pics, add code, add more text and details,

Done.

Do it.

Re: A serverless email server on AWS using S3 and SES

#190

Post by fiddlerwoaroof »

tuldia wrote: post #122
arno1 wrote: post #12

> This stack was created out of frustration due to the fact that to this day there's no easy way to have a full email server without the overhead of installing and configuring all servers needed to handle incoming and outgoing messages. Interesting approach, though I solved this frustration by the use of a Docker and kept "my data is mine" + "no vendor lock-in" + "I control all the gears" approach. (Though, it's not…

Thanks for sharing! Your setup seems sane and pretty close to what would be if you want to run the same thing on a single server. It makes the code base in the original link looks like a proprietary duct tape spaghetti :P I just recommend setup postscreen[1] and rspamd[2]. 1. http://www.postfix.org/POSTSCREEN_README.html 2. https://www.rspamd.com/

Another “serverless” route for Docker containers is to deploy the containers with Fargate which isn’t too hard and gives you autoscaling without having to re-architect your application for serverless. (And has correspondingly less vendor lock-in)
Post reply on HN