Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We saved $500k per year by rolling our own "S3"

engineering.nanit.com

Post reply on HN
Print view View original

251–260 of 261 posts

Re: We saved $500k per year by rolling our own "S3"

#251

Post by dxxvi »

jrochkind1 wrote: post #210
dxxvi wrote: post #40

So, you want a place to store many files in a short period of time and when there's a new file, somebody must be notified? Have you ever thought of using a postgresql db (also on aws) to store those files and use CDC to publish messages about those files to a kafka topic? In your original way, we need 3 aws services: s3, lambda and sqs. With this way, we need 2: postgresql and kafka. I'm not sure how well this method…

Like put the video blobs themselves in postgres data columns? Does putting very large (relative to what you normally put in postgres) files in pg work well? Genuine question, i do not know, I've been considering it too and hesitant about it.

> I've been considering it too and hesitant about it

Why hesitant? Just ask AI. It'll tell you how to do it and then you can experiment it yourself.

Re: We saved $500k per year by rolling our own "S3"

#252

Post by ravedave5 »

dxxvi wrote: post #40

So, you want a place to store many files in a short period of time and when there's a new file, somebody must be notified? Have you ever thought of using a postgresql db (also on aws) to store those files and use CDC to publish messages about those files to a kafka topic? In your original way, we need 3 aws services: s3, lambda and sqs. With this way, we need 2: postgresql and kafka. I'm not sure how well this method…

I've dealt with images in a database and it was a disaster, the transfer times are garbage.

Re: We saved $500k per year by rolling our own "S3"

#253

Post by kingnothing »

Nanit is horrible spyware. Do not buy their products.

If you have a router that lets you inspect data flowing out, you'll be astonished at what your little Nanit cam exfiltrates from your home network. Even if you don't pay for their subscription service, they still attempt to exfil all of the video footage caught on your camera to their servers. You can block it and it will still work, but you shouldn't have to do that in the first place if you don't pay for their cloud service.

Stay away if you value your privacy.

Re: We saved $500k per year by rolling our own "S3"

#254

Post by sbrother »

jrochkind1 wrote: post #238
sbrother wrote: post #64

Earlier quoted context omitted.

As a happy customer, I picked nanit because it actually worked. We didn’t even use the “smart” features, but “you can turn on the app from anywhere you happen to be and expect the video feed to work” is unfortunately a bar that no competitor I tried could meet. The others were mostly made by non-software companies with outsourced apps that worked maybe 50% of the time. I wish we could have local-first and e2ee consum…

> you can turn on the app from anywhere you happen to be and expect the video feed to work if i'm understanding "anywhere you happen to be" right: Real question -- I'm not a parent. What is your use case for wanting to monitor your baby remotely from a different location than your baby? Obviously someone is with them at the house or location with the baby! You don't trust em? Or just like seeing/hearing your baby whe…

It's a reasonable question! I'm past the baby monitor stage now, but when we used our Nanit from a different network it was for things like:

* Doing garage or yard work where Wifi coverage was spotty. May seem like an edge case but remember that when baby is sleeping is exactly when you want to be doing things like yard work.

* Hanging out across the street cooking out with the neighbors while baby sleeps

* Having a couple drinks at the hotel bar on vacation after baby goes to sleep. You're only ~30 seconds from your room if baby wakes up, but it's nice to not have to sit in a dark room for the whole evening after 7pm.

Re: We saved $500k per year by rolling our own "S3"

#255

Post by rhinoceraptor »

varenc wrote: post #246
rhinoceraptor wrote: post #231

Earlier quoted context omitted.

It calls back the classic, "you can already build such a system quite trivially": https://news.ycombinator.com/item?id=9224

Hah never thought I'd be compared to that legendary comment! It hits home for me because I worked at Dropbox for years. I did at least qualify that I couldn't in good conscience recommend my setup to non-techies :) If you don't want a baby camera system that's also a part-time hobby...Nanit does seem like the best option. I just lament that the best option requires giving up so much.

Don't worry, I also use Ubiquiti, and self-host Home Assistant on my TrueNAS :)

I have a little server rack cobbled together out of wood under my basement stairs, with a UDM Pro, 24 port POE switch, and an ancient Dell 2U poweredge for TrueNAS.

Re: We saved $500k per year by rolling our own "S3"

#256

Post by jrochkind1 »

dxxvi wrote: post #250
jrochkind1 wrote: post #210

Earlier quoted context omitted.

Like put the video blobs themselves in postgres data columns? Does putting very large (relative to what you normally put in postgres) files in pg work well? Genuine question, i do not know, I've been considering it too and hesitant about it.

I already checked with AI before putting the comment :-) 1GB with the bytea data type ( https://www.postgresql.org/docs/current/datatype-binary.html ) and 4TB with the BLOB data type ( https://wiki.postgresql.org/wiki/BinaryFilesInDB ).

Have you done this? I can google or AI for the max size that postgres will allow, sure. I have googled in the past for whether this actually works well, and have gotten answers leaning towards most advice against it in real world scenarios.

So if you have experience with this and it did work well, I'm curious to hear about it! That's why i asked about if it worked well, not about the maximum size postgres allowed in various data types.

If you have no experience with it, but are just posting advice based on what AI tells you about max sizes of data allowed by pg that I can get from the same source too, then okay, fair enough, and certainly no need to give me any more of that!

Re: We saved $500k per year by rolling our own "S3"

#257

Post by halJordan »

elevation wrote: post #232
Havoc wrote: post #3

Tbh I feel this in one of those that would be significantly cleaner without serverless in first place. Sticking something with 2 second lifespan on disk to shoehorn it into aws serverless paradigm created problems and cost out of thin air here Good solution moving at least partially to a in memory solution though

The scalable in-memory solution took quite a bit of testing to get right. Building this on the early side of the business when the requirements are not well known can be a giant budget and time tar pit. Plus without customers it’s hard to confidently test at scale. Using S3 for an MVP and marking this component as “done” seems like the right solution, regardless of the serverless paradigm.

Agreed, but the first design principle is "eliminate complexity at the design level." MVPs and what they represent (a failure to design) are an albatross

Re: We saved $500k per year by rolling our own "S3"

#258

Post by me-vs-cat »

varenc wrote: post #67
sbrother wrote: post #64

Earlier quoted context omitted.

As a happy customer, I picked nanit because it actually worked. We didn’t even use the “smart” features, but “you can turn on the app from anywhere you happen to be and expect the video feed to work” is unfortunately a bar that no competitor I tried could meet. The others were mostly made by non-software companies with outsourced apps that worked maybe 50% of the time. I wish we could have local-first and e2ee consum…

I self host my "baby monitor" with UniFi Protect on UCG-Max and a G6 Instant wireless camera. It's more work to setup, but pretty easy for a techie. It has the "turn on the app anywhere and it works" feature, and with a 2TB SSD I get a month+ of video storage. Because storage is local, it doesn't need to compress the video and I get a super clear 4K image. And I use Homebridge to expose the camera over Apple HomeKit…

Regarding out-of-home access, I'll drop a note about connecting your phone to your home network with a VPN. Now you're always connecting the same way for this and everything else.

It's not perfect because wifi networks might block the VPN, but for the one wifi network I use the most, Wireguard on port 53 works splendidly, for now.

Re: We saved $500k per year by rolling our own "S3"

#259

Post by mannyv »

ahoka wrote: post #203
tcdent wrote: post #8

Earlier quoted context omitted.

Yeah, so now you're basically running a heavy instance in order to get the network throughput and the RAM, but not really using that much CPU when you could probably handle the encode with the available headroom. Although the article lists TLS handshakes as being a significant source of CPU usage, I must be missing something because I don't see how that is anywhere near the top of the constraints of a system like thi…

TLS being a bottleneck when doing image processing is very-very weird.

They should terminate TLS at the LB and use normal connection inside...unless they have some HIPAA privacy-like regulation going on.

Re: We saved $500k per year by rolling our own "S3"

#260

Post by 0x6c6f6c »

Gasp0de wrote: post #184
alex_smart wrote: post #182

Earlier quoted context omitted.

From the article, individual video segments were 2-6 MB in size and SQS and Kinesis have a 1MB limit for individual records so they couldn’t have used either service directly. At least not without breaking their segments into even smaller chunks.

You're right, I didn't pay attention there. Still seems that there a many solutions better suited than S3. Probably a classic case of "We need an MVP fast, let's optimize later".

Agreed, but this isn't always bad. Optimizing early with unclear requirements can kill time, which at an early stage is just delaying product launch.

Migrate/optimize later when you're actually reaching scale is a perfectly reasonable approach.

In fact, if you have a decent abstraction or standard in place (e.g. S3 API or repository pattern) you can swap it out in place.

Post reply on HN