Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Built Litestream

litestream.io

Post reply on HN
Print view View original

61–70 of 178 posts

Re: Why I Built Litestream

#61

Post by benbjohnson »

cfors wrote: post #55

> If you exceed the capacity of a single node, sharding your data can allow you to scale horizontally to multiple nodes. It's so simple, just don't have a lot of data that doesn't need to interact with other data and your problem is solved! /s

This is take seems a little intellectually dishonest. The very next sentence is, "this works particularly well for SaaS applications where each customer is isolated from one another." Sharding doesn't work for all applications. Social networks are one example where all data can potentially interact with all other data.

Sarcasm noted though. :)

Re: Why I Built Litestream

#62

Post by benbjohnson »

polyrand wrote: post #8

Hi Ben, thank you so much for building this! I'm currently working on implementing a set of data structures (and more things later) on top of SQLite ( https://github.com/litements/ ) for some of the same reasons you mention in the article. I probably don't even have 1/10 of your experience, but your work is really motivating me to keep working on it, thanks!

I like where you're going with Litements. There are a lot of servers that could just be libraries running on SQLite. Job queues are one of those I find myself reimplementing sometimes.

Re: Why I Built Litestream

#63

Post by aidenn0 »

benbjohnson wrote: post #61
cfors wrote: post #55

> If you exceed the capacity of a single node, sharding your data can allow you to scale horizontally to multiple nodes. It's so simple, just don't have a lot of data that doesn't need to interact with other data and your problem is solved! /s

This is take seems a little intellectually dishonest. The very next sentence is, "this works particularly well for SaaS applications where each customer is isolated from one another." Sharding doesn't work for all applications. Social networks are one example where all data can potentially interact with all other data. Sarcasm noted though. :)

Social networks are also write-heavy so probably not a good fit for SQLite anyways, right?

Re: Why I Built Litestream

#64

Post by benbjohnson »

tomaszs wrote: post #43

So I create database on VPS, than set up Litestream on s3. Than what? Should I also setup the copied database and use it from my production application? It seems like I don't understand the workflow here in a real case scenario. It my be because I didn't work with such setups

Litestream runs on the VPS with your application & database and then streams changes to S3. If your VPS dies and you lose your data, you can run "litestream restore" to restore your database to the state it was just before your VPS died.

Re: Why I Built Litestream

#65

Post by benbjohnson »

skybrian wrote: post #38

Is Litestream as well-tested as SQLite?

No, not yet! :)

Litestream runs continuously on a test server with generated load and streams backups to S3. It uses physical replication so it'll actually restore the data from S3 periodically and compare the checksum byte-for-byte with the current database.

I made sure Litestream would be safe with the primary database so it only communicates through the SQLite API for locking and state. It should be completely safe to run unless there is a bug in SQLite itself. You can also continue to use a separate periodic backup strategy if you're not comfortable running Litestream alone for disaster recovery.

Re: Why I Built Litestream

#66

Post by camkego »

Ben, this is impressive work, and I will almost certainly use it. (Already believe in sqlite vs mysql/PG/Redis) But as a backend developer I am always trying to learn simpleR ways to build beautiful open source websites. Would you mind sharing how you built the site? I see Bootstrap. Would you share the tools you use? Thank you, Cameron

Re: Why I Built Litestream

#67

Post by cfors »

benbjohnson wrote: post #61
cfors wrote: post #55

> If you exceed the capacity of a single node, sharding your data can allow you to scale horizontally to multiple nodes. It's so simple, just don't have a lot of data that doesn't need to interact with other data and your problem is solved! /s

This is take seems a little intellectually dishonest. The very next sentence is, "this works particularly well for SaaS applications where each customer is isolated from one another." Sharding doesn't work for all applications. Social networks are one example where all data can potentially interact with all other data. Sarcasm noted though. :)

Awesome product, btw. Definitely see where it fits some niches.

Actual concerns about running a real application with real uptime requirements.

1. Say your EC2 or docker container that's hosting this goes down. Is that left up to the user to deal with? RDS handles this for you

2. No ACID transactions if you ever outgrow a DB. You talk about in your pitch that the vertical scaling, so you have to just keep bumping the VPS/container memory.

3. Sure a SaaS application where a customer specific DB is isolated, but as soon as you hit any _real_ scaling limits you immediately are back to the entire problem statement you are aiming to (at least you hint at that in your pitch) solve which is the crazy n-tier architectures we have.

While I was being sarcastic, I was not being intellectually dishonest. There are entire hosts of problems that you call out you are trying to solve without providing any real solution.

Re: Why I Built Litestream

#68

Post by benbjohnson »

camkego wrote: post #66

Ben, this is impressive work, and I will almost certainly use it. (Already believe in sqlite vs mysql/PG/Redis) But as a backend developer I am always trying to learn simpleR ways to build beautiful open source websites. Would you mind sharing how you built the site? I see Bootstrap. Would you share the tools you use? Thank you, Cameron

Thanks! The site is a Hugo[1] site with a modified version of the Doks template[2]. I changed some colors and the font and I hired a designer on Fiverr[3] to do a logo for $99.

[1]: https://gohugo.io/

[2]: https://getdoks.org/

[3]: https://www.fiverr.com/dieseelle

Re: Why I Built Litestream

#69

Post by bdcravens »

truetraveller wrote: post #28
devmunchies wrote: post #16

Awesome! I built a side-business that runs completely on Crystal + SQLite. Very light, fast service and makes ~$200k/mo. I just cp my sqlite file to S3 every 2 hours. From my app, i have a page[1] where i can load any snapshot database saved on S3. I can backup at anytime too with a click, which i do before deployment. [1]: https://i.imgur.com/Ls1Tnxc.png

Also got this gem from the Crave Cookies website: "The software is built "from scratch" like the cookies and is part of Crave's success story. No other food company has the software Crave has for managing deliveries." You should definitely do a write-up on Not Invented Here syndrom. And we sometimes "reinventing" the wheel, in moderation and for core components, really is the best solution.

Most of the time it's a matter of distilling to simplicity. No so much building a better mousetrap, as much as needing only a mousetrap instead of an Animal Processor 5000.

Re: Why I Built Litestream

#70

Post by camkego »

benbjohnson wrote: post #68
camkego wrote: post #66

Ben, this is impressive work, and I will almost certainly use it. (Already believe in sqlite vs mysql/PG/Redis) But as a backend developer I am always trying to learn simpleR ways to build beautiful open source websites. Would you mind sharing how you built the site? I see Bootstrap. Would you share the tools you use? Thank you, Cameron

Thanks! The site is a Hugo[1] site with a modified version of the Doks template[2]. I changed some colors and the font and I hired a designer on Fiverr[3] to do a logo for $99. [1]: https://gohugo.io/ [2]: https://getdoks.org/ [3]: https://www.fiverr.com/dieseelle

You are my hero! Thank you! Front end for a back-end guy like me is a learning journey. (I suppose I should stop labeling myself!)
Post reply on HN