Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Built Litestream

litestream.io

Post reply on HN
Print view View original

11–20 of 178 posts

Re: Why I Built Litestream

#11

Post by benbjohnson »

liuliu wrote: post #7

SQLite should be fine if your workload biased towards reads. But its design does prevent many-writers and these, even operates on completely different tables, has to be serialized between each other. This can be more problematic since many SQLite articles recommend big transactions to improve performance. And write transactions have to be serialized between each other. If you are a write-heavy workload, at some point…

Yes, SQLite does serialize writes and write-heavy workloads and long-running transactions are not a good fit. Small batching of writes can help but SQLite provides options to tradeoff throughput for durability if that works for your situation:

PRAGMA synchronous = NORMAL;

That avoids fsync() calls on the WAL until checkpointing. I don't know of any libraries that implement transaction coalescing as it can be different depending on if you need to batch sequential writes versus writes coming from multiple parallel threads

re: rqlite, yes it handles consensus/replication through Raft but IIRC it doesn't do sharding.

Re: Why I Built Litestream

#12

Post by avolcano »

I've been building a web service on a cheapo DigitalOcean box lately, so I'm excited to see explorations in this space, especially with an eye towards staying cheap! I'd probably only use this particular tool if it could hook up to Backblaze B2 instead of S3, since life's too short to ever have to engage with the hell that is AWS for a hobby project, but since B2's API-compatible it seems like a feature that could be added in the future.

That said, I've always been a little worried about trying SQLite since I'm so used to Postgres. I've currently got Postgres running alongside my app in a Docker container, which isn't too hard to manage. I'm curious whether anyone has switched from Postgres to SQLite in a web app context (whether in the same project, or when making a new project) and if they've found themselves missing any of the features Postgres offers. I've tried to research this before but always found just googling "sqlite vs postgres" just results in surface-level differences that mostly focus on performance, whereas I'm more curious about e.g. the differences in their JSON extensions.

Re: Why I Built Litestream

#13

Post by benbjohnson »

lancehasson wrote: post #9

> Solutions such as Kubernetes tout the benefits of zero-downtime deployments To be fair, these features of k8s are nice to have. Is there a similar tool for running single node (i.e. ec2) instances that offers zero-downtime deploys? One way you could do it is run a EKS cluster with a single ec2 node running your single-node setup. You could get the benefits of k8s while still running the simple single-node arch

You can do zero-downtime deploys on a single node with tools like seamless[1]. That works for Go but I'm sure there are other options for other languages. I'm not sure if/how you can do it with a containerized workload on a single node though. I haven't tried that.

[1] https://github.com/rs/seamless

Re: Why I Built Litestream

#14

Post by throwaway894345 »

> Anecdotally, I’ve run several VPS servers over the years which all have well over 99.9% uptime and have suffered no catastrophic failures.

Without advocating for Kubernetes, there's a pretty big difference between running a single application like WordPress or whathaveyou that receives only occasional updates and a SaaS application that is actively developed by hundreds or thousands of engineers deploying dozens of times per day. Yes, Kubernetes is complex and that complexity can introduce its own downtime issues, but that risk is a large constant whereas without it the risk increases with the number of deployments (and deploying larger deltas less frequently carries its own penalties). It's important to understand and acknowledge these dynamics in order to optimize for uptime and velocity.

Re: Why I Built Litestream

#15

Post by preetamjinka »

avolcano wrote: post #12

I've been building a web service on a cheapo DigitalOcean box lately, so I'm excited to see explorations in this space, especially with an eye towards staying cheap! I'd probably only use this particular tool if it could hook up to Backblaze B2 instead of S3, since life's too short to ever have to engage with the hell that is AWS for a hobby project, but since B2's API-compatible it seems like a feature that could be…

B2 seems to have an S3 Compatible API now.

https://www.backblaze.com/b2/docs/s3_compatible_api.html

Re: Why I Built Litestream

#16

Post by devmunchies »

Awesome! I built a side-business that runs completely on Crystal + SQLite. Very light, fast service and makes ~$200k/mo.

I just cp my sqlite file to S3 every 2 hours.

From my app, i have a page[1] where i can load any snapshot database saved on S3. I can backup at anytime too with a click, which i do before deployment.

[1]: https://i.imgur.com/Ls1Tnxc.png

Re: Why I Built Litestream

#17

Post by truetraveller »

devmunchies wrote: post #16

Awesome! I built a side-business that runs completely on Crystal + SQLite. Very light, fast service and makes ~$200k/mo. I just cp my sqlite file to S3 every 2 hours. From my app, i have a page[1] where i can load any snapshot database saved on S3. I can backup at anytime too with a click, which i do before deployment. [1]: https://i.imgur.com/Ls1Tnxc.png

Such a nice little mini-dashboard. Nice work. I'm copying it!

P.s. Crave Cookie looks neat.

Re: Why I Built Litestream

#18

Post by benbjohnson »

avolcano wrote: post #12

I've been building a web service on a cheapo DigitalOcean box lately, so I'm excited to see explorations in this space, especially with an eye towards staying cheap! I'd probably only use this particular tool if it could hook up to Backblaze B2 instead of S3, since life's too short to ever have to engage with the hell that is AWS for a hobby project, but since B2's API-compatible it seems like a feature that could be…

I haven't tried Backblaze B2 but I agree with you on engaging in AWS hell. Litestream should work with any S3-compatible API. I've added an issue to add guides for B2 & Minio[1].

Regarding Postgres vs SQLite, I've found that I can use much simpler SQL calls with embedded databases when I don't need to worry about N+1 query performance issues. That makes many of the query features moot. That being said, there is a JSON extension for SQLite[2] although I haven't tried it.

[1] https://github.com/benbjohnson/litestream/issues/41

[2] https://www.sqlite.org/json1.html

Re: Why I Built Litestream

#19

Post by renewiltord »

devmunchies wrote: post #16

Awesome! I built a side-business that runs completely on Crystal + SQLite. Very light, fast service and makes ~$200k/mo. I just cp my sqlite file to S3 every 2 hours. From my app, i have a page[1] where i can load any snapshot database saved on S3. I can backup at anytime too with a click, which i do before deployment. [1]: https://i.imgur.com/Ls1Tnxc.png

Is this a public business? Would you mind sharing, if so?

Re: Why I Built Litestream

#20

Post by benbjohnson »

devmunchies wrote: post #16

Awesome! I built a side-business that runs completely on Crystal + SQLite. Very light, fast service and makes ~$200k/mo. I just cp my sqlite file to S3 every 2 hours. From my app, i have a page[1] where i can load any snapshot database saved on S3. I can backup at anytime too with a click, which i do before deployment. [1]: https://i.imgur.com/Ls1Tnxc.png

Honestly, that's a great solution that I've used before too. It seems like a potential 2-hour data loss window isn't catastrophic for your side business so something like Litestream could be overkill.
Post reply on HN