Earlier quoted context omitted.
Thanks! The site is a Hugo[1] site with a modified version of the Doks template[2]. I changed some colors and the font and I hired a designer on Fiverr[3] to do a logo for $99. [1]: https://gohugo.io/ [2]: https://getdoks.org/ [3]: https://www.fiverr.com/dieseelle
You are my hero! Thank you! Front end for a back-end guy like me is a learning journey. (I suppose I should stop labeling myself!)
Why I Built Litestream
91–100 of 178 posts
Re: Why I Built Litestream
#92> “But nobody writes production applications with SQLite, right?" We've been doing it for 5 years now. Basic tricks we employ are: Use PRAGMA user_version for purposes of managing automatic migrations, a. la. Entity Framework. This means you can actually do one better than Microsoft's approach, because you don't need a special unicorn table to store migration info. A simple integer compared with your latest integer a…
Thanks for details. I yet wait to see how somebody serve 1M users on a web service using sqlite. Sounds like you can do all that since you create desktop app, you almost never need anything more then sqlite for that.
Re: Why I Built Litestream
#93Awesome! I built a side-business that runs completely on Crystal + SQLite. Very light, fast service and makes ~$200k/mo. I just cp my sqlite file to S3 every 2 hours. From my app, i have a page[1] where i can load any snapshot database saved on S3. I can backup at anytime too with a click, which i do before deployment. [1]: https://i.imgur.com/Ls1Tnxc.png
As a non-native speaker I struggle with "make". To me "make" for a side-business means "personal income" but it looks like you mean "revenue"?
Edit: It's also totally cool to ask for a clarification, e.g. "huh do you mean total annual revenue or this is your annual salary from your business?"
Re: Why I Built Litestream
#94> “But nobody writes production applications with SQLite, right?" We've been doing it for 5 years now. Basic tricks we employ are: Use PRAGMA user_version for purposes of managing automatic migrations, a. la. Entity Framework. This means you can actually do one better than Microsoft's approach, because you don't need a special unicorn table to store migration info. A simple integer compared with your latest integer a…
>We can afford to lose the last few minutes of work without anyone getting yelled at. Some modern virtualization technologies do help a lot in this regard. Running bare metal you need to be a little more careful. How does virtualization help with data loss? I would expect that a VM can't have guarantees better than the underlying physical hardware provides.
Re: Why I Built Litestream
#95Awesome! I built a side-business that runs completely on Crystal + SQLite. Very light, fast service and makes ~$200k/mo. I just cp my sqlite file to S3 every 2 hours. From my app, i have a page[1] where i can load any snapshot database saved on S3. I can backup at anytime too with a click, which i do before deployment. [1]: https://i.imgur.com/Ls1Tnxc.png
$200k/mo is remarkable for a "side" business! Can you share more of your story?
Re: Why I Built Litestream
#96I've been building a web service on a cheapo DigitalOcean box lately, so I'm excited to see explorations in this space, especially with an eye towards staying cheap! I'd probably only use this particular tool if it could hook up to Backblaze B2 instead of S3, since life's too short to ever have to engage with the hell that is AWS for a hobby project, but since B2's API-compatible it seems like a feature that could be…
I'm using multiples DBs and one of them is a simulation of key-value store. It's like a (Python) dictionary that is really an SQLite database, then I use keys like:
db["users:1000:email"] = "email@email.com"
The feature I think I miss is being able to connect to the production database from my laptop to do a quick check. With SQLite, I have to ssh into the server and run the SQLite CLI (or copy the whole file).Many people also mention concurrency, but I think that if you make your INSERT/UPDATE/DELETE statements fast and short + use WAL mode + use PRAGMA synchronous = 1, and some other optimizations, you can get quite far.
Re: Why I Built Litestream
#97This is why a database like SQLite has its miraculous read performance: "competing not with Postgres but with fopen()", as somebody said.
But any mutation locks the entire database. This, again, simplifies the implementation a lot, and guarantees serialized DML execution.
Rather few web apps need high write concurrency and low write latency. Great many web apps serve 99.9% of hits with SELECTs only, and when some new data needs to be persisted, the user can very well wait for a second or two, so rarely it happens. But this is often forgotten.
Same realization brought a wave of static site generators: updates are so rare that serving pages from a database makes no sense, and caching them makes little sense: just produce the "pre-cached" pages and serve them as is.
Maybe a similar wave can come to lighter-weight web apps: updates are so rare that you don't need Postgres to handle them. You can use SQLite, or Redis, or flat files as the source of your data, with massively less headache. Horizontal scaling becomes trivial. Updates are still possible, of course, you just pay a much lower complexity price for them, while paying a higher latency price.
It's easy to notice that horizontally scaled local databases are already known: it's called "sharding". Unless you need to run arbitrary analytical queries across shards, eventual replication of changes, where needed, is sufficient. (BTW this is how many very large distributed databases operate anyway.) Litestream already seems to support creation of read-only replicas. This can allow each shard have a copy of all the data of a cluster, while only being able to update its own partition. This is a very reasonable setup even for some rather high-load and data-packed sites.
Re: Why I Built Litestream
#98Use WAL mode (writers don't block readers):
PRAGMA journal_mode = 'WAL'
Use memory as temporary storage: PRAGMA temp_store = 2
Faster synchronization that still keeps the data safe: PRAGMA synchronous = 1
Increase cache size (in this case to 64MB), the default is 2MB PRAGMA cache_size = -64000
Lastly, use a modern version of SQLite. Many default installations come with versions from a few years ago. In Python for example, you can use pysqlite3[0] to get the latest SQLite without worrying about compiling it (and it also comes with excellent compilation defaults).Re: Why I Built Litestream
#99> “But nobody writes production applications with SQLite, right?" We've been doing it for 5 years now. Basic tricks we employ are: Use PRAGMA user_version for purposes of managing automatic migrations, a. la. Entity Framework. This means you can actually do one better than Microsoft's approach, because you don't need a special unicorn table to store migration info. A simple integer compared with your latest integer a…
In my experience, sqlite performance becomes problematic quite quickly, even with settings you mentioned (WAL etc).
Re: Why I Built Litestream
#100SQLite should be fine if your workload biased towards reads. But its design does prevent many-writers and these, even operates on completely different tables, has to be serialized between each other. This can be more problematic since many SQLite articles recommend big transactions to improve performance. And write transactions have to be serialized between each other. If you are a write-heavy workload, at some point…
Yes, SQLite does serialize writes and write-heavy workloads and long-running transactions are not a good fit. Small batching of writes can help but SQLite provides options to tradeoff throughput for durability if that works for your situation: PRAGMA synchronous = NORMAL; That avoids fsync() calls on the WAL until checkpointing. I don't know of any libraries that implement transaction coalescing as it can be differen…