100k TPS over a billion rows: the unreasonable effectiveness of SQLite
1–10 of 169 posts
Re: 100k TPS over a billion rows: the unreasonable effectiveness of SQLite
#2[deleted]
Re: 100k TPS over a billion rows: the unreasonable effectiveness of SQLite
#3The only caveat being this assumes all your data can fit on a single machine, and all your processing can fit on one machine. You can get a a u-24tb1.112xlarge with 448 vcores, 24TB RAM for 255/hour and attach 64TB of EBS -- that's a lot of runway.
Re: 100k TPS over a billion rows: the unreasonable effectiveness of SQLite
#4That's a helpful TPS Report.
TIL `SAVEPOINT` can occur in a BEGIN ... END SQLite transaction, and that works with optimizing batch size on a particular node with a given load.
Is there a solution for SQLite WAL corruption?
From https://news.ycombinator.com/item?id=45133444 :
> "PSA: SQLite WAL checksums fail silently and may lose data" https://news.ycombinator.com/item?id=44672902
> sqlite-parquet-vtable, [...]
Re: 100k TPS over a billion rows: the unreasonable effectiveness of SQLite
#5The only caveat being this assumes all your data can fit on a single machine, and all your processing can fit on one machine. You can get a a u-24tb1.112xlarge with 448 vcores, 24TB RAM for 255/hour and attach 64TB of EBS -- that's a lot of runway.
Or rent a bare-metal machine from hetzner with 2-3x performance per core and 90% less costs[1].
[1] Various HN posts regarding Hetzner vs AWS in terms of costs and perf.
Re: 100k TPS over a billion rows: the unreasonable effectiveness of SQLite
#6The only caveat being this assumes all your data can fit on a single machine, and all your processing can fit on one machine. You can get a a u-24tb1.112xlarge with 448 vcores, 24TB RAM for 255/hour and attach 64TB of EBS -- that's a lot of runway.
Scale-up solves a lot of problems for stable workloads. But elasticity is poor, so you either live with overprovisinoed capacity (multiples, not percentages) or fail under spiky load which often time is the most valuable moment (viral traffic, Black Friday, etc).
No one has solved this problem. Scale out is typically more elastic, at least for reads.
Re: 100k TPS over a billion rows: the unreasonable effectiveness of SQLite
#7That's a helpful TPS Report. TIL `SAVEPOINT` can occur in a BEGIN ... END SQLite transaction, and that works with optimizing batch size on a particular node with a given load. Is there a solution for SQLite WAL corruption? From https://news.ycombinator.com/item?id=45133444 : > "PSA: SQLite WAL checksums fail silently and may lose data" https://news.ycombinator.com/item?id=44672902 > sqlite-parquet-vtable , [...]
As mentioned in those threads, there is no SQLite WAL corruption if you have a working disk & file system. If you don't, then all bets are off - SQLite doesn't protect you against that, and most other databases won't either. And nested transactions (SAVEPOINT) won't have have any impact on this - all it does in this form is reduce the number of transactions you have.
Re: 100k TPS over a billion rows: the unreasonable effectiveness of SQLite
#8The only caveat being this assumes all your data can fit on a single machine, and all your processing can fit on one machine. You can get a a u-24tb1.112xlarge with 448 vcores, 24TB RAM for 255/hour and attach 64TB of EBS -- that's a lot of runway.
[deleted]
Re: 100k TPS over a billion rows: the unreasonable effectiveness of SQLite
#9The only caveat being this assumes all your data can fit on a single machine, and all your processing can fit on one machine. You can get a a u-24tb1.112xlarge with 448 vcores, 24TB RAM for 255/hour and attach 64TB of EBS -- that's a lot of runway.
> The only caveat being this assumes all your data can fit on a single machine
Does my data fit in RAM? https://yourdatafitsinram.net/
Not sure using EC2/AWS/Amazon is a good example here, if you're squeezing for large single-node performance you most certainly go for dedicated servers, or at least avoid vCPUs like a plague.
Re: 100k TPS over a billion rows: the unreasonable effectiveness of SQLite
#10Does anyone have examples of organizations that have leveraged SQLite and written about their experience? I've read a lot of theory and benchmarks about it lately and it seems extremely impressive, but I'm wondering if anyone has written about pushing it to its limits "in production"