71–80 of 148 posts
This no need to compile SQLite into your Cloudflare Worker. We provide it native on our platform as D1. And it gives you replication. https://blog.cloudflare.com/d1-turning-it-up-to-11/ Also, I think the idea of “edge” doesn’t make a ton of sense. What we really need is code and data that move around as needed for the best performance. See: https://blog.cloudflare.com/announcing-workers-smart-placeme... What people c…
Earlier quoted context omitted.
In broad strokes having 100k SQLite dbs is a bit like having 100k git repositories. It's doable and just about having tooling to manage it. It actually makes a few things simpler, say progressively rolling out visiting upgrades. You need to write schema migrations anyway, this lets you upgrade one customer at a time.
The problem is that one at a time times 100,000 DBs takes a long time. For every second a migration takes its 1.1 days. You can parallelize it, of course, but that's a challenge in of itself.
Not a challenge at all really.
I extensively used SQLite in a telemetry system for an electric race car. The car has an onboard computer, first a Raspberry Pi then a dual core Arm processor. Onboard code logs ~4000 messages a second into three SQLite databases. After a drive session a script merges the three databases into a single SQLite session log. The session log is decoded on a different computer to ~400 columns of time series data again stor…
Other have asked, but I'd also be very interested in knowing more if you can share more details. I had to spent a non trivial amount of time convincing others to use SQLite instead of dumb files in similar situations, stories like yours are pure gold.
* SQLite is 35% Faster Than The Filesystem [1]
* Writing to files safely is hard [2], and SQLite goes to great details to ensure consistency in case of application, OS, or computer crash [3]
[1]: https://sqlite.org/fasterthanfs.html
Why is SQLite gaining so much momentum and traction lately?
Why is SQLite gaining so much momentum and traction lately?
My personal opinion, is that the relational model is the best option to model data, any attempt to avoid it, replace it, mimic it adds more complexity than it solve
Linked Tables (relations) is the ultimate data modeling tool
Earlier quoted context omitted.
How's D1 meant to be used, since it has a very small maximum size (100mb I believe)? Should I create one database per user, for example? Genuine question.
the docs say the Alpha is limited to 100Mb, and, given the size of SQLite DBs i have seen, 100Mb is quite large, is it not?
This no need to compile SQLite into your Cloudflare Worker. We provide it native on our platform as D1. And it gives you replication. https://blog.cloudflare.com/d1-turning-it-up-to-11/ Also, I think the idea of “edge” doesn’t make a ton of sense. What we really need is code and data that move around as needed for the best performance. See: https://blog.cloudflare.com/announcing-workers-smart-placeme... What people c…
How's D1 meant to be used, since it has a very small maximum size (100mb I believe)? Should I create one database per user, for example? Genuine question.
You can't. D1 won't let you create databases at runtime. At least that was the case last time I tried it, and it's what killed it for me.
Earlier quoted context omitted.
How's D1 meant to be used, since it has a very small maximum size (100mb I believe)? Should I create one database per user, for example? Genuine question.
The maximum size will increase to at least 1GB in the near future, enabled by our recent rewrite of the underlying storage layer. Perhaps we can push it further, even, we'll see. But yes, I think the next step is then some sort of sharding. Sharding by user would be an obvious approach for many apps. I think we should build a framework to help manage this, so apps would only need to provide some callbacks e.g. to com…
Much like R2 has an S3 compat API as well as a way to retrieve files outside of workers (IIRC) will this be true of all forms of storage on the platform?
Sometimes it'd be nice to prime the cache with KV for example, without having to invoke a worker