81–90 of 148 posts
Earlier quoted context omitted.
How's D1 meant to be used, since it has a very small maximum size (100mb I believe)? Should I create one database per user, for example? Genuine question.
the docs say the Alpha is limited to 100Mb, and, given the size of SQLite DBs i have seen, 100Mb is quite large, is it not?
> It’s borderline impossible to compare it against networked database management systems like MySQL or Postgres, because SQLite is a library that operates on a local file — it bypasses all the costs incurred by the network, layers of serialization and deserialization, authentication, authorization, and more. Postgres can run locally, communicating via a Unix socket. You should try benchmarking this before stating tha…
I extensively used SQLite in a telemetry system for an electric race car. The car has an onboard computer, first a Raspberry Pi then a dual core Arm processor. Onboard code logs ~4000 messages a second into three SQLite databases. After a drive session a script merges the three databases into a single SQLite session log. The session log is decoded on a different computer to ~400 columns of time series data again stor…
It sounds unbelievable that you couldn't append some CSV or binary record to file via an open file descriptor more efficiently than inserting into SQLite.
(It's clear that to have the data already in SQLite form saves post-race steps.)
[0] https://www.splitgraph.com/blog/deploying-serverless-seafowl
Why is SQLite gaining so much momentum and traction lately?
10+ years ago there were 2 big names in open source embedded databases, Berkley DB, and SQLite.
Then the NoSQL fad hit, and a bunch of new key value stores emerged. Now developers are remembering what made relational databases great.
Even when you don't need relational features, SQLite is battle tested, and a great choice for storing key, value pairs.
Earlier quoted context omitted.
> Onboard code logs ~4000 messages a second into three SQLite databases. After a drive session a script merges the three databases into a single SQLite session log. A bit unrelated, but curious as to why you wrote to three separate databases only later to merge them.
Could be three different processes interacting with different systems. Say one process listening to a CAN bus, another to local Ethernet[1] and a third to some I2C/SPI sensors. If they don't interact otherwise, which they probably wouldn't given they're just logging telemetry, it's more flexible to just have them as separate processes. The I2C/SPI might be a Python script, the others in C/C++ or Rust say, whatever is…
Earlier quoted context omitted.
Yea, but does it scale? Yes, for 100 users maybe. But you have 100 000 small sqlite db:s I would be a bit worried about how it scales. Electric is super cool. Can't wait until it is a bit more mature.
In broad strokes having 100k SQLite dbs is a bit like having 100k git repositories. It's doable and just about having tooling to manage it. It actually makes a few things simpler, say progressively rolling out visiting upgrades. You need to write schema migrations anyway, this lets you upgrade one customer at a time.
Earlier quoted context omitted.
Yea, but does it scale? Yes, for 100 users maybe. But you have 100 000 small sqlite db:s I would be a bit worried about how it scales. Electric is super cool. Can't wait until it is a bit more mature.
> Yea, but does it scale? Yes, for 100 users maybe. But you have 100 000 small sqlite db:s If it's a case of 1 node per customer (node being vm, lambda, cloudflare worker, whatever) then there should be no limit. Maybe you're thinking of a more "traditional" approach with a server process managing 1 sqlite db per customer, which might make sense in order to keep cloud costs down. Even in that case it should be trivia…
This no need to compile SQLite into your Cloudflare Worker. We provide it native on our platform as D1. And it gives you replication. https://blog.cloudflare.com/d1-turning-it-up-to-11/ Also, I think the idea of “edge” doesn’t make a ton of sense. What we really need is code and data that move around as needed for the best performance. See: https://blog.cloudflare.com/announcing-workers-smart-placeme... What people c…
Server side rendering or “single optimization” as the “edge” is kind of an abuse of terms.
Earlier quoted context omitted.
How's D1 meant to be used, since it has a very small maximum size (100mb I believe)? Should I create one database per user, for example? Genuine question.
The maximum size will increase to at least 1GB in the near future, enabled by our recent rewrite of the underlying storage layer. Perhaps we can push it further, even, we'll see. But yes, I think the next step is then some sort of sharding. Sharding by user would be an obvious approach for many apps. I think we should build a framework to help manage this, so apps would only need to provide some callbacks e.g. to com…