[flagged]
Welcome to the future
LumoSQL
21–30 of 90 posts
Re: LumoSQL
#22Earlier quoted context omitted.
Most of my python projects use SQLite; 1 exception where i need multiprocessing access to the database and no locking problems and speed so i need to run the entire db in memory. https://docs.python.org/3/library/sqlite3.html The built in library makes it really quick and easy to use it. Whereas mysql or in my case id use postgresql if i needed a full db. You're looking for a third party library? I have used Psycopg…
I've used sqlite3 in node, and it's nice and performant for small cases, yes. Mostly I've used it for things small enough where a user could download an entire sqlite file of their data, and then re-upload it in case their data got lost. But ultimately this data gets stored in a true MySQL DB. I don't think I'd trust it to run a whole system with thousands of users and millions of entries... honestly, maybe my issue…
FWIW, I've been running a system with roughly 100K users, about 25 qps on average, with a single SQLite file for several years. No issues with data.
Re: LumoSQL
#23Possibly the wrong place to ask this, but: I've played with SQLite when it was still available in-browser, and I felt that was on the brink of being a game-changer. If it was still supported in-browser and we had replication from the browser, peer-to-peer, I think we'd be living in a much more useful world. It's a lovely tech, but I never built anything serious around it. At this point, as a front-end web technology…
One thing that's always brought up in these discussions, because it's worth bringing up, is the file format of the macOS image editor Acorn: https://flyingmeat.com/acorn/docs/technotes/ACTN002.html Personally I use it a bunch in mobile and desktop apps.
That's quite fun to know! Although, it's also funny that I downloaded the demo .acorn files on a Mac and the OS has no idea how to open them without searching the App Store.
I feel like a JSON file would be more compact and easier to read, but wtf do I know. Harder to query, I guess?
Re: LumoSQL
#24Earlier quoted context omitted.
Most of my python projects use SQLite; 1 exception where i need multiprocessing access to the database and no locking problems and speed so i need to run the entire db in memory. https://docs.python.org/3/library/sqlite3.html The built in library makes it really quick and easy to use it. Whereas mysql or in my case id use postgresql if i needed a full db. You're looking for a third party library? I have used Psycopg…
I've used sqlite3 in node, and it's nice and performant for small cases, yes. Mostly I've used it for things small enough where a user could download an entire sqlite file of their data, and then re-upload it in case their data got lost. But ultimately this data gets stored in a true MySQL DB. I don't think I'd trust it to run a whole system with thousands of users and millions of entries... honestly, maybe my issue…
SQLite is almost certainly more battle tested by this point than even MySQL for things like this. Alongside having somewhere around the ballpark of 10¹² current deployments (yes, 1 trillion) it has about 600 lines of testing code per line of its actual source code.
https://www.sqlite.org/testing.html
To give you an idea of just how hardcore this is, they stress test something as fundamental as malloc() independently:
>SQLite, like all SQL database engines, makes extensive use of malloc() [...] On servers and workstations, malloc() never fails in practice and so correct handling of out-of-memory (OOM) errors is not particularly important. But on embedded devices, OOM errors are frighteningly common and since SQLite is frequently used on embedded devices, it is important that SQLite be able to gracefully handle OOM errors.
>OOM testing is accomplished by simulating OOM errors. SQLite allows an application to substitute an alternative malloc() implementation using the sqlite3_config(SQLITE_CONFIG_MALLOC,...) interface. The TCL and TH3 test harnesses are both capable of inserting a modified version of malloc() that can be rigged to fail after a certain number of allocations. These instrumented mallocs can be set to fail only once and then start working again, or to continue failing after the first failure. OOM tests are done in a loop. On the first iteration of the loop, the instrumented malloc is rigged to fail on the first allocation. Then some SQLite operation is carried out and checks are done to make sure SQLite handled the OOM error correctly. Then the time-to-failure counter on the instrumented malloc is increased by one and the test is repeated. The loop continues until the entire operation runs to completion without ever encountering a simulated OOM failure. Tests like this are run twice, once with the instrumented malloc set to fail only once, and again with the instrumented malloc set to fail continuously after the first failure.
I don't say this as a hater of MySQL! SQLite is built with very different constraints in mind. But data consistency is something it really shines at.
Re: LumoSQL
#25Earlier quoted context omitted.
Most of my python projects use SQLite; 1 exception where i need multiprocessing access to the database and no locking problems and speed so i need to run the entire db in memory. https://docs.python.org/3/library/sqlite3.html The built in library makes it really quick and easy to use it. Whereas mysql or in my case id use postgresql if i needed a full db. You're looking for a third party library? I have used Psycopg…
I've used sqlite3 in node, and it's nice and performant for small cases, yes. Mostly I've used it for things small enough where a user could download an entire sqlite file of their data, and then re-upload it in case their data got lost. But ultimately this data gets stored in a true MySQL DB. I don't think I'd trust it to run a whole system with thousands of users and millions of entries... honestly, maybe my issue…
[deleted]
Re: LumoSQL
#26Possibly the wrong place to ask this, but: I've played with SQLite when it was still available in-browser, and I felt that was on the brink of being a game-changer. If it was still supported in-browser and we had replication from the browser, peer-to-peer, I think we'd be living in a much more useful world. It's a lovely tech, but I never built anything serious around it. At this point, as a front-end web technology…
Two things: 1. There's almost certainly a port of Sqlite3 to WASM that would be more than glad to run in your browser. 2. I'd really love to know what applications fit in the "we had replication from the browser, peer-to-peer, I think we'd be living in a much more useful world" situation. We've had GunDB, IPFS, etc. that live in the browser for decades (and projects like Urbit), and the killer app just... doesn't see…
Heh. Well, #2, brilliant question. But no, I'm not thinking of anything as sexy as totally distributed filesystems. 15 years ago when I was into crypto and ran a bitcoin casino I would have had much bigger ideas for fully distributed DBs (which surely would have tanked and caused me ruin). Currently, I deal with a lot of site-specific software installations that run their own MySQL servers, some of which have unexpected downtime or go offline. I have a lot of custom code to align them with a master source of truth when they come back up. At least a few times a year, one of them gets so corrupted that I have to just login remotely and rebuild its database by hand. If I could have designed them to share a single database peer to peer, it would have saved me a lot of personal time.
There are probably a lot of hub-and-spoke systems like this flying way under the radar that would be a lot better if there were a reliable technology to keep them synchronized. I keep looking at Litestream and thinking about it.
Re: LumoSQL
#27[flagged]
[dead]
Re: LumoSQL
#28Earlier quoted context omitted.
I've used sqlite3 in node, and it's nice and performant for small cases, yes. Mostly I've used it for things small enough where a user could download an entire sqlite file of their data, and then re-upload it in case their data got lost. But ultimately this data gets stored in a true MySQL DB. I don't think I'd trust it to run a whole system with thousands of users and millions of entries... honestly, maybe my issue…
FWIW, I've been running a system with roughly 100K users, about 25 qps on average, with a single SQLite file for several years. No issues with data.
That's... pretty amazing. It sounds crazy to me, I'm obsessive about hourly backups, but do you use something like Litestream to keep copies?
Re: LumoSQL
#29Earlier quoted context omitted.
Are these things just running sql.js? Doesn't that use a kind of unstable webstorage instead of writing client-side files? I don't have a good handle on the state of SQLite these days as a way to store semi-permanent data on the client. In a locked-in environment or a backend I feel like it might make sense, but... isn't there like a 50Mb limit on localStorage, and how does that play nicely with a potentially larger…
There are bindings to sqlite for pretty much most languages out there. Not just webapps https://docs.python.org/3/library/sqlite3.html https://www.sqlite.org/cintro.html https://docs.rs/sqlite/latest/sqlite/ etc :)
huh. Sorry, but do some languages have SQLite bindings to some other executable? I thought that sql.js and sqlite3 in JS actually were SQLite in its entirety, running in script. You don't need to run anything else to make them work.
Re: LumoSQL
#30Earlier quoted context omitted.
There are bindings to sqlite for pretty much most languages out there. Not just webapps https://docs.python.org/3/library/sqlite3.html https://www.sqlite.org/cintro.html https://docs.rs/sqlite/latest/sqlite/ etc :)
huh. Sorry, but do some languages have SQLite bindings to some other executable? I thought that sql.js and sqlite3 in JS actually were SQLite in its entirety, running in script. You don't need to run anything else to make them work.
They have an interface to work with the sqlite (proper), which will be shipped alongside the application in question. sql.js is using sqlite in WASM/Emscripten, so it's pretty much analagous.