Earlier quoted context omitted.
Would Redis be a valid solution for your case?
This adds another service you now need to deploy and maintain. For small personal projects, I definitely would like to avoid that if I can!
I agree.
Consider SQLite
191–200 of 274 posts
Re: Consider SQLite
#192I've been doing some ETL exploration with opening a sqlite :memory: connection, ingesting small-to-medium data, and then doing
"VACUUM INTO somefile.sqlite;"
to dump the RAM copy to disk.
What a great tool.
Re: Consider SQLite
#193Earlier quoted context omitted.
With WAL, writes for something like a ToDo app finish in a small fraction of a millisecond so unless your todo webapp is writing to the DB at a rate exceeding 20k writes per second, the fact that writes are not concurrent becomes largely irrelevant.
This can't be right. As far as I can tell, WAL allows concurrent READS and WRITE, not concurrent WRITES. Am I doing this wrong all these years?
That's correct, it doesn't allow concurrent writes, but if the writes finish fast enough, that's somewhat academic.
Re: Consider SQLite
#194Also, consider just using the filesystem with binary encoding. Serialize / deserialize directly from / to data structures. It is faster and simpler than any database you'll ever use, assuming you don't need the functionality a database provides.
Re: Consider SQLite
#195Earlier quoted context omitted.
I'm pretty sure you do need FB and Azure level infra for you recipe app. I've read a couple blog posts, watched a video on YouTube and copied the code from Stack Overflow, so I'm pretty much an expert on this, so trust me.
My recipe application is going to be written in a purely functional language with CQRS and a message bus that provides "guaranteed once" determinism. I should be able to spin up no more than a half dozen containers on two VM's to handle this architecture. The database will probably be redundant MSSQL and Oracle (just in case one technology is fundamentally broken). Database writes will be proxied by a Redis instance.…
> The database will probably be redundant MSSQL and Oracle (just in case one technology is fundamentally broken)
Only one technology?
Re: Consider SQLite
#196Earlier quoted context omitted.
> How is this setup fault tolerant? It is not. > What happens if there is a hardware failure? The product would suffer a total outage until manual intervention takes place. A restore of the VM from snapshot would be carried out by the customer. Some loss of the most recent business data would occur (i.e. between latest snapshot and time of failure). All of this is understood and agreed to by our customers. > How do y…
Interesting. For an extremely specific use case and with users who understand and accept the caveats of this approach I'm sure it would work well enough. The most confusing thing to me is that there is apparently an intersection of users who are ok with an outage and data loss with users who want a product which can > execute queries and reliably receive results within microseconds What is your product? Who are these…
> users who are ok with an outage and data loss
It depends on how you couch it. Working with large fleets of cloud servers has jaded us (well, me) into forgetting just how high you can get the uptime of a single server or well-managed VM if you're careful.
Sure, power cords get tripped over and hardware fails sometimes, but I'd imagine a lot of users who do care about reliability are making totally reasonable judgements that single points of failure in their systems are acceptable if the SPOFs are hardened and the oh-shit plan (restoring from a VM snapshot in this example) is something that they have confidence in.
Heck, given the availability issues that arise from the complexity of redundant/HA systems, I think there are probably plenty of seasoned enterprise customers who prefer hardened SPOFs managed by experts over distributed systems, especially once the dimension of cost enters into the discussion.
Re: Consider SQLite
#197There are some important things that SQLite does not do. It is not client/server; a process must be able to fopen() the database file. NFS and SMB are options that can convey access to remote systems, but performance will not likely be good. Only a single process can write to the database at any time; it does not support concurrent writers. The backup tools do not support point-in-time recovery to a specific past tim…
Note -- the single process write at any one time is a killer for most web apps, where for example within SaaS you have many users doing things at the same time.
Even in write-heavy environments, I've used SQLite effectively with multiple thousands of concurrent (as in logged in and changing things at the speed of human site interactions, not as in simultaneous web requests at the same instant) users. In this context, SQLite is effectively a write linearizer.
With appropriate pragmas and database handles stored persistently between requests in server processes, the rate (in bytes of data affected by INSERT/UPDATE/DELETE) at which users could make DB changes got pretty darn close to the rate at which a file could be sequentially written on the filesystem, which is more than enough aggregate throughput for 1000s of concurrent non-bulk changes in most web applications, I'd imagine.
Re: Consider SQLite
#198Earlier quoted context omitted.
It's not really an issue if you have 1 db per customer
If a customer has 1000 employees all using your app, it is.
Not always. In fact, I'd strengthen that and say that this is not a problem in general/in most cases with thousands of concurrent users. See my other comment one level up.
Re: Consider SQLite
#199There are some important things that SQLite does not do. It is not client/server; a process must be able to fopen() the database file. NFS and SMB are options that can convey access to remote systems, but performance will not likely be good. Only a single process can write to the database at any time; it does not support concurrent writers. The backup tools do not support point-in-time recovery to a specific past tim…
None of those matter to me, but I'm being forced to switch from SQLite simply because of its lack of uint64 support.
Being forced as in you can't make things work without that support? If so, is parsing/processing blobs infeasible for some reason?
Re: Consider SQLite
#200I'm exactly at a point where I'm considering SQLite for its single file db advantage, but I'm struggling to find solutions for my use case. I need to import some 30k JSONs of external monitor data from Lunar ( https://lunar.fyi ) into a normalized form so that everyone can query it. I'd love to get this into a single SQLite file that can be served and cached through CDN and local browser cache. But is there something…
Check out datasette: https://datasette.io/