LiteFS Cloud: Distributed SQLite with Managed Backups • Hacker News Board

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#91

Post by catgirlinspace » Thu, Jul 06, 2023, 8:05 PM UTC

benbjohnson wrote: post #53

catgirlinspace wrote: post #51

Earlier quoted context omitted.

Gotcha. Would performance be better with LiteFS? Think I saw someone mention that because there'd be no network requests between machines.

It's hard to say definitively. If you're issuing multiple database queries per request and you aren't using really complicated Postgres SQL then I would guess that you'll see a performance boost. In tests I've done with Postgres, I see a ~1ms round trip between the application server and database server—even when both are in the same region. If you run 10 queries to serve a single HTTP request then that's 10ms right…

Hm, will definitely have to look into it then as I’m pretty bad at making efficient queries and use a lot sometimes just have to get over my fear of like, SQLite being only for client and testing stuff and not servers…

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#92

Post by gunapologist99 » Fri, Jul 07, 2023, 2:07 PM UTC

tptacek wrote: post #76

superq wrote: post #74

Earlier quoted context omitted.

so... no to those questions?

Would I build a billion-record Facebook replacement in SQLite? Probably not. I probably wouldn't try to scale it up on vanilla Postgres, either.

Everything I listed at least claims to be a distributed database (also in the title of this article: "distributed sqlite").

However, Postgres is not and never claimed to be a distributed database, so you are erecting and attacking a strawman argument by choosing something that wasn't even in the list.

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#93

Post by [deleted] » Sat, Jul 15, 2023, 5:36 PM UTC

bjyule wrote: post #67

Spunkie wrote: post #6

I'm really glad to see this complete solution drop. I've cobbling together sqlite point-in-time restores, automatic snapshots, and S3 backups with litestream and scripts for too long. I've been bugging cloudflare to do this with D1+tunnels since D1 was announced and they constantly seemed confused what I was even talking about.

What do you have in mind with respect to D1+Tunnels? We're launching PITR support for the new experimental backend within the next 1-2 weeks. Stay tuned. (I am the eng director for Cloudflare Workers)

[deleted]

Re: LiteFS Cloud: Distributed SQLite with Managed Backups

#94

Post by Spunkie » Sat, Jul 15, 2023, 5:39 PM UTC

bjyule wrote: post #67

Spunkie wrote: post #6

I'm really glad to see this complete solution drop. I've cobbling together sqlite point-in-time restores, automatic snapshots, and S3 backups with litestream and scripts for too long. I've been bugging cloudflare to do this with D1+tunnels since D1 was announced and they constantly seemed confused what I was even talking about.

What do you have in mind with respect to D1+Tunnels? We're launching PITR support for the new experimental backend within the next 1-2 weeks. Stay tuned. (I am the eng director for Cloudflare Workers)

I wanted this when D1 was first teased, and before I figured out that D1 is a database product that happens to be built on SQLite rather than SQLite being the star of the D1 show. Nonetheless, what I want is to be able to point a CF tunnel at a local SQLite file and instantly be able to leverage all the benefits of D1.

I should be able to...

    1. Configure a CF tunnel through the config file or the dash, pointing it to a local path of an existing SQLite file.

        1a. On startup, the SQLite file should create and link a new D1 db.
    2. Configure a CF tunnel through the config file or the dash, pointing it to an existing D1 db.

        2a. On startup, the local sqlite db should be either overwritten or created.
    3. Create read replicates through CF tunnels on another server or the local computer/wrangler for development.

    4. Have automatic global read replicas on the edge, which is inherent to D1 no?

    5. Have automatic/rolling backups and export and import those backups automatically with S3/R2.

    6. Do a PITR through the dash or CF tunnel CLI.

        6a. PITR should be possible to either the existing live DB or restoring it into a separate new D1 copy.
    7. Leverage my SQlite files in workers, another automatic bonus of connecting it to D1.

There are likely plenty of more cool things that could be done if D1 could be exposed as a normal SQLite file. Ultimately, CF tunnels are just in the mix because it seems like the obvious choice for plumbing SQLite files into and out of the CF network/edge.