Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why I Built Litestream

litestream.io

Post reply on HN
Print view View original

81–90 of 178 posts

Re: Why I Built Litestream

#81

Post by notacoward »

Sayrus wrote: post #49
benatkin wrote: post #44

"Litestream is a tool that runs in a separate process and continuously replicates a SQLite database to Amazon S3." Why Amazon S3? Was it a huge invention by people at Amazon, and even if so, does it assign credit where it is deserved? Is Amazon helping the open web? I don't think so.

They did invent it and name it like this. Is there an issue with calling it Amazon S3? Can't they get credit for it? Even software such as Ceph calls themselves compatible with "Amazon S3 API".

They did not invent object storage so much as popularize it and standardize the terminology we now use. S3 was announced in 2006. By then, many other distributed object stores had already existed for years, mostly for purely archival use and often a bit enterprise-ish (e.g. FilePool/Centera in 2002). Depending on how much similarity you require, you could even trace lineage back to NASD in 1995. There's also a closely related space of P2P applications which had different goals but similar APIs and implementation details. For example, Freenet and Gnutella both started in 2000. It's practically certain that some techniques worked out in that space informed the design of S3.

Unfortunately Amazon has not done a very good job crediting predecessors, but they do deserve credit for bringing what had previously been rather niche ideas (I know because I was there throughout) to the masses.

Re: Why I Built Litestream

#82

Post by pbowyer »

cpach wrote: post #59
mplewis wrote: post #54

Earlier quoted context omitted.

If the side business makes $200k/mo, that usually means the revenue of the side business is $200k/mo.

Another non-native speaker here. I was also confused by this wording.

I'm a native speaker and it confused me to. If I make $50k, I mean I am taking home $50k If I've revenue of $50k, I am taking home less than $50k

Re: Why I Built Litestream

#83

Post by xrendan »

grandinj wrote: post #72

Embedded databases where the db is a single file are the bomb. We build several products in Java using H2 as our database. What a pleasure. Full SQL support, zero configuration/installation/etc. Just copy data-files/dbs around using normal file tools. Just start up one process and your app is ready.

This is my first time hearing about H2, what's the benefit of H2 over sqlite?

Re: Why I Built Litestream

#84

Post by mmcdermott »

xrendan wrote: post #83
grandinj wrote: post #72

Embedded databases where the db is a single file are the bomb. We build several products in Java using H2 as our database. What a pleasure. Full SQL support, zero configuration/installation/etc. Just copy data-files/dbs around using normal file tools. Just start up one process and your app is ready.

This is my first time hearing about H2, what's the benefit of H2 over sqlite?

I think the big one is that it's written in Java so it runs seamlessly on the JVM. While JNI allows you to wrap native libraries for the JVM, it tends to mix poorly with the common artifact management and deployment pipelines.

The main use I've put it to over the years is as an in-memory database for integration tests.

Re: Why I Built Litestream

#85

Post by pbowyer »

bob1029 wrote: post #24

> “But nobody writes production applications with SQLite, right?" We've been doing it for 5 years now. Basic tricks we employ are: Use PRAGMA user_version for purposes of managing automatic migrations, a. la. Entity Framework. This means you can actually do one better than Microsoft's approach, because you don't need a special unicorn table to store migration info. A simple integer compared with your latest integer a…

> We can afford to lose the last few minutes of work without anyone getting yelled at. Some modern virtualization technologies do help a lot in this regard. Running bare metal you need to be a little more careful.

Can you say more about how (and which) modern virtualization technologies help? RTO is something I've never found a happy to, since piecing together any missing data is painful, but avoiding a clustered DB setup (or the cost of Aurora) is always welcome.

Re: Why I Built Litestream

#86

Post by benbjohnson »

aidenn0 wrote: post #63
benbjohnson wrote: post #61

Earlier quoted context omitted.

This is take seems a little intellectually dishonest. The very next sentence is, "this works particularly well for SaaS applications where each customer is isolated from one another." Sharding doesn't work for all applications. Social networks are one example where all data can potentially interact with all other data. Sarcasm noted though. :)

Social networks are also write-heavy so probably not a good fit for SQLite anyways, right?

Social networks have a lot of lurkers who mostly read so I'm not sure about the balance of reads vs writes.

Re: Why I Built Litestream

#87

Post by jitl »

benbjohnson wrote: post #32
bob1029 wrote: post #24

> “But nobody writes production applications with SQLite, right?" We've been doing it for 5 years now. Basic tricks we employ are: Use PRAGMA user_version for purposes of managing automatic migrations, a. la. Entity Framework. This means you can actually do one better than Microsoft's approach, because you don't need a special unicorn table to store migration info. A simple integer compared with your latest integer a…

Those are awesome tips. I didn't even think of using "user_version" for storing a migration version. I'm definitely stealing that trick.

This is how we manage SQLite migrations in Notion’s native apps.

Re: Why I Built Litestream

#88

Post by thaumasiotes »

pbowyer wrote: post #82
cpach wrote: post #59

Earlier quoted context omitted.

Another non-native speaker here. I was also confused by this wording.

I'm a native speaker and it confused me to. If I make $50k, I mean I am taking home $50k If I've revenue of $50k, I am taking home less than $50k

> If I make $50k, I mean I am taking home $50k If I've revenue of $50k, I am taking home less than $50k

That is not normal native usage. If you make $50k, you're salaried at $50k, but you take home considerably less than that.

Re: Why I Built Litestream

#89

Post by mbreese »

xrendan wrote: post #83
grandinj wrote: post #72

Embedded databases where the db is a single file are the bomb. We build several products in Java using H2 as our database. What a pleasure. Full SQL support, zero configuration/installation/etc. Just copy data-files/dbs around using normal file tools. Just start up one process and your app is ready.

This is my first time hearing about H2, what's the benefit of H2 over sqlite?

Derby is/was the other alternative for an in JVM database.

For a long time, it was possible/practical to use SQLite from Java. Now, it is, but not if you want to keep things as pure Java (and another commenter mentioned). But really, in my mind, that’s the only real benefit for H2, the fact that’s it’s pure Java. So if you need that, you’re good.

But otherwise, I try to stick to SQLite.

Re: Why I Built Litestream

#90

Post by benbjohnson »

cfors wrote: post #67
benbjohnson wrote: post #61

Earlier quoted context omitted.

This is take seems a little intellectually dishonest. The very next sentence is, "this works particularly well for SaaS applications where each customer is isolated from one another." Sharding doesn't work for all applications. Social networks are one example where all data can potentially interact with all other data. Sarcasm noted though. :)

Awesome product, btw. Definitely see where it fits some niches. Actual concerns about running a real application with real uptime requirements. 1. Say your EC2 or docker container that's hosting this goes down. Is that left up to the user to deal with? RDS handles this for you 2. No ACID transactions if you ever outgrow a DB. You talk about in your pitch that the vertical scaling, so you have to just keep bumping the…

> Awesome product, btw. Definitely see where it fits some niches.

Thanks! I appreciate it.

> Say your EC2 or docker container that's hosting this goes down. Is that left up to the user to deal with? RDS handles this for you

Yes, that's out of scope for Litestream since there are a lot of ways to manage that depending on your application. I agree that RDS wins here for simplicity.

> No ACID transactions if you ever outgrow a DB. You talk about in your pitch that the vertical scaling, so you have to just keep bumping the VPS/container memory.

You still have ACID transactions but they are just per-shard. For a SaaS application, that seems reasonable since they're localized to the customer (assuming you're sharding by customer).

> Sure a SaaS application where a customer specific DB is isolated, but as soon as you hit any _real_ scaling limits you immediately are back to the entire problem statement you are aiming to (at least you hint at that in your pitch) solve which is the crazy n-tier architectures we have. [...] There are entire hosts of problems that you call out you are trying to solve without providing any real solution.

I'm not trying to solve an infinite scaling problem. If you're seeing sustained 100K request/sec on your application then you'll need specific solutions. But I'd argue that 98% of applications never come near that threshold and those are the applications that could benefit from simpler architecture.

Thanks for all the feedback. I hope I'm not coming off as argumentative.

Post reply on HN