Why I Built Litestream • Hacker News Board

Re: Why I Built Litestream

#131

Post by liuliu » Fri, Feb 12, 2021, 4:21 AM UTC

polyrand wrote: post #98

Regarding SQLite's performance, some things I've found very useful: Use WAL mode (writers don't block readers): PRAGMA journal_mode = 'WAL' Use memory as temporary storage: PRAGMA temp_store = 2 Faster synchronization that still keeps the data safe: PRAGMA synchronous = 1 Increase cache size (in this case to 64MB), the default is 2MB PRAGMA cache_size = -64000 Lastly, use a modern version of SQLite. Many default inst…

In case you don't control which SQLite version you use (for example, on iOS or on Android, iOS normally comes with sane defaults), SQLite's default page size only recently changed to 4KiB, and update that can help with performance as well: https://www.sqlite.org/pgszchng2016.html

Re: Why I Built Litestream

#132

Post by chillfox » Fri, Feb 12, 2021, 4:29 AM UTC

devmunchies wrote: post #77

ibraheemdev wrote: post #34

Earlier quoted context omitted.

What crystal web framework do you use?

none. I had built a Crystal web framework that i retired[1]. I created something else that I use that's faster than the popular ones. Not open sourced. Funnily, been into F# the past few months and built a web framework for that too. Still in progress[2]. [1]: https://github.com/samueleaton/raze [2]: https://wiz.run/

Would you be willing to opensource that "something else" or if not then perhaps write a blog post about how it works?

Re: Why I Built Litestream

#133

Post by philip142au » Fri, Feb 12, 2021, 4:42 AM UTC

What is this doing, I mean, cannot append files in s3, so how does it store the objects?

Re: Why I Built Litestream

#134

Post by bobthebuilders » Fri, Feb 12, 2021, 4:42 AM UTC

mbreese wrote: post #89

xrendan wrote: post #83

Earlier quoted context omitted.

This is my first time hearing about H2, what's the benefit of H2 over sqlite?

Derby is/was the other alternative for an in JVM database. For a long time, it was possible/practical to use SQLite from Java. Now, it is, but not if you want to keep things as pure Java (and another commenter mentioned). But really, in my mind, that’s the only real benefit for H2, the fact that’s it’s pure Java. So if you need that, you’re good. But otherwise, I try to stick to SQLite.

I mean H2 is only really useful in Java, but the perf compared to sqlite in Java is equal or better due to better JDBC integration and not having to deal with JNI.

Re: Why I Built Litestream

#135

Post by chillfox » Fri, Feb 12, 2021, 5:28 AM UTC

sciurus wrote: post #126

With this approach, people have to start sharding their data as soon as one server can't handle their application and sqlite usage. Why not advocate instead for people to start out running their application and postgresql on the same server, with wal-e for backups to S3? That has almost the same benefits while everything fits on one server, but opens up alternative approaches to scaling if one day things no longer fi…

because sqlite is super simple to use and most things won't ever need to scale beyond one server.

Re: Why I Built Litestream

#136

Post by lahcim2000 » Fri, Feb 12, 2021, 5:38 AM UTC

Very awesome project!!!

Re: Why I Built Litestream

#137

Post by ignorantguy » Fri, Feb 12, 2021, 5:59 AM UTC

I use boltdb for all my personal projects. Thank you so much for writing it.

Re: Why I Built Litestream

#138

Post by jhgg » Fri, Feb 12, 2021, 6:43 AM UTC

benbjohnson wrote: post #18

avolcano wrote: post #12

I've been building a web service on a cheapo DigitalOcean box lately, so I'm excited to see explorations in this space, especially with an eye towards staying cheap! I'd probably only use this particular tool if it could hook up to Backblaze B2 instead of S3, since life's too short to ever have to engage with the hell that is AWS for a hobby project, but since B2's API-compatible it seems like a feature that could be…

I haven't tried Backblaze B2 but I agree with you on engaging in AWS hell. Litestream should work with any S3-compatible API. I've added an issue to add guides for B2 & Minio[1]. Regarding Postgres vs SQLite, I've found that I can use much simpler SQL calls with embedded databases when I don't need to worry about N+1 query performance issues. That makes many of the query features moot. That being said, there is a JSO…

A bit more literature on the N+1 situation in SQLite: https://www.sqlite.org/np1queryprob.html

Re: Why I Built Litestream

#139

Post by gigatexal » Fri, Feb 12, 2021, 6:55 AM UTC

Seems like something I want to play with. Max size of an object in S3 is 5TB so that’s plenty of head room.

Re: Why I Built Litestream

#140

Post by chubot » Fri, Feb 12, 2021, 6:56 AM UTC

awill wrote: post #129

"Solutions such as Kubernetes tout the benefits of zero-downtime deployments but ignore that their inherent complexity causes availability issues." This is completely accurate. I've seen several teams do kubernetes, only to both spend 50% of their dev time on ops, AND cause outages due to kubernetes complexity. They do this all while boasting about zero downtime deployments. It's comical really.

Well then you just need multiple Kubernetes clusters for redundancy :-) :-)

From another thread on the home page right now:

The current trend goes to multi-cluster environments, because it's way too easy to destroy a single k8s cluster due to bugs, updates or human mistake. Just like it's not an very unlikely event to kill a single host in the network e.g. due to updates/maintenance.

For instance, we had several outages when upgrading the kubernetes version in our clusters. If you have many small cluster it's much easier and more save to apply cluster wide updates, one cluster at a time.

https://news.ycombinator.com/item?id=26106353

:-( :-(