61–70 of 345 posts
Earlier quoted context omitted.
No it won't. sqlite "only" works with up to 281TB [0] [1] [0] https://www.sqlite.org/releaselog/3_33_0.html [1] https://www.sqlite.org/limits.html (#12)
You can split up into 10 SQLite DB's on this individual server.
Eventually all programs will be able to read email.
Earlier quoted context omitted.
One could use Litestream [1] [1]: https://litestream.io
What if a continuous replication system has a bug one day, and you realize you are just a bit corrupted and have to rerun? Or is it the same with cloud tools?
So did the engineers who proposed this get some kind of bonus considering how much money they saved the company?
The original[1][2] articles are a better read IMO. The link is just a summary of the two with added spelling and grammatical errors that materially impact the meaning. 1. https://www.uber.com/blog/how-ledgerstore-supports-trillions... 2. https://www.uber.com/blog/migrating-from-dynamodb-to-ledgers...
Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?
"Uber Migrates" (beginning: company that I'm interested in does something) "1T records" (middle: that's a lot of records; I wonder what happened) "from DynamoDB to LedgerStore" (hmm, how do they compare?) "to Save $6M Annually" (end: that's a good chunk of change for me, but was it worth it to Uber? Why did it save that amount? Let me read more)
It's a simple and engaging "there and back again" story that paves the way for a sequel.
Versus:
"How LedgerStore Supports Trillions of Indexes at Uber" (ah, okay, a technology supports trillions of indexes. Moving on to the next article in my feed)
"Migrating a Trillion Entries of Uber’s Ledger Data from DynamoDB to LedgerStore" (ah, a big migration. I'm not sure who did it or whether anything interesting came of it, or even whether it happened or is just theoretical because of the gerund, and moving one trillion of something is cool but not something I probably need to read about right now, so let's move on)
YMMV. Some probably prefer the more abstract/less narrative titles, but the first one is more of an attention grabber for me.
Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?
Earlier quoted context omitted.
No it won't. sqlite "only" works with up to 281TB [0] [1] [0] https://www.sqlite.org/releaselog/3_33_0.html [1] https://www.sqlite.org/limits.html (#12)
You can split up into 10 SQLite DB's on this individual server.
1 is a scalable, managed, highly available service, with economies of scale the other is a fixed size, capital expenditure with fixed performance, limited DR, requiring a couple of SRE/DevOps and colo
There is also the will it always work question
So did the engineers who proposed this get some kind of bonus considering how much money they saved the company?
Earlier quoted context omitted.
What if a continuous replication system has a bug one day, and you realize you are just a bit corrupted and have to rerun? Or is it the same with cloud tools?
That's why you always test your backup. I backup the full sqlite.db every day and test the litestream replication every week. So far litestream have been solid.
How do you detect restored but bit flipped data ?
Earlier quoted context omitted.
If you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.
> or with backups take a while to restore a PB and a way to take a hot backup without impacting the service that by itself is a task or snapshots which is more disk > 1:1 replication Depending on the amount of writes could be a ton of extra disk and a bucket for network cost
Except network cost there's no extra disk required. It's just broadcasted writes consumed on the other hand.
These boxes are not dumb JBODS. They support their own replication/backup subsystems, so everything is transparent.