Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

31–40 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#31

So did the engineers who proposed this get some kind of bonus considering how much money they saved the company?

If a project fails, do you pay for the loss since you want a share of the profits as well?

Someone probably gets fired so I guess someone does pay the ultimate price.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#32

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

One observation I have regarding your homepage is that the message isn't very clear. The headline doesn't mention any benefits I get from using your software.

I think you should invest some time into improving your landing page and maybe you may see some traction. A good resource for this which I've bookmarked is here(1). Hope that helps.

(1) https://www.indiehackers.com/post/my-step-by-step-guide-to-l...

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#33
post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

No it won't. sqlite "only" works with up to 281TB [0] [1]

[0] https://www.sqlite.org/releaselog/3_33_0.html

[1] https://www.sqlite.org/limits.html (#12)

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#34
post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

Also a bit scary to have a system without a scaling mechnism built-in in the path of customer traffic. At some point you may be racing to upgrade it.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#35
post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

No it won't. sqlite "only" works with up to 281TB [0] [1] [0] https://www.sqlite.org/releaselog/3_33_0.html [1] https://www.sqlite.org/limits.html (#12)

You can split up into 10 SQLite DB's on this individual server.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#36
post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

How would you replicate that SQLite DB onto other hosts to achieve redundancy?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#37
post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

it will take forever to create that index. Link describes 10B rows dataset.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#38

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?

At one end of the spectrum, some people here claim to write this kind of software over a weekend. Some others claim they require a salary of $600,000, and still need nine additional colleagues to pull something like this off.

There is a lot of room in between, where cost estimates are more realistic.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#40
post #29
post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB. I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.

If you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.
Post reply on HN