Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

41–50 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#41

The original[1][2] articles are a better read IMO. The link is just a summary of the two with added spelling and grammatical errors that materially impact the meaning. 1. https://www.uber.com/blog/how-ledgerstore-supports-trillions... 2. https://www.uber.com/blog/migrating-from-dynamodb-to-ledgers...

Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#42
post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

Sometimes things just aren't nails, even when you have a really good hammer.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#43
post #36
post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

How would you replicate that SQLite DB onto other hosts to achieve redundancy?

One could use Litestream [1]

[1]: https://litestream.io

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#44

Does no one ever delete data? It's hard to believe there's much business value in keeping every individual payment record dating back to 2017.

I'm not sure if they have regulatory obligations to keep them, or what, but it still seems like you could back them up to cold storage after a reasonable period of time.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#45

Earlier quoted context omitted.

If a project fails, do you pay for the loss since you want a share of the profits as well?

Someone probably gets fired so I guess someone does pay the ultimate price.

Losing your job because the outcome of your efforts (or even external events) is not what I would call the ultimate price.

"The metaverse division has now lost more than $45 billion since the end of 2020"

Your compensation for your work is your salery. So I would say that it's fair that the actual risk taker is benefiting from the potential rewards?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#46

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

Looking at pricing, it's crazy expensive (and that comparing to AWS, which is crazy expensive). How do you justify that?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#47

The original[1][2] articles are a better read IMO. The link is just a summary of the two with added spelling and grammatical errors that materially impact the meaning. 1. https://www.uber.com/blog/how-ledgerstore-supports-trillions... 2. https://www.uber.com/blog/migrating-from-dynamodb-to-ledgers...

Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?

Yes, that’s definitely the main reason. It’s called “burying the lede”.

Saving $6M is key information that makes this story interesting. It’s buried all the way at the bottom of the first blog and is completely missing from the second blog which focuses specifically on the migration

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#48
post #3

Earlier quoted context omitted.

1T "records". Any given transaction can have N records. I'm assuming this includes Uber Eats as well.

Still, they have 10B rides in 2023 including Eats, say 75-100B since inception. What would be a record such that each transaction needs 10-15 on average?

I can see that as transactions with credit cards go through lots of process (withholding, approval, charging, settling, etc..)

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#49

I read the article so I roughly know what LedgerStore is - but I have no idea where it is hosted.

From one of the original sources linked in this thread

> LSG promised shorter indexing lag (i.e., time between when a record is written and its secondary index is created). Additionally, it would give us faster network latency because it was running on-premises within Uber’s data centers.

https://www.uber.com/en-AU/blog/migrating-from-dynamodb-to-l...

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#50

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?

> I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance I'm guessing they know a lot about their costs, and you know very little. There's little value in insulting the team members like this.

It’s not insulting to speculate in a conversational way around the errors we very very commonly see
Post reply on HN