Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

301–310 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#301

Post by leoqa »

heavenlyblue wrote: post #233
leoqa wrote: post #208

Earlier quoted context omitted.

Bitter story time: I made a config change to our AWS instances and projected approximately $10MM/year in AWS costs savings (pre-savings). My boss asked me "Who told you to do this? We need to focus on $project instead". I found another team and transferred out. 3 months later there was a big fire drill about AWS costs and they took my 1-pager and executed it. Didn't get any credit in the shipped email nor did the man…

Of course you didn't. You used your time to promote yourself instead of doing what you were asked to do instead. That could have cost a promotion for your manager who could have promoted you.

We were migrating to Kubernetes from fixed hosts and had to buy the capacity. I just spent a day looking at our utilization and made a better decision about what instance type to use.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#302

Post by smileysteve »

amluto wrote: post #251
xyst wrote: post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics. If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly…

This. DynamoDB will export to parquet (or csv); s3 select becomes possible for data. More importantly, Athena can query it all

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#303

Post by tuananh »

ec109685 wrote: post #290
influx wrote: post #106

I would gladly pay 6 million/year to not be on call, and have to worry about things like bios and ssd firmware ever again.

They didn’t say they are running this on prem.

> Additionally, it would give us faster network latency because it was running on-premises within Uber’s data centers.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#304

Post by klodolph »

cj wrote: post #214
leoqa wrote: post #208

Earlier quoted context omitted.

Bitter story time: I made a config change to our AWS instances and projected approximately $10MM/year in AWS costs savings (pre-savings). My boss asked me "Who told you to do this? We need to focus on $project instead". I found another team and transferred out. 3 months later there was a big fire drill about AWS costs and they took my 1-pager and executed it. Didn't get any credit in the shipped email nor did the man…

The organization still got the end result, though (to play devil's advocate). That sounds like a win for the company. They got the cost savings, plus they redirected attention back to a project that was higher priority than saving $10m 3 months earlier than they could have.

It may be a win for the company in some categories, but a loss in others.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#305

Post by Retric »

omeid2 wrote: post #297
Retric wrote: post #136

Earlier quoted context omitted.

Economies of scale generally have upper limits. Often when you approach the largest scale the existing market will supply you essentially need to become your own supplier which then runs into span of control issues. The organization needs to become competitive in that new market or their costs increase. Keep scaling and eventually vertical integration ends up looking like a Soviet style planned economy. Your remote m…

Yes, there is an upper limit, and that is the entirety of global economy. But that assertion is as useless as it is true.

Hardly the global economy, even approaching a global monopoly for something tends to hit such limits. Few companies deal with this today because most markets are fragmented.

But even at the 30% marketshare the iPhone has been dealing with these issues for a while. They just can’t buy 200 million volume buttons or whatever off the shelf. Now imagine what happens if one of their suppliers would fail days before the phone launches. They have such tight integration not just with the manufacturer’s process but also their finances because Apple simply can’t get replacements at scale on short notice. And remember that’s at ~30% market share, it just gets worse after that.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#306

Post by bjornsing »

erik_seaberg wrote: post #260
bjornsing wrote: post #215

Earlier quoted context omitted.

> The ratio of writes to reads for your pricing needs to be 100:1 or more for it to make sense for me Correct. I bet Uber’s use case here is something like 1000:1. I’ve worked on systems that were over 1000000:1. That’s where HaystackDB makes sense. > but I'm more like 10-20:1. Then RDS is hard to beat.

You might be surprised how often a deep graph of microservices ends up rereading the same prior transactions over the course of stateful payment processing and on-demand payouts. DynamoDB can give you 2.6 million short reads of base load (1 RCU/s provisioned) for $0.12 per month, which would make a $65 alternative (2.6 * $5 + 2.6 * $20) a hard sell.

Typically you have a “hot-cold” architecture where transactions stay in “hot” storage for a number of days (I think 12 days in the OP if I remember correctly), then are moved to “cold” storage. HaystackDB is primarily designed for the “cold” portion of that architecture. But I am considering adding an open source / source available “frontend” that would provide low-cost “hot” storage.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#307

Post by sokoloff »

shawabawa3 wrote: post #243
cj wrote: post #198

Earlier quoted context omitted.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…

> we spent $25m of engineering resources to save $6m annually This is a huge ROI. Borrowing $25m costs about $1.25m/yr so you're winning even with no upfront costs

That math assumes the on-going maintenance of LedgerStore is no higher than that of using DynamoDB was.

I suspect there's now a team of technologists maintaining, securing, and operating LedgerStore.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#308

Post by citizenpaul »

remus wrote: post #279
citizenpaul wrote: post #217

I think there is some reckoning of cloud service providers coming(assuming logical actors...). I was doing some contract work for a small place that had a GCP Bigtable that was costing $11k+ per month for some reports that were based on data from a 375MB !!! mysql db into big-table for the reports to run. They hired some out of school data scientist to do reports and they were doing crazy ineffective things with the…

Not that I disagree with your overall point, but I don't think this > I was doing some contract work for a small place that had a GCP Bigtable that was costing $11k+ per month for some reports that were based on data from a 375MB !!! mysql db into big-table for the reports to run. Is a good example. It's just a badly architected system, and you'd have exactly the same problem if you were running the same thing on a m…

I think its a perfect example that cloud providers are profiting on a unsustainable scheme. They rely on their services papering over (literally with money) lack of skill/knowlege of their customers. This tactic either kills the customer or creates someone that is desperate to no longer be your customer.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#309

Post by chii »

cj wrote: post #198
xyst wrote: post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…

$25m spend to save $6m annually is a return on investment of 4 (and a bit) years to break even, then it's positive afterwards. So as long as the project runs for more than 4 years, they'd be ahead.

The only problem is that the database code might need maintenance, specs change or situations change, and the cost is _not_ 25m, but way more per year than anticipated. So much so that after a few years, a new batch of engineers start eyeing writing a new database system to purportedly save money...

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#310

Post by rubiquity »

PeterZaitsev wrote: post #239
SkyMarshal wrote: post #236

It seems LedgerStore is not open source [1], and finding any info on it requires following a trail of backlinked Uber blog posts. Here's one with the most info on LedgerStore that I can find, from 2021: https://www.uber.com/en-US/blog/dynamodb-to-docstore-migrati... [1]: https://github.com/uber

Yeah. This looks like some internal solution. In general Uber seems to be high on "not invented here" scale - they like to conclude no existing Open Source solutions are good enough for them and they need to build their own... this is different from Facebook approach for example which chose to made MySQL better by adding MyRocks/RocksDB to it and keep them Open Source.

Don't worry it's 99% likely LedgerStore is built on top of MySQL.
Post reply on HN