Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

211–220 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#211

Post by bjornsing »

yau8edq12i wrote: post #191
bjornsing wrote: post #168

Earlier quoted context omitted.

Well, it’s pretty simple: HaystackDB is designed from the ground up for write intensive workloads, so it’s much more economical than existing off-shelf-solutions for that type of workload. Is that not clear from the landing page? From pricing: “$0.2 per million writes, $20 per million reads”. The typical cost profile is $2 per million read/writes, or even more for writes.

Forgive me because what follows will sound harsh, but I think you need to hear it based on your response. > HaystackDB is designed from the ground up for write intensive workloads Okay. > so it’s much more economical than existing off-shelf-solutions for that type of workload. That's a leap in logic. Just because you designed it with this workload in mind, well, doesn't automatically mean that it's any good for this…

> Forgive me because what follows will sound harsh, but I think you need to hear it based on your response.

No worries. I appreciate you taking the time.

> you need proof that it's better than the competition, not a declaration, that it's better than the competition

Fair point. I realize I’ll need that before making any sales. But I was hoping to get a few leads from the contact form without it.

> Perhaps you're betting on your customers being stupid and not realizing that taking a 10x hit on the price of reads will lose them (and earn you) more money in the long run. After all, what good is writing to a DB if you never read from it...?

No it’s not a malicious trick. There are use-cases where most records will never be read back. For example, if you go into the Uber app you can find a history of all your trips and you can click one and bring up a receipt for it. Most users will rarely if ever do that. So you end up writing many more receipts to your database than what you’ll ever retrieve.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#212

Post by bastawhiz »

bjornsing wrote: post #186
bastawhiz wrote: post #170

Earlier quoted context omitted.

> $20 per million reads Quite frankly, this is not gonna work. I manage a system with a very write-heavy workload (lots of small writes) and even though our writes far outpace our reads, this pricing makes your system about ten times more expensive than an RDS cluster. There's no data about performance. There's no information on how or whether data is persisted to durable storage before a write is acknowledged. There…

All valid points. I guess I’m hesitant to put time into documentation and similar, if I can’t somehow find a steady stream of sales prospects. > even though our writes far outpace our reads, this pricing makes your system about ten times more expensive than an RDS cluster That indeed sounds off… Are you sure you’re comparing the total cost to that of an RDS cluster? I am aware that reads will be more expensive (due t…

My total database is maybe 400 gigs. Most of the writes overwrite existing data, so storage cost isn't a concern. With the cost of an upfront RI for the year on RDS (with basically as many iops as I can use), your solution gives me ~100 million reads. That's...like a month of usage at best.

At least I'm my case, the fundamental problem you're facing is that reads are just too expensive. Writes and reads tend to grow at the same pace in many products: there's a ratio that tends to stay the same as you scale. $20/million reads is just a _lot_. The ratio of writes to reads for your pricing needs to be 100:1 or more for it to make sense for me, but I'm more like 10-20:1.

> I guess I’m hesitant to put time into documentation and similar, if I can’t somehow find a steady stream of sales prospects.

This is part of why a database company is hard to build. You will simply not find anyone willing to give you money, because the alternative is going to be a solution your customers already know and understand and which is likely extremely mature. You're competing with Postgres and Mongo. You can't ship a database product that doesn't work: you're asking people to build on you for their storage primitive. If you fuck up, that's a business-ending event for your customer. You've either got to come to the table with an extremely compelling product ("I couldn't build my business without this") or you've got to show why someone should trust you over an established but somewhat more expensive alternative.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#213

Post by anonzzzies »

thangngoc89 wrote: post #43
sgt wrote: post #36

Earlier quoted context omitted.

How would you replicate that SQLite DB onto other hosts to achieve redundancy?

One could use Litestream [1] [1]: https://litestream.io

Any open source doing something similar ?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#214

Post by cj »

leoqa wrote: post #208
cj wrote: post #198

Earlier quoted context omitted.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…

Bitter story time: I made a config change to our AWS instances and projected approximately $10MM/year in AWS costs savings (pre-savings). My boss asked me "Who told you to do this? We need to focus on $project instead". I found another team and transferred out. 3 months later there was a big fire drill about AWS costs and they took my 1-pager and executed it. Didn't get any credit in the shipped email nor did the man…

The organization still got the end result, though (to play devil's advocate). That sounds like a win for the company. They got the cost savings, plus they redirected attention back to a project that was higher priority than saving $10m 3 months earlier than they could have.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#215

Post by bjornsing »

bastawhiz wrote: post #212
bjornsing wrote: post #186

Earlier quoted context omitted.

All valid points. I guess I’m hesitant to put time into documentation and similar, if I can’t somehow find a steady stream of sales prospects. > even though our writes far outpace our reads, this pricing makes your system about ten times more expensive than an RDS cluster That indeed sounds off… Are you sure you’re comparing the total cost to that of an RDS cluster? I am aware that reads will be more expensive (due t…

My total database is maybe 400 gigs. Most of the writes overwrite existing data, so storage cost isn't a concern. With the cost of an upfront RI for the year on RDS (with basically as many iops as I can use), your solution gives me ~100 million reads. That's...like a month of usage at best. At least I'm my case, the fundamental problem you're facing is that reads are just too expensive. Writes and reads tend to grow…

> The ratio of writes to reads for your pricing needs to be 100:1 or more for it to make sense for me

Correct. I bet Uber’s use case here is something like 1000:1. I’ve worked on systems that were over 1000000:1. That’s where HaystackDB makes sense.

> but I'm more like 10-20:1.

Then RDS is hard to beat.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#216

Post by password4321 »

xyst wrote: post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

https://news.ycombinator.com/item?id=38300425#38322311

> Uber is famous for NIH syndrome

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#217

Post by citizenpaul »

I think there is some reckoning of cloud service providers coming(assuming logical actors...). I was doing some contract work for a small place that had a GCP Bigtable that was costing $11k+ per month for some reports that were based on data from a 375MB !!! mysql db into big-table for the reports to run.

They hired some out of school data scientist to do reports and they were doing crazy ineffective things with the tiny dataset. Wanted me to fix it for pennies tomorrow and I declined.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#218

Post by alexchantavy »

gronky_ wrote: post #47
intunderflow wrote: post #41

Earlier quoted context omitted.

Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?

Yes, that’s definitely the main reason. It’s called “burying the lede”. Saving $6M is key information that makes this story interesting. It’s buried all the way at the bottom of the first blog and is completely missing from the second blog which focuses specifically on the migration

I'm usually guilty of this. The hands-on person involved in a highly technical project gets excited and bogged down in the details of the project that they end up not being the most compelling storyteller about it.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#219

Post by leadingthenet »

IanCal wrote: post #184
nvr219 wrote: post #180

Earlier quoted context omitted.

Loads fine for me with ublock. Perhaps you have a custom rule blocking something?

Nothing custom, so it must be on a list somewhere. edit - it doesn't have to really be blocking the actual post here even, if their loading code breaks when some other tracking code doesn't run, that could explain it.

I have the exact same problem, except on Uber Eats.
Post reply on HN