Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

191–200 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#191

Post by yau8edq12i »

bjornsing wrote: post #168
macspoofing wrote: post #150

Earlier quoted context omitted.

>I guess what I’m trying to say is that I was hoping that someone with a write intensive workload would want to spend some time evaluating a product built specifically for that. Again, it's not clear to me exactly what it is you're doing that's any different from the plethora of existing off-the-shelf solutions. You're saying that you started this project/company because you were looking for a solution to a specific…

Well, it’s pretty simple: HaystackDB is designed from the ground up for write intensive workloads, so it’s much more economical than existing off-shelf-solutions for that type of workload. Is that not clear from the landing page? From pricing: “$0.2 per million writes, $20 per million reads”. The typical cost profile is $2 per million read/writes, or even more for writes.

Forgive me because what follows will sound harsh, but I think you need to hear it based on your response.

> HaystackDB is designed from the ground up for write intensive workloads

Okay.

> so it’s much more economical than existing off-shelf-solutions for that type of workload.

That's a leap in logic. Just because you designed it with this workload in mind, well, doesn't automatically mean that it's any good for this workload (or any workload). If solving a problem was as easy as declaring "I will design my solution from the ground up for this problem", then we'd all live in peace and harmony. So that's what people are asking you here: how do you make your DB "much more economical" for that type of workload? What technology, what ideas have you had to make it possible? If you don't want to reveal that, then you need proof that it's better than the competition, not a declaration, that it's better than the competition.

> Is that not clear from the landing page?

It's clear that you want to market your solution as something good for write-heavy workloads. Why should we believe you've done a good job designing your solution?

> From pricing: “$0.2 per million writes, $20 per million reads”. The typical cost profile is $2 per million read/writes, or even more for writes.

Who knows how you came up with pricing? Perhaps you're betting on your customers being stupid and not realizing that taking a 10x hit on the price of reads will lose them (and earn you) more money in the long run. After all, what good is writing to a DB if you never read from it...? Or perhaps it's some kind of promotional / loss leader pricing that will change soon in the future. In any case, it's, again, not proof that your solution is adapted to the customer's problem.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#192

Post by cess11 »

geodel wrote: post #119

More power to them. At this point even technically decent teams/companies have given up on developing large, complex systems in favor of SaaS. After carefully evaluating our strategic course of action answer always is AWS. Its only team who propose alternative they have to justify rigorously how come they differ in conclusion.

Not if you're in the EU, due to, among other things, Schrems II.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#193

Post by Closi »

_xnmw wrote: post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

1.7 petabytes on Sqlite?

Sqlite's own advice:

> If your data will grow to a size that you are uncomfortable or unable to fit into a single disk file, then you should select a solution other than SQLite. SQLite supports databases up to 281 terabytes in size, assuming you can find a disk drive and filesystem that will support 281-terabyte files.

> Even so, when the size of the content looks like it might creep into the terabyte range, it would be good to consider a centralized client/server database [over SQLite].

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#194

Post by Closi »

bayindirh wrote: post #40
kondro wrote: post #29

Earlier quoted context omitted.

Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB. I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.

If you install a RAID controller and a couple of disk boxes, it's possible with 1:1 replication, or with backups. 60 disk 3.5" units already exist, so 2.5" SSD racks. It won't be cheap, but will be resilient and fast. Bloody fast if you have the budget.

Resilient and fast from a disk perspective, but in practice massively bottlenecked by the fact that Sqlite can only have 1 writer at a time.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#195

Post by renegade-otter »

smokel wrote: post #38
sha_r_roh wrote: post #6

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?

At one end of the spectrum, some people here claim to write this kind of software over a weekend. Some others claim they require a salary of $600,000, and still need nine additional colleagues to pull something like this off. There is a lot of room in between, where cost estimates are more realistic.

Plenty of things can be prototyped over a weekend, but many will require months and even years to get production-ready, feature-complete, and useful, especially at scale.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#196

Post by chasil »

sgt wrote: post #35
BlackLotus89 wrote: post #33

Earlier quoted context omitted.

No it won't. sqlite "only" works with up to 281TB [0] [1] [0] https://www.sqlite.org/releaselog/3_33_0.html [1] https://www.sqlite.org/limits.html (#12)

You can split up into 10 SQLite DB's on this individual server.

Beware of WAL mode, as you sacrifice ACID in this configuration.

https://sqlite.org/lang_attach.html

'Transactions involving multiple attached databases are atomic, assuming that the main database is not ":memory:" and the journal_mode is not WAL. If the main database is ":memory:" or if the journal_mode is WAL, then transactions continue to be atomic within each individual database file. But if the host computer crashes in the middle of a COMMIT where two or more database files are updated, some of those files might get the changes where others might not.'

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#197

Post by admax88qqq »

rapht wrote: post #116
scop wrote: post #91

Earlier quoted context omitted.

That was where my mind went to when I saw the headline. Granted that while I’m not on the Finance side of things and am in fact a developer, “six million” didn’t seem like much at all considering engineer salaries. It’s certainly an achievement, but at what short and long term salaried cost?

A primer on valuation: in many financial contexts, $1 of operating savings may be worth much more than $1 of investment. That is because an investment is a one-off, so it's actually worth $1, but the savings are recurring, so they are worth the same number of years that a company's profits are valued. Depending on sector and investors' beliefs in the future of companies, this factor is typically in the 5-20x range. T…

If that $1 of investment doesn't yield any returns that's not an investment it's just an expense.

So yes $1 of savings is worth more than $1 of spending.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#198

Post by cj »

xyst wrote: post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1]

If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings.

It very well might be the case that "We spent $25m of engineering resources to save $6m annually".

[0] https://www.levels.fyi/companies/uber/salaries/software-engi...

[1] https://www.levels.fyi/companies/uber/salaries/software-engi...

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#199

Post by booi »

cj wrote: post #198
xyst wrote: post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…

That’s what I was thinking and fully loaded cost at least 35% more than their salary.

Imagine trading 5 headcount full-time to manage the 1T+ fully custom database on an ongoing basis when they could have just used DynamoDB and have been done with it.

Or better, having to engineer a new feature that already existed in DynamoDB and just losing money at that point.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#200

Post by mbesto »

robocat wrote: post #177
mbesto wrote: post #141

Earlier quoted context omitted.

$6m in perpetuity is like infinite and engineers can be fired.

It really isn't. The first years dominate the value and later years are worth nothing due to inflation. Google a calculator and use a reasonable discount rate and I suspect you will find that todays value for an infinite perpetuity is a lot less than your intuition might guess. It always surprises me.

What?

It costs me $10M to run something every year, it now costs me $4M to run something every year. I have $6M in my pocket every year now in perpetuity. Compound that annually with the assumption that I maintain or increase top line revenues and thats pure extra profit.

Note - I admit, all of this ignores two key things (a) we dont know the engineers salaries who built this and (b) we dont know the ongoing maintenance costs.

Post reply on HN