Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

91–100 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#91

Post by scop »

washywashy wrote: post #67

I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some crit…

That was where my mind went to when I saw the headline. Granted that while I’m not on the Finance side of things and am in fact a developer, “six million” didn’t seem like much at all considering engineer salaries. It’s certainly an achievement, but at what short and long term salaried cost?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#92

Post by nolongerthere »

ForHackernews wrote: post #39

Does no one ever delete data? It's hard to believe there's much business value in keeping every individual payment record dating back to 2017.

At an individual level I appreciate when an app or service I use maintains all records from the start of our relationship, I’ve infrequently found myself going back and looking for something, and it’s always a breath of fresh air to see that nothing was deleted.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#93

Post by zaphirplane »

thangngoc89 wrote: post #79
zaphirplane wrote: post #69

Earlier quoted context omitted.

By the time the TB is restored, time to start the next test How do you detect restored but bit flipped data ?

I do this in backup testing: sqlite3 /path/to/db sqlite> PRAGMA integrity_check; See SQLite3 documentation: https://www.sqlite.org/pragma.html#pragma_integrity_check

Sounds like it will take awhile for TB and it checks db integrity not data integrity

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#94

Post by narnarpapadaddy »

bjornsing wrote: post #80
padjo wrote: post #30

Earlier quoted context omitted.

Yep nobody with the problem you’re offering to solve is going to solve it by googling and picking some random company they’ve never heard of with no track record.

Not even click a search ad and fill in a contact form? When I’m on the other side of the table I do that. But perhaps I’m unique in that aspect? (I understand there won’t be any significant business without enterprise sales. But that’s not what I’m looking for at this stage.)

The companies that have these types of problems all have AWS reps (or whatever vendor) that get first crack at a solution, even if their senior engineers or CTOs do some googling. Frequently discounts can be negotiated on products that aren’t a perfect fit, or companies will get early access to new products that solve their problem (AWS calls this “limited preview”).

A good chunk of B2B infrastructure products like this are developed using a “golden partner” model. The first customer (or few) gets a free or reduced cost license, the developer gets a real-world scenario with real data to use to figure out what the minimal functionality actually is to be a marketable product and to work out bugs. This arrangement frequently requires a preexisting relationship and trust between both parties.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#95

Post by apwell23 »

bjornsing wrote: post #73
emerongi wrote: post #56

Earlier quoted context omitted.

Besides all the other good feedback here, I will offer my extremely petty reason why I wouldn't spend much time evaluating this product. In the FAQ, under the "Are transactions fully ACID?" heading, there is a typo: "simultaineously". It gives me the impression that not enough care has gone into an important part of this product. I know it's not a fair jugdgement, but first impressions matter.

That’s easy to fix, thanks.

typo was not the point of the comment though :)

also is there a demo or some sort of technical whitepaper.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#97

Post by inoop »

sha_r_roh wrote: post #6

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?

I'd be curious as well to see a more complete cost-benefit analysis, and I'd be especially interested in labor cost.

We don't know how much time and head count Uber committed to this project, but I would be impressed if they were able to pull this off with fewer than 6-8 people. We can use that to get a very rough lower-bound cost estimate.

For example, AWS internally uses a rule of thumb where each developer should generate about $1MM ARR (annual recurring revenue). So, if you have 20 head count, your service should bring in about $20MM annually. If Uber pulled this off with a team of ~6 engineers, by AWS logic, they should about break even.

Another rule of thumb I sometimes see applied is 2x developer salary. So for example, let's assume a 7-person team of 2xSDE1, 3xSDE2, 1xSDE3, and 1xSTAFF, then according to levels.fyi that would be a total annual salary of $2.3MM. Double that, and you get $4.6MM/year to justify that team annual cost footprint, which is still less than $6MM.

Of course, this is assuming a small increase in headcount to operate this new, custom data store, and does not factor in a potentially significant development and migration cost.

So unless my math is completely off, it sounds to me like the cost of development, migration, and ownership is not that far off from the cost of the status quo (i.e. DynamoDb).

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#98

Post by bjornsing »

narnarpapadaddy wrote: post #94
bjornsing wrote: post #80

Earlier quoted context omitted.

Not even click a search ad and fill in a contact form? When I’m on the other side of the table I do that. But perhaps I’m unique in that aspect? (I understand there won’t be any significant business without enterprise sales. But that’s not what I’m looking for at this stage.)

The companies that have these types of problems all have AWS reps (or whatever vendor) that get first crack at a solution, even if their senior engineers or CTOs do some googling. Frequently discounts can be negotiated on products that aren’t a perfect fit, or companies will get early access to new products that solve their problem (AWS calls this “limited preview”). A good chunk of B2B infrastructure products like t…

Yep. Always a bit dangerous to go up against AWS and similar. My hope here is that this product is too niche for the major cloud vendors to invest in. But since Uber is building stuff themselves that assumption may be wrong.

A “golden partner” model makes a lot of sense, thanks.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#99

Post by inoop »

bachmeier wrote: post #27
sha_r_roh wrote: post #6

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?

> I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance I'm guessing they know a lot about their costs, and you know very little. There's little value in insulting the team members like this.

> I'm guessing they know a lot about their costs, and you know very little.

I'm curious what makes you believe the OP doesn't know about cost? They might be director-level at a large tech company with 20+ years experience for all you know...

> There's little value in insulting the team members like this.

I'd argue it's not insulting to question a claim (i.e. 'we saved $6MM') that is offered with little explanation.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#100

Post by cynicalsecurity »

zinglersen wrote: post #45
deadbabe wrote: post #31

Earlier quoted context omitted.

Someone probably gets fired so I guess someone does pay the ultimate price.

Losing your job because the outcome of your efforts (or even external events) is not what I would call the ultimate price. "The metaverse division has now lost more than $45 billion since the end of 2020" Your compensation for your work is your salery. So I would say that it's fair that the actual risk taker is benefiting from the potential rewards?

Is it an offer to become a shareholder without actually buying any shares? That would be absolutely great, but unfortunately, it doesn't work this way.
Post reply on HN