Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

201–210 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#201

Post by PretzelPirate »

ddorian43 wrote: post #183
_xnmw wrote: post #173

Earlier quoted context omitted.

The problem with outsourcing the risk components is that you don't know for sure whether they are properly taken care of. Major cloud providers have been caught "oopsing" your data, and bam it is gone. Furthermore, they have no incentive to be more efficient about it, they could easily be using 10x the amount of resources necessary, and you wouldn't even have a clue, you're just paying for evermore expensive crap tha…

But the cloud providers compete with other other! Look at the efficient market in display in their bandwidth pricing!

For very large customers, the cloud providers do compete with each other on cost. They often pay different prices than are advertised.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#202

Post by cgh »

ramesh31 wrote: post #127

Another victim of the "Great Normalization", i.e. that entire generation of garbage tech debt generated during the 2010s that was built on NoSQL stores that never should have been, is now coming due. You could probably make an entire consulting business out of migrating these things to MySQL.

This is exactly the comment I came here to make. The NoSql technical debt accretes like dead leaves blocking a sewer drain. Eventually someone has to wade in and normalize…the sewer grate, I guess. Okay, it’s not a perfect analogy.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#203

Post by dangus »

cj wrote: post #198
xyst wrote: post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…

Of course now we are assuming that the existing solution didn’t also require engineering salaries to maintain.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#204

Post by simion314 »

pests wrote: post #169
IanCal wrote: post #132

Earlier quoted context omitted.

Other than the comments about titles, the entire blogpost doesn't show for me with ublock. So I'll open it, see a picture of some birds, scroll around for a bit then give up.

That's probably because you are running software that is meant to hide content on a page.

Ad Blocking is recommended by USA government agency for security reasons, not running an ad blocker is a dangerous and suggest lack of information/education about IT stuff.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#205

Post by dangus »

washywashy wrote: post #67

I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some crit…

Because they often are a sunk cost.

E.g., you can’t lay off an entire SRE team and have nobody on the on-call rotation. If some of their project work is cost control that is basically free cost savings.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#207

Post by smohare »

brushfoot wrote: post #64
intunderflow wrote: post #41

Earlier quoted context omitted.

Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?

Personally, yes, the rehash's title is stronger. It tells a story whose ending piques your curiosity to read more. "Uber Migrates" (beginning: company that I'm interested in does something) "1T records" (middle: that's a lot of records; I wonder what happened) "from DynamoDB to LedgerStore" (hmm, how do they compare?) "to Save $6M Annually" (end: that's a good chunk of change for me, but was it worth it to Uber? Why…

[dead]

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#208

Post by leoqa »

cj wrote: post #198
xyst wrote: post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…

Bitter story time:

I made a config change to our AWS instances and projected approximately $10MM/year in AWS costs savings (pre-savings).

My boss asked me "Who told you to do this? We need to focus on $project instead". I found another team and transferred out. 3 months later there was a big fire drill about AWS costs and they took my 1-pager and executed it. Didn't get any credit in the shipped email nor did the manager reach out to apologize.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#209

Post by TheNewsIsHere »

admax88qqq wrote: post #197
rapht wrote: post #116

Earlier quoted context omitted.

A primer on valuation: in many financial contexts, $1 of operating savings may be worth much more than $1 of investment. That is because an investment is a one-off, so it's actually worth $1, but the savings are recurring, so they are worth the same number of years that a company's profits are valued. Depending on sector and investors' beliefs in the future of companies, this factor is typically in the 5-20x range. T…

If that $1 of investment doesn't yield any returns that's not an investment it's just an expense. So yes $1 of savings is worth more than $1 of spending.

You could potentially take it as an investment loss for tax purposes. Whether that’s proper depends greatly on the circumstances surrounding how the money was accounted for and spent.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#210

Post by cj »

appplication wrote: post #174
hackernewds wrote: post #164

Earlier quoted context omitted.

is it? if you consider the value 20 engineers could drive instead in that time if you assume they wouldn't have had anything else meaningful to work on during that time to save money, then you have a different problem in the company. $6M seems like the value 1 engineer can drive in a company at the scale of Uber

You don’t need to consider the cost they could drive during that time. You have a direct and tangible savings for engineering time invested. That possible value they could otherwise derive is moot and hypothetical, this is the real deal! But if we’re being honest, there isn’t actually any meaningful quantification of engineering time to understand return on investments at this level (not to say there’s none , but it…

>You don’t need to consider the cost they could drive during that time.

You don't need to, but you 100% should. "Opportunity cost" (cost of not doing something) is real.

This is the problem with all refactoring/migration projects. It's very easy to get a lot of people to agree a company should migrate from Node to Go or Monolith to Microservices (or to clean up a mountain of tech debt), but it's much harder to justify the time it takes away from building things your users care about.

Post reply on HN