Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

241–250 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#241

Post by cthalupa »

inoop wrote: post #101
cdchn wrote: post #65

Earlier quoted context omitted.

Developing and maintaining a totally bespoke DB system with that kind of volume even for $5m/yr, spitball you could get yourself 25 top-notch engineers without AI PhDs and have another mil left over for metal. Sounds plenty feasible to have a nice tailored suit for a core part of your business.

> you could get yourself 25 top-notch engineers without AI PhD Not in the US though. According to levels.fyi, an SDE2 makes ~275k/year at Uber. Hire 25 of those and you're already at $6.875MM. In reality you're going to have a mix of SDE1, SDE2, SDE3, and staff so total salaries will be higher. Then you gotta add taxes, office space, dental, medical, etc. You may as well double that number. And that's just the cost o…

> Then you gotta add taxes, office space, dental, medical, etc. You may as well double that number.

Economies of scale help a bit with this for larger companies, so it's probably not quite double for Uber, but yeah, not too far off as a general rule of thumb. Probably a 75% increase on the employee facing total comp to get fairly close to the company's actual cost for the employee.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#242

Post by pests »

IanCal wrote: post #188
pests wrote: post #169

Earlier quoted context omitted.

That's probably because you are running software that is meant to hide content on a page.

What's the purpose of this comment? My point is that a random dev running a pretty plain adblock (aren't we all?) simply cannot view their post. This is down to uber, their practices, an external developer and how uber create their blog (they don't just have the content in the page). If I'm not a special case with extremely weird luck, a bunch of devs seeing links to their posts will open them and not see any actual…

I have no issues reading their blog with uBlock Origin.

You are running software that is blocking content you want to read. That is my point.

If I put on blinders and then complain I can't see your stuff, that's my fault not yours - regardless if your stuff is good or the worst annoying spam ever. If I want to see it for some reason, maybe I should take off the blinders

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#243

Post by shawabawa3 »

cj wrote: post #198
xyst wrote: post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…

> we spent $25m of engineering resources to save $6m annually

This is a huge ROI. Borrowing $25m costs about $1.25m/yr so you're winning even with no upfront costs

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#244

Post by dang »

nolongerthere wrote: post #92
ForHackernews wrote: post #39

Does no one ever delete data? It's hard to believe there's much business value in keeping every individual payment record dating back to 2017.

At an individual level I appreciate when an app or service I use maintains all records from the start of our relationship, I’ve infrequently found myself going back and looking for something, and it’s always a breath of fresh air to see that nothing was deleted.

Sorry for the offtopicness, but please see https://news.ycombinator.com/item?id=40418627 regarding a flamewar that happened over a week ago. It's important that this not happen again.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#245

Post by graemep »

cess11 wrote: post #192
geodel wrote: post #119

More power to them. At this point even technically decent teams/companies have given up on developing large, complex systems in favor of SaaS. After carefully evaluating our strategic course of action answer always is AWS. Its only team who propose alternative they have to justify rigorously how come they differ in conclusion.

Not if you're in the EU, due to, among other things, Schrems II.

AFAIK Schrems II prevents transfers of data to the US.

AWS has datacentres around the world, including multiple locations in the EU.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#246

Post by cheeze »

Closi wrote: post #193
_xnmw wrote: post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

1.7 petabytes on Sqlite? Sqlite's own advice: > If your data will grow to a size that you are uncomfortable or unable to fit into a single disk file, then you should select a solution other than SQLite. SQLite supports databases up to 281 terabytes in size, assuming you can find a disk drive and filesystem that will support 281-terabyte files. > Even so, when the size of the content looks like it might creep into the…

This is the worry IMO. It's fine to dump it on a server with SQLite, but once you start hitting scaling limits, you're in for a potentially rough migration.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#247

Post by influx »

geodel wrote: post #147
influx wrote: post #106

I would gladly pay 6 million/year to not be on call, and have to worry about things like bios and ssd firmware ever again.

Thats a great situation to be in when one can spend 6 million even when there was some chance to save. I tried same for ready to eat meal everyday to save me from potential kitchen disasters but sadly numbers didn't work out.

You're not saving money, controlling your own destiny sure. That's worth something, maybe even more than 6 million, but I was a SRE at Uber who had to be oncall for systems like this, believe it or not, people like me aren't free either :)

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#248

Post by nemothekid »

sgt wrote: post #35
BlackLotus89 wrote: post #33

Earlier quoted context omitted.

No it won't. sqlite "only" works with up to 281TB [0] [1] [0] https://www.sqlite.org/releaselog/3_33_0.html [1] https://www.sqlite.org/limits.html (#12)

You can split up into 10 SQLite DB's on this individual server.

Once you are splitting up 10 sqlite dbs you have a bespoke distributed system anyways, and you will find yourself doing all the headache of LedgerStore anyways.

Most of the novel work in LedgerStore is probably around managing the headaches of distributed storage, not the persistence layer.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#249

Post by valiant55 »

shawabawa3 wrote: post #243
cj wrote: post #198

Earlier quoted context omitted.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…

> we spent $25m of engineering resources to save $6m annually This is a huge ROI. Borrowing $25m costs about $1.25m/yr so you're winning even with no upfront costs

I mean, ideally you are still employing those people into the future. Plus was there other opportunities to drive value that would've been better spent?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#250

Post by ownagefool »

khaki54 wrote: post #103
_xnmw wrote: post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

The value proposition of commercial loud isn't cost savings unless you manage to quantify all of the ancillary and extrinsic factors such as security risk, HVAC, datacenter personnel, and hardware lifecycle. Any well capitalized and organized company could build their own cloud much more cheaply, but really a significant portion of the calculation is outsourcing the risk components.

> Any well capitalized and organized company could build their own cloud much more cheaply

Lots of orgs fail to turn money into talent and then talent into products.

It just takes one bad hire at senior level and suddenly your cloud is a vmware install where all machines are boot off network disk, and contention makes the entire thing fall over.

Post reply on HN