Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

101–110 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#101

Post by inoop »

cdchn wrote: post #65
sha_r_roh wrote: post #6

Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?

Developing and maintaining a totally bespoke DB system with that kind of volume even for $5m/yr, spitball you could get yourself 25 top-notch engineers without AI PhDs and have another mil left over for metal. Sounds plenty feasible to have a nice tailored suit for a core part of your business.

> you could get yourself 25 top-notch engineers without AI PhD

Not in the US though. According to levels.fyi, an SDE2 makes ~275k/year at Uber. Hire 25 of those and you're already at $6.875MM. In reality you're going to have a mix of SDE1, SDE2, SDE3, and staff so total salaries will be higher.

Then you gotta add taxes, office space, dental, medical, etc. You may as well double that number.

And that's just the cost of labor, you haven't spun up a single machine and or sent a single byte across a wire.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#102

Post by narnarpapadaddy »

bjornsing wrote: post #98
narnarpapadaddy wrote: post #94

Earlier quoted context omitted.

The companies that have these types of problems all have AWS reps (or whatever vendor) that get first crack at a solution, even if their senior engineers or CTOs do some googling. Frequently discounts can be negotiated on products that aren’t a perfect fit, or companies will get early access to new products that solve their problem (AWS calls this “limited preview”). A good chunk of B2B infrastructure products like t…

Yep. Always a bit dangerous to go up against AWS and similar. My hope here is that this product is too niche for the major cloud vendors to invest in. But since Uber is building stuff themselves that assumption may be wrong. A “golden partner” model makes a lot of sense, thanks.

If you know a technology leader at a company that has this problem, reach out and ask if they’ve had any pain related to it. Ask to take them out to lunch and tell them about a solution you’ve been working out. Or even a short demo call. See if they’d be interested in an “innovation partnership.” You give them a discounted/free license (can be time limited to a year or however long it takes to validate that your solution works and saves them money - then returns to full-price), they agree to feature prominently in your marketing material or provide a reference for your next lead.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#103

Post by khaki54 »

_xnmw wrote: post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

The value proposition of commercial loud isn't cost savings unless you manage to quantify all of the ancillary and extrinsic factors such as security risk, HVAC, datacenter personnel, and hardware lifecycle. Any well capitalized and organized company could build their own cloud much more cheaply, but really a significant portion of the calculation is outsourcing the risk components.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#104

Post by zinglersen »

HeatrayEnjoyer wrote: post #72
zinglersen wrote: post #45

Earlier quoted context omitted.

Losing your job because the outcome of your efforts (or even external events) is not what I would call the ultimate price. "The metaverse division has now lost more than $45 billion since the end of 2020" Your compensation for your work is your salery. So I would say that it's fair that the actual risk taker is benefiting from the potential rewards?

The "risk takers" are not taking at any risk at all. What's the chance they end up on the street, or even suffer personal financial stress about their life? That they will have to move, sell their car, home, etc. It's 0%.

What.. they are taking a lot of risk...

But I guess we first have to agree on "who" we are taking about - is it the company itself or the owner / shareholders ?

Back to your question, yes that could happen in several different cases. But of course the risk/benefit is not split 50/50 (nor 0 risk, 100 upside, as you said), in reality the future outcome depends on both internal and external events.

Even the richest(?) man in the world was relatively close to loosing it all;

Musk, who had $200 million in cash at one point, invested “his last cent in his businesses” and said in a 2010 divorce proceeding, “About four months ago, I ran out of cash.” Musk told the New York Times https://www.cnbc.com/2017/04/27/the-crucial-decision-teslas-... https://archive.nytimes.com/dealbook.nytimes.com/2010/06/22/...

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#107

Post by macspoofing »

bjornsing wrote: post #7

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

The market is saturated with storage products, so it's tough to differentiate yourself. Your site does not help by the way. You're also not selling an end-user product to the public, rather you're selling a technical and infrastructure solution to very technical people - that's a different type of sale. To get those people interested, you must put together technical whitepapers/blog posts/webinars/youtube videos/etc. to explain your approach.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#108

Post by Rastonbury »

scop wrote: post #91
washywashy wrote: post #67

I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some crit…

That was where my mind went to when I saw the headline. Granted that while I’m not on the Finance side of things and am in fact a developer, “six million” didn’t seem like much at all considering engineer salaries. It’s certainly an achievement, but at what short and long term salaried cost?

$6m across 2.5 years is like $15.5m, how many engineers man-months to breakeven, I'm pretty sure it was worth the work.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#109

Post by sgt »

thangngoc89 wrote: post #81
mickeyp wrote: post #75

Earlier quoted context omitted.

Would you care to tell us what your backup and restore policy would be for 1.7 PB of data?

I'm replying to the question of how one would replicate SQLite 3 in production for redundancy. I myself consider 10GB would be the limit for using SQLite 3 in read/write in production and switch to PostgreSQL.

That's a huge discrepancy. One half of HN wants to put petabytes on SQLite, while your limit is only 10GB.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#110

Post by macspoofing »

bjornsing wrote: post #80
padjo wrote: post #30

Earlier quoted context omitted.

Yep nobody with the problem you’re offering to solve is going to solve it by googling and picking some random company they’ve never heard of with no track record.

Not even click a search ad and fill in a contact form? When I’m on the other side of the table I do that. But perhaps I’m unique in that aspect? (I understand there won’t be any significant business without enterprise sales. But that’s not what I’m looking for at this stage.)

>When I’m on the other side of the table I do that

No, you don't. There are many established storage solutions out there. If you're in the market for one, you can easily fill days, weeks or months vetting those. So, why would you bother dealing with a sales rep from a random one you never heard of before, and isn't used by anyone. You don't even provide any details on what makes it different or better from anything else out there.

Post reply on HN