Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

321–330 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#321

Post by simion314 »

pests wrote: post #238
simion314 wrote: post #204

Earlier quoted context omitted.

Ad Blocking is recommended by USA government agency for security reasons, not running an ad blocker is a dangerous and suggest lack of information/education about IT stuff.

Agreed, but if legit content gets blocked you only have yourself to blame. Like turning off JS and saying webapps don't work anymore.

>but if legit content gets blocked you only have yourself to blame.

If the bug is on the devs then the devs are to blame, for maybe expecting teh ads are loaded, or the tracking third party code.

The project I am working on works with ad blocker on. Also we had issues with users that had a spellchecking extension active, it would create a ton of hidden markup on a contenteditable element, and we made code to handle the issue instead of having many tickets to our support complaining and we telling them that is their fault for using a popular extension.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#322

Post by ckdarby »

nostrebored wrote: post #300
noncoml wrote: post #254

Earlier quoted context omitted.

I don’t know you are downvoted. Aligning the personal interest vector with the companies interest vector is a huge problem that is usually underrepresented in NH comments. Usually we only complain about the short sighting of the CEOs that prioritize short term stock gains over long term prosperity, but that also is just a specialized case of the success vector misalignment

This is a poor way to view salaried work. If OP is meeting deliverables and did this on the side, this is a great example of a story that would make me inclined to hire.

The problem is OP didn't meet deliverables and the team was behind schedule.

Helpful in the right light, but equally not helpful in another.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#323

Post by Retric »

omeid2 wrote: post #316
Retric wrote: post #305

Earlier quoted context omitted.

Hardly the global economy, even approaching a global monopoly for something tends to hit such limits. Few companies deal with this today because most markets are fragmented. But even at the 30% marketshare the iPhone has been dealing with these issues for a while. They just can’t buy 200 million volume buttons or whatever off the shelf. Now imagine what happens if one of their suppliers would fail days before the pho…

Outsourcing doesn't mean off the shelf. Apple still works with third parties to manufacture parts as they can leverage the scale of their overall operations to do it cheaper than it would cost apple to do.

They aren’t just working with from a design or quality control standpoint that’s normal enough, they are also purchasing equipment for 3rd party companies to use which isn’t. At this point they need to be world class experts in everything from batteries, sensors, software, and processors to glass.

Tesla does some stuff in house because they can, but they just didn’t have any options when it came to scaling battery production. They hit the limits of what the market could supply without getting directly involved.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#324

Post by collyw »

ramesh31 wrote: post #127

Another victim of the "Great Normalization", i.e. that entire generation of garbage tech debt generated during the 2010s that was built on NoSQL stores that never should have been, is now coming due. You could probably make an entire consulting business out of migrating these things to MySQL.

This seemed so obvious at the time, you were throwing away so many useful features for the promise of "web scale". I argued with many developers at the time, but they insisted that NoSQL (Mongo) needed to be used.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#325

Post by kikimora »

nikolajan wrote: post #291
amluto wrote: post #286

Earlier quoted context omitted.

> This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride? - Reconciliation of Uber's finances. This definitely needs to be correct and consistent and maybe even auditable, but it's also done at the end of any given bill…

>This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride? Exactly what he the OP said, authorization. Card networks cap latency of authorization requests (the time the merchant submits a request, to when it makes its make through the merchant processor, networks, issuer processor and back.)

Capture can happen days after successful authorization.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#326

Post by kikimora »

alexey-salmin wrote: post #261

I don't know about economics of this particular project but damn dynamodb is expensive. At some point I was thinking that everyone else was just using it wrong, doing scans and queries instead of point-wise lookups into pre-computed tables. It turns out however that even when you use it as a distributed hashtable you still pay a huge premium.

Why? 120 usd per 100 WCU per year, 30 usd per 100 RCU, does not sound expensive. 1 RCU reads up to 4Kb, to read 100 MBs you would need 100 000 RCU which would cost 30 000 usd/year, or 2500/year. Unless my math is off I don’t think anything come close in terms of price.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#327

Post by apwell23 »

bjornsing wrote: post #163
apwell23 wrote: post #95

Earlier quoted context omitted.

typo was not the point of the comment though :) also is there a demo or some sort of technical whitepaper.

Not yet. Is that something you’d find compelling? Anything in particular you’d like to see?

would love to see snowflake like paper. I am not sure if you are targetting enterprise sales or trying to woo developers who can advocate for your product in their companies. If you are targeting latter, Its unlikely that developers are going to fill a contact us form to try out your product.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#328

Post by marcyb5st »

amluto wrote: post #251
xyst wrote: post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics. If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly…

In some countries (e.g. Italy, France) you are required by law to keep records of transactions and other data for 10y. But I agree with the data tiering part. Anything older than X months should go to storage (parquet files in some cloud storage).

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#329

Post by shermantanktop »

ckdarby wrote: post #322
nostrebored wrote: post #300

Earlier quoted context omitted.

This is a poor way to view salaried work. If OP is meeting deliverables and did this on the side, this is a great example of a story that would make me inclined to hire.

The problem is OP didn't meet deliverables and the team was behind schedule. Helpful in the right light, but equally not helpful in another.

“Team behind schedule” usually means that some manager promised an arbitrary date based on an imaginary timeline.

Every team needs slack. Bad managers hate slack. Good managers hide slack.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#330

Post by MarkMarine »

surfingdino wrote: post #312
MarkMarine wrote: post #252

Earlier quoted context omitted.

I heard from some X-Uber people that you could call Uber a database company as much as you could call it a transportation company. Something like 80+ databases invented there in one form or another. Promotion-driven development. I suppose better than blog post driven development, but marginally so.

> I heard from some X-Uber people that you could call Uber a database company Someone once called Airbus planes "Sun servers with wings". All businesses today have an IT department and those build around the idea of real-time scheduling done by computers are compute/storage businesses first. That's why Uber was able to launch Uber Eats or e-bikes. I am not surprised they have a bunch of busy devs building cool stuff.…

nih-syndrome is a real thing. I'm not saying they don't make cool stuff but the engineering culture that prioritizes creating a new database over using something that already exists, then promoting the people who develop the (70th) new DB over the people who were focused on actually delivering value for the customer, it's not my choice in working environment.

The architecture where all the magic lives in the database layer and the services treat the DB like this magical thing that synchronizes across multiple DCs for you, and takes care of every complicated part of engineering a distributed system is a "now you have two problems" situation. It's a massive complicated system that will fail in novel ways, and you don't have a large community of other users that you can fall back on for expertise.

Post reply on HN