Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

281–290 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#281

Post by scarface_74 »

gtirloni wrote: post #269
scarface_74 wrote: post #267

Earlier quoted context omitted.

No because you still have to deal with permissions, integrations with AWS services like networking, training, security audits, regression testing, often physical network connections (Direct Connect), DNS… And you’re dealing with your PMO department, project managers, finance, security, contract negotiations, retraining your ops department… And you know that Aurora MySQL instance that was suppose to prevent “lock in”?…

Thanks for the insights. So in the case that it's actually more expensive to migrate your own tech stack somewhere else than, say, migrate from AWS proprietary to GCP proprietary, it seems there might be other reasons.

The difficulty would be worse of course if you depend on anything proprietary from the cloud vendor.

But the main question is, once you do all of this work and spend time to be “cloud agnostic”, does it add business value?

In the case of Dropbox, it made sense to move from the cloud. In the case of Netflix, they decided to move to the cloud.

But you can’t stay completely “cloud agnostic”.

Let’s take a simple case of using Kubernetes and building the underlying infrastructure using Terraform.

The entire idea behind Kubernetes is to abstract your infrastructure - storage, load balancers, etc.

But eventually, you still have to deal with what’s underneath. I used AWS’s own Docker orchestration service for years - ECS. But I just learned Kubernetes last month.

I still had to know how to troubleshoot problems with IAM permissions, load balancers, view CloudTrail logs for permission issues, know how the underlying storage providers worked, make sure I had the right plug installed for K8s to work with AWS’s infrastructure etc.

Once I got all of that figured out, then I could go through the tutorials and mind map the difference between ECS and AWS’s Kubernetes implementation - EKS.

But I had years of experience with AWS. I could have never easily troubleshoot the same types of issues with Azure’s or GCP’s version of K8s. Now multiply that by an entire department.

Once everything is configured correctly, a developers experience would be the same across environments

Migrations at scale are always a pain from one system to another.

Source: I worked at AWS in the Professional Services department for three years. I’m mostly a developer and I dealt with the “modernization” side of “lift and shift and then modernize”.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#282

Post by 8ytecoder »

crmd wrote: post #265
amluto wrote: post #251

Earlier quoted context omitted.

The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics. If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly…

The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes. Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to de…

Which is …normal. Different types of data access have different SLA requirements. Any company that cares about cost will warehouse data after x time frame. It’s done even in banking. Uber has a much more lenient need to make this data instantly available.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#283

Post by jiggawatts »

crmd wrote: post #265
amluto wrote: post #251

Earlier quoted context omitted.

The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics. If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly…

The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes. Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to de…

Immutable data is always consistent. This is almost certainly an append-only ledger, a well-established solution for a simple problem.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#284

Post by PartiallyTyped »

yazaddaruvala wrote: post #277

Reading the article it’s clear pretty quickly that Uber was using DynamoDB poorly. It seems they need strong consistency for certain CUJs and then a lot of data warehousing for historical transactions. It’s strange to me that they didn’t first convert their 2 table DynamoDB architecture into DynamoDB and Redshift architecture or similar. This is a pretty common pattern.

I don't understand why they needed 2 weeks of immutable transactions in Dynamo. Could anyone give any hints?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#285

Post by cced »

yazaddaruvala wrote: post #277

Reading the article it’s clear pretty quickly that Uber was using DynamoDB poorly. It seems they need strong consistency for certain CUJs and then a lot of data warehousing for historical transactions. It’s strange to me that they didn’t first convert their 2 table DynamoDB architecture into DynamoDB and Redshift architecture or similar. This is a pretty common pattern.

Can you post some references to this pattern?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#286

Post by amluto »

crmd wrote: post #265
amluto wrote: post #251

Earlier quoted context omitted.

The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics. If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly…

The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes. Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to de…

> This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements.

This seems like an odd requirement to me. What are the use cases beyond a few minutes past the end of the ride?

- Reconciliation of Uber's finances. This definitely needs to be correct and consistent and maybe even auditable, but it's also done at the end of any given billing period. A daily roll-up would be sufficient.

- Showing each driver and each rider their own history. This ought to be fully correct (especially for drivers), but it's a massive double-edged sword. Uber's app, for example, appears to be entirely missing the "please forget my ride history" button. And this isn't surprising: Uber apparently stores it in a ledger, fully indexed, complete since at least 2017, that is cryptographically immutable. Is this actually a good thing? What happens when a privacy regulator tells Uber to expunge, anonymize, or pseudonymize old data?

- Handling chargebacks? Having a high quality record going back 180-365 days seems useful. But the cost to Uber of occasionally losing a record is very low and is barely more than proportional to the probability of loss of a given record.

So I don't get it. If I were running an operation like this, I would prefer not to have a fully immutable ledger full of personally identifiable location, usage, and transactional data.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#287

Post by hallway_monitor »

perryh2 wrote: post #235
mobilemidget wrote: post #221

Earlier quoted context omitted.

Do you have any good recommendation for such 1U server with 32 slots? Thanks

Supermicro https://www.supermicro.com/en/products/nvme?pro=formfactor%3...

Shows 2U in order to fit 32 drives, but hopefully you can find a 2U slot for this absolute monster of a storage box if you have that need.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#288

Post by jrockway »

washywashy wrote: post #67

I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some crit…

At Google we had a conversion chart between things like memory / cpu and SWE hours, so if you had an idea that would save 8TB of RAM and take 4 hours to do it, you could decide if it would be worth your time or not.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#289

Post by andreygrehov »

ndriscoll wrote: post #179
kondro wrote: post #29

Earlier quoted context omitted.

Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB. I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.

There are 61.44 TB NVMe drives (best price I've seen right now is ~6200. They were ~4800 earlier this year). You can have a 1U server with 32 E1.L slots so you should be able to fit ~1.9PB raw storage into 1U for a little over $200k. Don't know how business financing works, but at 8% interest with a 5 year amortization, that's a bit over $4k/month.

Nimbus ExaDrive is 100TB [1].

[1] https://nimbusdata.com/products/exadrive/

Post reply on HN