Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

261–270 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#261
I don't know about economics of this particular project but damn dynamodb is expensive. At some point I was thinking that everyone else was just using it wrong, doing scans and queries instead of point-wise lookups into pre-computed tables.

It turns out however that even when you use it as a distributed hashtable you still pay a huge premium.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#262
post #192

Earlier quoted context omitted.

Not if you're in the EU, due to, among other things, Schrems II.

AFAIK Schrems II prevents transfers of data to the US. AWS has datacentres around the world, including multiple locations in the EU.

Where did you learn that?

Schrems II prohibits transfer of personal information to companies reachable by the CLOUD act.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#264

Earlier quoted context omitted.

If anything, it reduces Uber's exposure to AWS' proprietary technology. I don't know how to measure how much that's worth but they probably do.

This usually comes from people who have never done a mass migration at scale. You’re always dependent on your infrastructure. Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. No, just use Terraform and Kubernetes is not the answer. The typical enterprise is dependent on depending on the source between 80 - 120 SaaS products - ie outside vendors.

*> Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate.

I'd assume it takes fewer millions to migrate your own tech stack from AWS to somewhere else than it takes to migrate from AWS proprietary solutions. Is that reasonable?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#265
post #251
post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics. If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly…

The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes.

Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to deal with in order to query on transactions.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#266
post #242
post #188

Earlier quoted context omitted.

What's the purpose of this comment? My point is that a random dev running a pretty plain adblock (aren't we all?) simply cannot view their post. This is down to uber, their practices, an external developer and how uber create their blog (they don't just have the content in the page). If I'm not a special case with extremely weird luck, a bunch of devs seeing links to their posts will open them and not see any actual…

I have no issues reading their blog with uBlock Origin. You are running software that is blocking content you want to read. That is my point. If I put on blinders and then complain I can't see your stuff, that's my fault not yours - regardless if your stuff is good or the worst annoying spam ever. If I want to see it for some reason, maybe I should take off the blinders

> You are running software that is blocking content you want to read. That is my point.

Yes. It's my point too. I am running very standard software for a dev and it is stopping their dev blog posts being visible.

> If I put on blinders and then complain

I'm not complaining. I'm explaining, given the evidence I have, why they may be seeing poor results on HN. If I'm not alone (and since I have no custom setup designed to keep our their blog posts that would be a surprise) then there are other developers who cannot see their posts.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#267

Earlier quoted context omitted.

This usually comes from people who have never done a mass migration at scale. You’re always dependent on your infrastructure. Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. No, just use Terraform and Kubernetes is not the answer. The typical enterprise is dependent on depending on the source between 80 - 120 SaaS products - ie outside vendors.

*> Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. I'd assume it takes fewer millions to migrate your own tech stack from AWS to somewhere else than it takes to migrate from AWS proprietary solutions. Is that reasonable?

No because you still have to deal with permissions, integrations with AWS services like networking, training, security audits, regression testing, often physical network connections (Direct Connect), DNS…

And you’re dealing with your PMO department, project managers, finance, security, contract negotiations, retraining your ops department…

And you know that Aurora MySQL instance that was suppose to prevent “lock in”? I bet you someone somewhere in your org thought about creating an ETL job and then said forget it and used “select into S3” to move data from MySQL into S3.

As a project manager trying to ship code so you can show “impact” to put on your promo doc, are you going to choose for your team to spend weeks to write an ETL job to prevent “lock in” or are you going to tell the developer to write that one line of SQL?

There are all sorts of choices you can make that will save time and money and ship features that actually deliver value instead of worrying about the boogie man of “lock in”.

And I really hope that there was some better technical reason than just saving $6 million dollars a year for a multibillion dollar company to go through the migration.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#268
post #238

Earlier quoted context omitted.

Ad Blocking is recommended by USA government agency for security reasons, not running an ad blocker is a dangerous and suggest lack of information/education about IT stuff.

Agreed, but if legit content gets blocked you only have yourself to blame. Like turning off JS and saying webapps don't work anymore.

And if someone with a js heavy blog asked why it wasn't getting traction on a lynx centered forum they'd probably be told that their content wasn't readable for a portion of the users.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#269

Earlier quoted context omitted.

*> Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. I'd assume it takes fewer millions to migrate your own tech stack from AWS to somewhere else than it takes to migrate from AWS proprietary solutions. Is that reasonable?

No because you still have to deal with permissions, integrations with AWS services like networking, training, security audits, regression testing, often physical network connections (Direct Connect), DNS… And you’re dealing with your PMO department, project managers, finance, security, contract negotiations, retraining your ops department… And you know that Aurora MySQL instance that was suppose to prevent “lock in”?…

Thanks for the insights. So in the case that it's actually more expensive to migrate your own tech stack somewhere else than, say, migrate from AWS proprietary to GCP proprietary, it seems there might be other reasons.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#270
post #198
post #185

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…

Right, but that's $25m in R&D investment. Much better than $6m in cost of good/services delivered! Former is great innovation and will be ignored by investors because it's just a fixed cost on the way to becoming profitable. The latter is going to appear in the marginal cost of services calculation.
Post reply on HN