It turns out however that even when you use it as a distributed hashtable you still pay a huge premium.
Migrating Uber's ledger data from DynamoDB to LedgerStore
261–270 of 345 posts
Re: Migrating Uber's ledger data from DynamoDB to LedgerStore
#262Earlier quoted context omitted.
Not if you're in the EU, due to, among other things, Schrems II.
AFAIK Schrems II prevents transfers of data to the US. AWS has datacentres around the world, including multiple locations in the EU.
Schrems II prohibits transfer of personal information to companies reachable by the CLOUD act.
Re: Migrating Uber's ledger data from DynamoDB to LedgerStore
#263„how much talent is wasted on pointless things that help noone in the world while getting paid heaps for nothing”
We could accomplish everything if ppl stopped wasting time on pointless tasks.
Re: Migrating Uber's ledger data from DynamoDB to LedgerStore
#264Earlier quoted context omitted.
If anything, it reduces Uber's exposure to AWS' proprietary technology. I don't know how to measure how much that's worth but they probably do.
This usually comes from people who have never done a mass migration at scale. You’re always dependent on your infrastructure. Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. No, just use Terraform and Kubernetes is not the answer. The typical enterprise is dependent on depending on the source between 80 - 120 SaaS products - ie outside vendors.
I'd assume it takes fewer millions to migrate your own tech stack from AWS to somewhere else than it takes to migrate from AWS proprietary solutions. Is that reasonable?
Re: Migrating Uber's ledger data from DynamoDB to LedgerStore
#265Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.
The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics. If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly…
Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to deal with in order to query on transactions.
Re: Migrating Uber's ledger data from DynamoDB to LedgerStore
#266Earlier quoted context omitted.
What's the purpose of this comment? My point is that a random dev running a pretty plain adblock (aren't we all?) simply cannot view their post. This is down to uber, their practices, an external developer and how uber create their blog (they don't just have the content in the page). If I'm not a special case with extremely weird luck, a bunch of devs seeing links to their posts will open them and not see any actual…
I have no issues reading their blog with uBlock Origin. You are running software that is blocking content you want to read. That is my point. If I put on blinders and then complain I can't see your stuff, that's my fault not yours - regardless if your stuff is good or the worst annoying spam ever. If I want to see it for some reason, maybe I should take off the blinders
Yes. It's my point too. I am running very standard software for a dev and it is stopping their dev blog posts being visible.
> If I put on blinders and then complain
I'm not complaining. I'm explaining, given the evidence I have, why they may be seeing poor results on HN. If I'm not alone (and since I have no custom setup designed to keep our their blog posts that would be a surprise) then there are other developers who cannot see their posts.
Re: Migrating Uber's ledger data from DynamoDB to LedgerStore
#267Earlier quoted context omitted.
This usually comes from people who have never done a mass migration at scale. You’re always dependent on your infrastructure. Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. No, just use Terraform and Kubernetes is not the answer. The typical enterprise is dependent on depending on the source between 80 - 120 SaaS products - ie outside vendors.
*> Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. I'd assume it takes fewer millions to migrate your own tech stack from AWS to somewhere else than it takes to migrate from AWS proprietary solutions. Is that reasonable?
And you’re dealing with your PMO department, project managers, finance, security, contract negotiations, retraining your ops department…
And you know that Aurora MySQL instance that was suppose to prevent “lock in”? I bet you someone somewhere in your org thought about creating an ETL job and then said forget it and used “select into S3” to move data from MySQL into S3.
As a project manager trying to ship code so you can show “impact” to put on your promo doc, are you going to choose for your team to spend weeks to write an ETL job to prevent “lock in” or are you going to tell the developer to write that one line of SQL?
There are all sorts of choices you can make that will save time and money and ship features that actually deliver value instead of worrying about the boogie man of “lock in”.
And I really hope that there was some better technical reason than just saving $6 million dollars a year for a multibillion dollar company to go through the migration.
Re: Migrating Uber's ledger data from DynamoDB to LedgerStore
#268Earlier quoted context omitted.
Ad Blocking is recommended by USA government agency for security reasons, not running an ad blocker is a dangerous and suggest lack of information/education about IT stuff.
Agreed, but if legit content gets blocked you only have yourself to blame. Like turning off JS and saying webapps don't work anymore.
Re: Migrating Uber's ledger data from DynamoDB to LedgerStore
#269Earlier quoted context omitted.
*> Even if you have nothing but everything hosted on a bunch of VMs, it can take years and millions of dollars to migrate. I'd assume it takes fewer millions to migrate your own tech stack from AWS to somewhere else than it takes to migrate from AWS proprietary solutions. Is that reasonable?
No because you still have to deal with permissions, integrations with AWS services like networking, training, security audits, regression testing, often physical network connections (Direct Connect), DNS… And you’re dealing with your PMO department, project managers, finance, security, contract negotiations, retraining your ops department… And you know that Aurora MySQL instance that was suppose to prevent “lock in”?…
Re: Migrating Uber's ledger data from DynamoDB to LedgerStore
#270Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.
$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…