311–320 of 345 posts
Earlier quoted context omitted.
I'm replying to the question of how one would replicate SQLite 3 in production for redundancy. I myself consider 10GB would be the limit for using SQLite 3 in read/write in production and switch to PostgreSQL.
That's a huge discrepancy. One half of HN wants to put petabytes on SQLite, while your limit is only 10GB.
Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time. Usually to aim for a significant promotion. “Designed and built homegrown system to save $Xm! Give me promo, bro?” Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.
I heard from some X-Uber people that you could call Uber a database company as much as you could call it a transportation company. Something like 80+ databases invented there in one form or another. Promotion-driven development. I suppose better than blog post driven development, but marginally so.
Someone once called Airbus planes "Sun servers with wings". All businesses today have an IT department and those build around the idea of real-time scheduling done by computers are compute/storage businesses first. That's why Uber was able to launch Uber Eats or e-bikes. I am not surprised they have a bunch of busy devs building cool stuff. Not all of it is ego-driven. I liked their logger for Go last time I used it.
Earlier quoted context omitted.
At an individual level I appreciate when an app or service I use maintains all records from the start of our relationship, I’ve infrequently found myself going back and looking for something, and it’s always a breath of fresh air to see that nothing was deleted.
Sorry for the offtopicness, but please see https://news.ycombinator.com/item?id=40418627 regarding a flamewar that happened over a week ago. It's important that this not happen again.
I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...
1.7 petabytes on Sqlite? Sqlite's own advice: > If your data will grow to a size that you are uncomfortable or unable to fit into a single disk file, then you should select a solution other than SQLite. SQLite supports databases up to 281 terabytes in size, assuming you can find a disk drive and filesystem that will support 281-terabyte files. > Even so, when the size of the content looks like it might creep into the…
Earlier quoted context omitted.
The thing I find odd about this is that the headline figure is about old immutable records. Almost all of that 1.7PB is ancient by what seems to be to by any practical standard. Uber is not likely to care about the credit card authorization flow for a ride two years ago, except maybe for analytics. If I were doing this, I would be looking at data warehousing systems. 1.7PB of, say, Parquet files in S3 is not terribly…
The primary use case is not analytics. This data store is the system of record in their credit card authorization and billing pipeline, and so it has extreme consistency requirements. The lion’s share of its engineering is to provide consistency across a large spectrum of failure modes. Old data could probably live at lower cost in a data warehouse, but then developers would have multiple systems and namespaces to de…
Yes, which they already had as DynamoDB only held 8 weeks of data. Presumably all payments to drivers were calculated monthly and turned into invoices. Any corrections would end up as adjustments on future invoices. Seems pretty normal.
Earlier quoted context omitted.
Yes, there is an upper limit, and that is the entirety of global economy. But that assertion is as useless as it is true.
Hardly the global economy, even approaching a global monopoly for something tends to hit such limits. Few companies deal with this today because most markets are fragmented. But even at the 30% marketshare the iPhone has been dealing with these issues for a while. They just can’t buy 200 million volume buttons or whatever off the shelf. Now imagine what happens if one of their suppliers would fail days before the pho…
Earlier quoted context omitted.
$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…
> we spent $25m of engineering resources to save $6m annually This is a huge ROI. Borrowing $25m costs about $1.25m/yr so you're winning even with no upfront costs
Opportunity cost is a cost, too
Earlier quoted context omitted.
> we spent $25m of engineering resources to save $6m annually This is a huge ROI. Borrowing $25m costs about $1.25m/yr so you're winning even with no upfront costs
Question is: Where else could they have spent those $25m? Could they have built something with more value? Opportunity cost is a cost, too
Earlier quoted context omitted.
> we spent $25m of engineering resources to save $6m annually This is a huge ROI. Borrowing $25m costs about $1.25m/yr so you're winning even with no upfront costs
That math assumes the on-going maintenance of LedgerStore is no higher than that of using DynamoDB was. I suspect there's now a team of technologists maintaining, securing, and operating LedgerStore.
Hosted services doesn't reduce maintenance costs to zero
Earlier quoted context omitted.
That math assumes the on-going maintenance of LedgerStore is no higher than that of using DynamoDB was. I suspect there's now a team of technologists maintaining, securing, and operating LedgerStore.
I suspect they already had a team maintaining/debugging/operating dynamodb Hosted services doesn't reduce maintenance costs to zero
With LedgerDB, they now have to do both.