Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

231–240 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#231

Post by ckluis »

intunderflow wrote: post #41
drpotato wrote: post #15

The original[1][2] articles are a better read IMO. The link is just a summary of the two with added spelling and grammatical errors that materially impact the meaning. 1. https://www.uber.com/blog/how-ledgerstore-supports-trillions... 2. https://www.uber.com/blog/migrating-from-dynamodb-to-ledgers...

Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?

Just put all your articles into a customGPT with the examples from the rehashes for each one and then ask the GPT to rewrite your title to the a “rehash” like title for the new posts ;)

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#232

Post by callalex »

_xnmw wrote: post #18

I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...

One of the main reasons you put up with the annoyances of tuple-based storage like DynamoDB is because you want extremely high availability that simply cannot be provided by one computer in one physical location.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#233

Post by heavenlyblue »

leoqa wrote: post #208
cj wrote: post #198

Earlier quoted context omitted.

$6m in annual cost savings is truly unremarkable, if we are to believe levels.fyi [0] [1] If you're truly paying engineers, project managers, etc $500k a head, it dramatically undermines the financial cost savings. It very well might be the case that "We spent $25m of engineering resources to save $6m annually". [0] https://www.levels.fyi/companies/uber/salaries/software-engi... [1] https://www.levels.fyi/companies/u…

Bitter story time: I made a config change to our AWS instances and projected approximately $10MM/year in AWS costs savings (pre-savings). My boss asked me "Who told you to do this? We need to focus on $project instead". I found another team and transferred out. 3 months later there was a big fire drill about AWS costs and they took my 1-pager and executed it. Didn't get any credit in the shipped email nor did the man…

Of course you didn't. You used your time to promote yourself instead of doing what you were asked to do instead. That could have cost a promotion for your manager who could have promoted you.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#234

Post by macspoofing »

bjornsing wrote: post #168
macspoofing wrote: post #150

Earlier quoted context omitted.

>I guess what I’m trying to say is that I was hoping that someone with a write intensive workload would want to spend some time evaluating a product built specifically for that. Again, it's not clear to me exactly what it is you're doing that's any different from the plethora of existing off-the-shelf solutions. You're saying that you started this project/company because you were looking for a solution to a specific…

Well, it’s pretty simple: HaystackDB is designed from the ground up for write intensive workloads, so it’s much more economical than existing off-shelf-solutions for that type of workload. Is that not clear from the landing page? From pricing: “$0.2 per million writes, $20 per million reads”. The typical cost profile is $2 per million read/writes, or even more for writes.

>Is that not clear from the landing page?

The marketing byline you have on your landing page is clear enough, but nobody will take that seriously without a deeper technical description.

When I read it, I assumed you wrote some code to move data in and out of lower-cost S3 or Glacier storage tiers because you don't control storage pricing and you run on top of existing public cloud infrastructure. Maybe I'm right, maybe I'm wrong - but if I'm looking for a solution, I need to assess whether I should invest time and effort to do a deeper dive, and that's the box I would put you in, without any more detail.

Anyway, good luck. Hope it works out.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#235

Post by perryh2 »

mobilemidget wrote: post #221
ndriscoll wrote: post #179

Earlier quoted context omitted.

There are 61.44 TB NVMe drives (best price I've seen right now is ~6200. They were ~4800 earlier this year). You can have a 1U server with 32 E1.L slots so you should be able to fit ~1.9PB raw storage into 1U for a little over $200k. Don't know how business financing works, but at 8% interest with a 5 year amortization, that's a bit over $4k/month.

Do you have any good recommendation for such 1U server with 32 slots? Thanks

Supermicro https://www.supermicro.com/en/products/nvme?pro=formfactor%3...

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#236

Post by SkyMarshal »

It seems LedgerStore is not open source [1], and finding any info on it requires following a trail of backlinked Uber blog posts. Here's one with the most info on LedgerStore that I can find, from 2021:

https://www.uber.com/en-US/blog/dynamodb-to-docstore-migrati...

[1]:https://github.com/uber

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#237

Post by dang »

drpotato wrote: post #15

The original[1][2] articles are a better read IMO. The link is just a summary of the two with added spelling and grammatical errors that materially impact the meaning. 1. https://www.uber.com/blog/how-ledgerstore-supports-trillions... 2. https://www.uber.com/blog/migrating-from-dynamodb-to-ledgers...

Ok, we've changed to the second link from https://www.infoq.com/news/2024/05/uber-dynamodb-ledgerstore....

Submitters: "Please submit the original source. If a post reports on something found on another site, submit the latter." - https://news.ycombinator.com/newsguidelines.html

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#238

Post by pests »

simion314 wrote: post #204
pests wrote: post #169

Earlier quoted context omitted.

That's probably because you are running software that is meant to hide content on a page.

Ad Blocking is recommended by USA government agency for security reasons, not running an ad blocker is a dangerous and suggest lack of information/education about IT stuff.

Agreed, but if legit content gets blocked you only have yourself to blame.

Like turning off JS and saying webapps don't work anymore.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#239

Post by PeterZaitsev »

SkyMarshal wrote: post #236

It seems LedgerStore is not open source [1], and finding any info on it requires following a trail of backlinked Uber blog posts. Here's one with the most info on LedgerStore that I can find, from 2021: https://www.uber.com/en-US/blog/dynamodb-to-docstore-migrati... [1]: https://github.com/uber

Yeah. This looks like some internal solution. In general Uber seems to be high on "not invented here" scale - they like to conclude no existing Open Source solutions are good enough for them and they need to build their own... this is different from Facebook approach for example which chose to made MySQL better by adding MyRocks/RocksDB to it and keep them Open Source.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#240

Post by bjornsing »

superasn wrote: post #226
bjornsing wrote: post #84

Earlier quoted context omitted.

Thanks. So you think it would work better if it would just say “save money”, rather than jump straight into the “what”? To me, when I read the below, that just screams “save money”. But maybe I should do that conversation for the reader so to speak? From benefits box: “Sometimes you need to index a huge amount of data, to accelerate just a few search queries. But building indexes and keeping them in hot storage can b…

I asked chatgpt for a headline based on your prompt and it gave me this: HaystackDB: Swift Searches, Massive Savings - Index Billions, Store Smartly, Query in a Flash!”

Thanks! That headline is pretty good. :)
Post reply on HN