Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

221–230 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#221

Post by mobilemidget »

ndriscoll wrote: post #179
kondro wrote: post #29

Earlier quoted context omitted.

Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB. I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.

There are 61.44 TB NVMe drives (best price I've seen right now is ~6200. They were ~4800 earlier this year). You can have a 1U server with 32 E1.L slots so you should be able to fit ~1.9PB raw storage into 1U for a little over $200k. Don't know how business financing works, but at 8% interest with a 5 year amortization, that's a bit over $4k/month.

Do you have any good recommendation for such 1U server with 32 slots? Thanks

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#222

Post by anonzzzies »

frankjr wrote: post #220
anonzzzies wrote: post #213

Earlier quoted context omitted.

Any open source doing something similar ?

Litestream is open source. https://github.com/benbjohnson/litestream

Ai. I remember another product and thought it was this. Sorry. Move on and keep up the good work.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#223

Post by barfbagginus »

vsnf wrote: post #178
barfbagginus wrote: post #144

Earlier quoted context omitted.

And you can't have blazing fast without rust, and a little kvetching about lifetimes

While your broader point is well taken, isn’t Uber a famous Go shop?

Lol I was not being on topic or constructive - just repeating the meme that rust is synonymous with "blazing fast", because of endless statements to the effect of "rust is blazing fast," or "if you want blazing fast code, use rust," or the endless blazing fast rust libraries:

https://duckduckgo.com/?q=blazing+fast+rust

Now I'm not an expert in either rust or go. But I know my deductive meme logic:

1. Uber's solution is not blazing fast

2. They are a Go house

Then the meme implies:

3. Their solution is slow because they did not use rust!

Q.E.M. (Quod Erat Memonstrandum)

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#224

Post by bachmeier »

szundi wrote: post #54
bachmeier wrote: post #27

Earlier quoted context omitted.

> I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance I'm guessing they know a lot about their costs, and you know very little. There's little value in insulting the team members like this.

That was not a nice reply for a non-insult. Do you have anything to add maybe?

> That was not a nice reply for a non-insult.

It's an insult if you dismissively explain basic things to the folks working on the project.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#226

Post by superasn »

bjornsing wrote: post #84
superasn wrote: post #32

Earlier quoted context omitted.

One observation I have regarding your homepage is that the message isn't very clear. The headline doesn't mention any benefits I get from using your software. I think you should invest some time into improving your landing page and maybe you may see some traction. A good resource for this which I've bookmarked is here(1). Hope that helps. (1) https://www.indiehackers.com/post/my-step-by-step-guide-to-l...

Thanks. So you think it would work better if it would just say “save money”, rather than jump straight into the “what”? To me, when I read the below, that just screams “save money”. But maybe I should do that conversation for the reader so to speak? From benefits box: “Sometimes you need to index a huge amount of data, to accelerate just a few search queries. But building indexes and keeping them in hot storage can b…

I asked chatgpt for a headline based on your prompt and it gave me this:

HaystackDB: Swift Searches, Massive Savings - Index Billions, Store Smartly, Query in a Flash!”

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#227

Post by ComodoHacker »

alexchantavy wrote: post #218
gronky_ wrote: post #47

Earlier quoted context omitted.

Yes, that’s definitely the main reason. It’s called “burying the lede”. Saving $6M is key information that makes this story interesting. It’s buried all the way at the bottom of the first blog and is completely missing from the second blog which focuses specifically on the migration

I'm usually guilty of this. The hands-on person involved in a highly technical project gets excited and bogged down in the details of the project that they end up not being the most compelling storyteller about it.

Don't blame yourself. Not everyone is here for the money, many of us are here for the tech.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#229

Post by rmbyrro »

vertis wrote: post #135
washywashy wrote: post #67

I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some crit…

A better strategy as a company this size would be to write the PRD for moving and then call AWS and negotiate.

I think it's likely that they tried this. But DynamoDB is expensive to consume probably because it's expensive to run and maintain. If you develop for a particular use case, a lot of optimizations can reduce these costs. For a large enough business, the fixed costs of in-house are easily amortized. It'd be hard for AWS to compete.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#230

Post by hibikir »

cj wrote: post #210
appplication wrote: post #174

Earlier quoted context omitted.

You don’t need to consider the cost they could drive during that time. You have a direct and tangible savings for engineering time invested. That possible value they could otherwise derive is moot and hypothetical, this is the real deal! But if we’re being honest, there isn’t actually any meaningful quantification of engineering time to understand return on investments at this level (not to say there’s none , but it…

>You don’t need to consider the cost they could drive during that time. You don't need to, but you 100% should. "Opportunity cost" (cost of not doing something) is real. This is the problem with all refactoring/migration projects. It's very easy to get a lot of people to agree a company should migrate from Node to Go or Monolith to Microservices (or to clean up a mountain of tech debt), but it's much harder to justif…

True, but often the project that was supposed to build something users care about turns to dust. On one side, you have rosy projections. On the other, a cap on gains, so sure everyone picks the first, but nobody measures if it worked.

One can build a great career working only on key, promising initiatives that never amount to any value in the end. By the time it's clear the project lost money outright, you are on to something else.

Post reply on HN