Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

161–170 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#161

Post by bjornsing »

Sevii wrote: post #153
bjornsing wrote: post #7

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

Most places I've worked we couldn't even consider using a product that wasn't supported by a major cloud provider like this. What problem does your product solve that customers absolutely 100% need it? Structurally, you are a small entity trying to compete on cost with hyper scalar cloud providers and open source software. Most ISVs like you charge a ton of money for big problems very few enterprise customers have. I…

This is a good summary of why I’m hesitant to put more work into it.

The counter argument I guess is that developing your own data store in-house should be even more of a no-no, and companies do that. (One example is obviously Uber, but my previous employer is another example.)

Do you think the option to self-host the product would help tip the scale?

> What problem does your product solve that customers absolutely 100% need it?

To be blunt there is no such problem: you can always throw more money e.g. at DynamoDB. But if you have a very write intensive workload (such as the use-case described in the OP), then you can save 90% of that money.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#163

Post by bjornsing »

apwell23 wrote: post #95
bjornsing wrote: post #73

Earlier quoted context omitted.

That’s easy to fix, thanks.

typo was not the point of the comment though :) also is there a demo or some sort of technical whitepaper.

Not yet. Is that something you’d find compelling? Anything in particular you’d like to see?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#164

Post by hackernewds »

dboreham wrote: post #143
washywashy wrote: post #67

I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some crit…

$6M/y is something like 20 heads (depending on where they are, could be more). So probably it's a win. Hard to see that this could take more than about 5. Add cost of hay and water of course.

is it? if you consider the value 20 engineers could drive instead in that time

if you assume they wouldn't have had anything else meaningful to work on during that time to save money, then you have a different problem in the company. $6M seems like the value 1 engineer can drive in a company at the scale of Uber

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#165

Post by k1t »

dboreham wrote: post #112
gronky_ wrote: post #47

Earlier quoted context omitted.

Yes, that’s definitely the main reason. It’s called “burying the lede”. Saving $6M is key information that makes this story interesting. It’s buried all the way at the bottom of the first blog and is completely missing from the second blog which focuses specifically on the migration

TaaS : title as a service

People have done this, eg https://www.reddit.com/r/GrowthHacking/comments/k20g42/ai_to...

However that appears to be defunct now

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#166

Post by qwertyuiop_ »

Assuming there are a minimum of two teams a total of 20 maintaining this in-house software, I gave 250k as cost per engineer (salary plus health and other benefit costs to the company). Thats $5 million right there. I am estimating lowest range. Thats why Amazon calls these efforts undifferentiated heavy lifting. is there a slight premium to pay than rolling your own and maintaining yes. Its worth all the trouble and security and management overhead into rolling your own.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#167

Post by superzamp »

ledgerdev wrote: post #162

Say you wanted to build an app on a database like LedgerStore but at much smaller scale, what are the best open source options out there right now?

We have a pretty minimal setup at formancehq/ledger[1] that uses pg as a storage backend, and comes with a programmability layer to model complex transactions (e.g. sourcing $100 from three accounts in cascade).

[1] https://github.com/formancehq/ledger

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#168

Post by bjornsing »

macspoofing wrote: post #150
bjornsing wrote: post #139

Earlier quoted context omitted.

Well the reason I’m working on this in the first place is that when I was on the other side of the table I was looking for one. I filled in the contact forms of a couple of different startups that had products somewhat in line with what I was looking for, and talked to their sales reps. Admittedly they weren’t as early stage as my project, but on the other hand they weren’t 100% focused on my use-case either. I guess…

>I guess what I’m trying to say is that I was hoping that someone with a write intensive workload would want to spend some time evaluating a product built specifically for that. Again, it's not clear to me exactly what it is you're doing that's any different from the plethora of existing off-the-shelf solutions. You're saying that you started this project/company because you were looking for a solution to a specific…

Well, it’s pretty simple: HaystackDB is designed from the ground up for write intensive workloads, so it’s much more economical than existing off-shelf-solutions for that type of workload. Is that not clear from the landing page?

From pricing: “$0.2 per million writes, $20 per million reads”. The typical cost profile is $2 per million read/writes, or even more for writes.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#169

Post by pests »

IanCal wrote: post #132
intunderflow wrote: post #41

Earlier quoted context omitted.

Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?

Other than the comments about titles, the entire blogpost doesn't show for me with ublock. So I'll open it, see a picture of some birds, scroll around for a bit then give up.

That's probably because you are running software that is meant to hide content on a page.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#170

Post by bastawhiz »

bjornsing wrote: post #7

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

> $20 per million reads

Quite frankly, this is not gonna work. I manage a system with a very write-heavy workload (lots of small writes) and even though our writes far outpace our reads, this pricing makes your system about ten times more expensive than an RDS cluster.

There's no data about performance. There's no information on how or whether data is persisted to durable storage before a write is acknowledged. There's not even any information on how big keys or values can be. There's no public information on support.

When choosing a system like yours, my priorities are:

1. Data safety

2. Performance

3. Cost

... In that order. You've done nothing to educate me on 1 and 2 and your pricing isn't better than what you're seeking to displace.

When your product is a tool for developers, show up with hard facts about your product. Zero people (as you've seen) are even remotely interested in building a product on top of a system without knowing whether the system will hold up to their use case. And other than a very anemic FAQ section, you have no documentation at all, whatsoever.

Post reply on HN