Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

71–80 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#71

Post by crabbone »

ForHackernews wrote: post #39

Does no one ever delete data? It's hard to believe there's much business value in keeping every individual payment record dating back to 2017.

In systems that deal with money, money-related data is virtually never deleted. The reason is the fear that deletion can be exploited somehow in the future, rather than the old data being actionable.

For example, if a customer registers with the name of a deleted customer, which will resurface some "unfinished" transactions or rules associated with the older version of the "same" customer that haven't been properly deleted but appeared to be deleted for a while.

Also, in general, deletion is very difficult because money doesn't just disappear. You'd need some sort of compaction (think: Git squash) rather than deletion to be able to balance the system's books... but then you'd be filling the system with fake transactions...

From my experience from working with these kinds of systems, the typical solution is to label entities with active/inactive labels to substitute deletion. But entities never go away.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#72

Post by HeatrayEnjoyer »

zinglersen wrote: post #45
deadbabe wrote: post #31

Earlier quoted context omitted.

Someone probably gets fired so I guess someone does pay the ultimate price.

Losing your job because the outcome of your efforts (or even external events) is not what I would call the ultimate price. "The metaverse division has now lost more than $45 billion since the end of 2020" Your compensation for your work is your salery. So I would say that it's fair that the actual risk taker is benefiting from the potential rewards?

The "risk takers" are not taking at any risk at all. What's the chance they end up on the street, or even suffer personal financial stress about their life? That they will have to move, sell their car, home, etc. It's 0%.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#73

Post by bjornsing »

emerongi wrote: post #56
bjornsing wrote: post #7

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

Besides all the other good feedback here, I will offer my extremely petty reason why I wouldn't spend much time evaluating this product. In the FAQ, under the "Are transactions fully ACID?" heading, there is a typo: "simultaineously". It gives me the impression that not enough care has gone into an important part of this product. I know it's not a fair jugdgement, but first impressions matter.

That’s easy to fix, thanks.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#74

Post by mlnj »

sgt wrote: post #35
BlackLotus89 wrote: post #33

Earlier quoted context omitted.

No it won't. sqlite "only" works with up to 281TB [0] [1] [0] https://www.sqlite.org/releaselog/3_33_0.html [1] https://www.sqlite.org/limits.html (#12)

You can split up into 10 SQLite DB's on this individual server.

Just storing petabytes of data is not the issue. Managing and querying it reliably is.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#75

Post by mickeyp »

thangngoc89 wrote: post #62
szundi wrote: post #51

Earlier quoted context omitted.

What if a continuous replication system has a bug one day, and you realize you are just a bit corrupted and have to rerun? Or is it the same with cloud tools?

That's why you always test your backup. I backup the full sqlite.db every day and test the litestream replication every week. So far litestream have been solid.

Would you care to tell us what your backup and restore policy would be for 1.7 PB of data?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#76

Post by bjornsing »

smokel wrote: post #20
bjornsing wrote: post #7

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

With a disclaimer that I have no formal nor practical background in marketing, here are some ideas: 1. It is a bit unclear to me when I would use Haystack. The main advantage seems to be cost cutting. It would be nice to see some realized examples of this. 2. When competing for price, you may look like the cheap, and thereby untrusted alternative. There is a risky business paradox here, for which I am sure a fellow H…

1. Good point, thanks.

2. True. One reason I haven’t priced it ridiculously cheap is to avoid this judgement, and fate. With this pricing I won’t necessarily have a smaller profit margin than competitors. The cost advantage comes from a smarter architecture. Any ideas on how I can communicate that would be greatly appreciated.

3. I used to work for one that needed it. I’ve also interviewed at one that had the same problem. A bit hesitant to reach out to potential customers though before I have a solid product I can deliver. But perhaps I shouldn’t be?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#77

Post by debarshri »

There was an era around 2015, when all the cool tech companies like netflix, spotify, soundcloud, uber and others were building alot of infrastructure and database tools. Nowadays, engineers often talk in AWS/Cloud terminologies.

It is breathe of fresh air to see that orgs are still building tools like that.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#78

Post by bjornsing »

csomar wrote: post #46
bjornsing wrote: post #7

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

Looking at pricing, it's crazy expensive (and that comparing to AWS, which is crazy expensive). How do you justify that?

The idea is that it should be about a tenth of the cost compared to S3 or DynamoDB. Is that not how you read the pricing? Or do you just think that’s still too expensive?

EDIT: Or maybe it’s because reads are expensive? That’s a consequence of the write optimization. The idea is that potential customers will be doing 90%+ writes.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#79

Post by thangngoc89 »

zaphirplane wrote: post #69
thangngoc89 wrote: post #62

Earlier quoted context omitted.

That's why you always test your backup. I backup the full sqlite.db every day and test the litestream replication every week. So far litestream have been solid.

By the time the TB is restored, time to start the next test How do you detect restored but bit flipped data ?

I do this in backup testing:

    sqlite3 /path/to/db
    sqlite> PRAGMA integrity_check;
See SQLite3 documentation: https://www.sqlite.org/pragma.html#pragma_integrity_check

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#80

Post by bjornsing »

padjo wrote: post #30
sanswork wrote: post #16

Earlier quoted context omitted.

You need to be doing enterprise sales not marketing. There is a lot of advice here and in general on that but you definitely need to be making calls with that type of business.

Yep nobody with the problem you’re offering to solve is going to solve it by googling and picking some random company they’ve never heard of with no track record.

Not even click a search ad and fill in a contact form? When I’m on the other side of the table I do that. But perhaps I’m unique in that aspect?

(I understand there won’t be any significant business without enterprise sales. But that’s not what I’m looking for at this stage.)

Post reply on HN