Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Migrating Uber's ledger data from DynamoDB to LedgerStore

uber.com

Post reply on HN
Print view View original

181–190 of 345 posts

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#181

Post by bjornsing »

victor106 wrote: post #176
bjornsing wrote: post #7

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

I would seriously consider an open source business model with an appropriate licensing model. I see lot of companies are open to experimenting with open source db's.

Good point. I’m thinking about releasing an open source (or source available) “frontend” for it, and just charge for the “cold storage backend”. How would you feel about that?

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#182

Post by oldprogrammer2 »

bjornsing wrote: post #7

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

The homepage could benefit from more tangible examples, because right now I can't discern where it fits into my current stack. For most companies, it would be replacing something in a specific context.

Like a side-by-side example. Doing "work" on BigTable (show code examples) versus doing the same "work" on Haystack. Then show the specific metrics on how Haystack is cheaper/faster/better.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#183

Post by ddorian43 »

_xnmw wrote: post #173
khaki54 wrote: post #103

Earlier quoted context omitted.

The value proposition of commercial loud isn't cost savings unless you manage to quantify all of the ancillary and extrinsic factors such as security risk, HVAC, datacenter personnel, and hardware lifecycle. Any well capitalized and organized company could build their own cloud much more cheaply, but really a significant portion of the calculation is outsourcing the risk components.

The problem with outsourcing the risk components is that you don't know for sure whether they are properly taken care of. Major cloud providers have been caught "oopsing" your data, and bam it is gone. Furthermore, they have no incentive to be more efficient about it, they could easily be using 10x the amount of resources necessary, and you wouldn't even have a clue, you're just paying for evermore expensive crap tha…

But the cloud providers compete with other other! Look at the efficient market in display in their bandwidth pricing!

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#184

Post by IanCal »

nvr219 wrote: post #180
IanCal wrote: post #132

Earlier quoted context omitted.

Other than the comments about titles, the entire blogpost doesn't show for me with ublock. So I'll open it, see a picture of some birds, scroll around for a bit then give up.

Loads fine for me with ublock. Perhaps you have a custom rule blocking something?

Nothing custom, so it must be on a list somewhere.

edit - it doesn't have to really be blocking the actual post here even, if their loading code breaks when some other tracking code doesn't run, that could explain it.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#185

Post by xyst »

Uber must have picked up some Google rejects. This type of homegrown project was seen at Google all the time.

Usually to aim for a significant promotion.

“Designed and built homegrown system to save $Xm! Give me promo, bro?”

Just so happened to ignore that it took X+Y additional to build. Also it will probably be going to the G graveyard in a few years.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#186

Post by bjornsing »

bastawhiz wrote: post #170
bjornsing wrote: post #7

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

> $20 per million reads Quite frankly, this is not gonna work. I manage a system with a very write-heavy workload (lots of small writes) and even though our writes far outpace our reads, this pricing makes your system about ten times more expensive than an RDS cluster. There's no data about performance. There's no information on how or whether data is persisted to durable storage before a write is acknowledged. There…

All valid points. I guess I’m hesitant to put time into documentation and similar, if I can’t somehow find a steady stream of sales prospects.

> even though our writes far outpace our reads, this pricing makes your system about ten times more expensive than an RDS cluster

That indeed sounds off… Are you sure you’re comparing the total cost to that of an RDS cluster? I am aware that reads will be more expensive (due to the write optimization), but I was hoping most customers would make it back on cheap writes. Also the storage itself ($0.23 per GB-month) should be much cheaper than RDS.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#187

Post by jeltz »

szundi wrote: post #51
thangngoc89 wrote: post #43

Earlier quoted context omitted.

One could use Litestream [1] [1]: https://litestream.io

What if a continuous replication system has a bug one day, and you realize you are just a bit corrupted and have to rerun? Or is it the same with cloud tools?

Then the same happens as when there is a bug in Aurora's replication. You lose data. I know this from personal experience.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#188

Post by IanCal »

pests wrote: post #169
IanCal wrote: post #132

Earlier quoted context omitted.

Other than the comments about titles, the entire blogpost doesn't show for me with ublock. So I'll open it, see a picture of some birds, scroll around for a bit then give up.

That's probably because you are running software that is meant to hide content on a page.

What's the purpose of this comment?

My point is that a random dev running a pretty plain adblock (aren't we all?) simply cannot view their post. This is down to uber, their practices, an external developer and how uber create their blog (they don't just have the content in the page). If I'm not a special case with extremely weird luck, a bunch of devs seeing links to their posts will open them and not see any actual content. They will then, I assume, be less likely to upvote them.

Given that they are seeing problems with posts being upvoted this seems somewhat relevant.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#189

Post by cess11 »

bjornsing wrote: post #7

I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/

To consider your product an alternative I'd like to see benchmarks that seem trustworthy, something like a Jepsen analysis or case studies at existing customers, and be able to test it within the EU, i.e. not on US:ian services.

Seems you're in the vicinity of Lund, should be a 'science park' or similar close to the uni where you can find companies that have problems you could solve. Talk to 'incubators', 'accelerators' and the like there.

Re: Migrating Uber's ledger data from DynamoDB to LedgerStore

#190

Post by minkzilla »

robocat wrote: post #177
mbesto wrote: post #141

Earlier quoted context omitted.

$6m in perpetuity is like infinite and engineers can be fired.

It really isn't. The first years dominate the value and later years are worth nothing due to inflation. Google a calculator and use a reasonable discount rate and I suspect you will find that todays value for an infinite perpetuity is a lot less than your intuition might guess. It always surprises me.

But they will save more than $6M the second year because AWS will up their prices.
Post reply on HN