81–90 of 345 posts
Earlier quoted context omitted.
That's why you always test your backup. I backup the full sqlite.db every day and test the litestream replication every week. So far litestream have been solid.
Would you care to tell us what your backup and restore policy would be for 1.7 PB of data?
Congrats to anyone who worked on it! However, I'm guessing the cost of just running this team be quite large and not significantly different from the savings (6M), and add on top of it the overhead of maintenance. Payments would not likely be a long-term bet as well, so kind of interesting why teams take up such projects ? Is it some kind of sunk-cost with the engineering teams you already have?
Earlier quoted context omitted.
At one end of the spectrum, some people here claim to write this kind of software over a weekend. Some others claim they require a salary of $600,000, and still need nine additional colleagues to pull something like this off. There is a lot of room in between, where cost estimates are more realistic.
This answer pretty much sums a lot of my experience. Of course when the guy somehow pulls this off in 2 weeks it is seen as an easy side project with proof that it is, haha
I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/
One observation I have regarding your homepage is that the message isn't very clear. The headline doesn't mention any benefits I get from using your software. I think you should invest some time into improving your landing page and maybe you may see some traction. A good resource for this which I've bookmarked is here(1). Hope that helps. (1) https://www.indiehackers.com/post/my-step-by-step-guide-to-l...
To me, when I read the below, that just screams “save money”. But maybe I should do that conversation for the reader so to speak?
From benefits box: “Sometimes you need to index a huge amount of data, to accelerate just a few search queries. But building indexes and keeping them in hot storage can be expensive. HaystackDB builds only the indexes needed for sub-second query latency, across billions of keys, while keeping all your data in low-cost object storage like S3.”
I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/
Maybe find more articles like the above try connect to the respective people at said teams via LinkedIn and ask feedback
I’m working on a specialized data store[1] that would be perfect for this kind of use case (large “cold” storage with indexing). But I’m having trouble finding potential customers. I’ve tried Google search ads but got 99% spam and 1% potential investors, but 0% potential customers. If anybody has any ideas I’m all ears. 1. https://www.haystackdb.dev/
Some B2B and B2B2C products just don't work with walk-in leads. You need someone to create and chase down a set of leads.
I wonder if 1.7 petabytes of data (1T indexed records) could fit on a single (very) beefy baremetal server for under a couple thousand dollars a month, served by SQLite. Like this: https://use.expensify.com/blog/scaling-sqlite-to-4m-qps-on-a...
Given 30.7TB SSD’s are about $5500 each and you’d need 56 to to get to 1.7PB (with no redundancy). Not to mention that SQLite’s maximum DB size is 140TB. I don’t think you’d be able to fit this much storage into a single machine, especially not for a few thousand a month and SQLite wouldn’t be appropriate for this use-case.
I pretty much never see engineering salaries factored into these types of savings projects. I assume because engineers are already viewed as a sunk cost or maybe it’s just because it’s way less tangible. Have seen many designs describe how X saves Y dollars but ignores the engineering effort to maintain and build it. Half the time I suspect it’s just so people have something to work on, rather than it being some crit…
The original[1][2] articles are a better read IMO. The link is just a summary of the two with added spelling and grammatical errors that materially impact the meaning. 1. https://www.uber.com/blog/how-ledgerstore-supports-trillions... 2. https://www.uber.com/blog/migrating-from-dynamodb-to-ledgers...
Seems to happen with all our blog posts that appear on here (I work at Uber) - I don't get why the originals don't get upvoted but these rehashes do - are our titles just not as good?