I for one am enjoying this. Sure they could probably just use a regular ol' database, but where is the excitement in that!
Excitement is only for people who are learning.
A database for 2022
231–240 of 336 posts
Re: A database for 2022
#232Not commenting on tailscale, but for the state of databases. Sometimes boring is the right choice. PostgreSQL has worked for decades now, and seems to have regained much of the performance that MySQL once boasted. If you do this for the money, investing in tried and true (but boring) software should be the default solution. I watched with dread how the MongoDB fiasco played out a decade ago. Meanwhile, I kept using P…
> I watched with dread how the MongoDB fiasco played out a decade ago. Could you elaborate on this? MongoDB as a company is worth $30B, so it looks like they did at least some things right.
they did their marketing right.
Re: A database for 2022
#233I'm so desperate for a SQL database where I can just put it on a bunch of commodity hardware via Docker, connect them up and never worry about this again. Ideally it'd monitor my queries and create indexes for me. pleeeeeeaaaaaaaaaaaassssseeeeeeeeee /dream FoundationDB is similar but you have to do so much yourself. It's more or less what I'm describing for a key value store though. Not sure why it's not more popular…
Solving performance by mindlessly adding indexes works great as long as you don't have all too much data and have endless RAM.
The index backing the unique constraint on the URL table for my search engine is around 23 Gb. The entire server has 128 Gb of RAM, for comparison.
Re: A database for 2022
#234This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful. Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams t…
+1 on good design and programming. My go to language of choice is PHP/MySQL. I've never not been able to scale a project. At one point while working at Comcast I prototyped a system for every error code from every cable set top box (150 million) in the country on a MySQL server instance running on a MacBook Pro pulling data from Splunk in real time. All so I could generate some png charts and embed them in Slack. I k…
"know there are limits to MySQL"
I have gone past mysql limits a few times. Will you go beyond its capability is a simple measure. Can your hardware write fast enough to keep up? If your hardware maxes out writing to disk 200 megs per second, welp mysql maxes out at 200 megs per second. If you need 400 megs per second of writing, well you need two instances that can write independently. That is when you are fucked. At that point you get into sharding* and doing whacky things mysql isn't meant todo. Reading things from disk, mysql's replication scales just fine.
Re: A database for 2022
#235Not commenting on tailscale, but for the state of databases. Sometimes boring is the right choice. PostgreSQL has worked for decades now, and seems to have regained much of the performance that MySQL once boasted. If you do this for the money, investing in tried and true (but boring) software should be the default solution. I watched with dread how the MongoDB fiasco played out a decade ago. Meanwhile, I kept using P…
> I watched with dread how the MongoDB fiasco played out a decade ago. Could you elaborate on this? MongoDB as a company is worth $30B, so it looks like they did at least some things right.
Their paid managed service works well for enterprise SaaS companies who need It also works well for early stage Consumer Internet startups that haven't yet achieved huge growth but care a lot about developer productivity while churning features at high velocity.
But it gets blamed for reliability issues suffered by highly successful Consumer Internet companies that have achieved scale and have DAU, MAU in > 10M and have lot more than 4TB of data in a single cluster.
But that's basically a good problem to have at that stage. Usually such a company would have many more problems – monolithic application with monolithic database with huge unmanageable schema and indexes gone wild etc. Usual solution at that stage would be some sort of rearchitecture towards Microservices with multiple specialized databases for different use-cases – usually cloud hosted managed databases with horizontal scalability dedicated for online user-path workloads and separate OLAP tech stack for offline ETL/analytics workloads.
Re: A database for 2022
#236Earlier quoted context omitted.
This comment trivializing Dropbox: https://news.ycombinator.com/item?id=9224
That comment is unfairly judged, taking into account how much they used AWS instead of their own solution. "...Half-a-billion people stored files on Dropbox. Well, sort of. Really, the files were in Amazon’s cloud. .." [1] [1] "The Epic Story of Dropbox's Exodus From the Amazon Cloud Empire" https://www.wired.com/2016/03/epic-story-dropboxs-exodus-ama...
Why can't a company use cloud? It's not like there's Amazonbox client I could install now and get the same service. Dropbox never was about the storage itself, IMHO - it was about the ease of usage. There were enough options to get storage - cloud or non-cloud - when it came out, but none of them had the easy folder-based sync and sharing (public folder) of Dropbox.
Re: A database for 2022
#237Earlier quoted context omitted.
Managed DBs have tremendous lock in. Just try migrating off RDS with zero downtime. You can't, because they've "managed" your ability to configure external replicas. Then built a whole brittle data migration service that probably won't work for your DB.
they've already had to migrate twice due to questionable technology decisions. having to migrate again due to needing to change cloud providers seems a lot less likely than having to migrate again because their outside the box technology choice didn't pan out.
Choosing a technology that supports your business needs and growth, and that also lets you easily migrate to something else once it stops being suitable (or your requirements change, or you discover a way in which the tech doesn't work for you, or whatever), doesn't sound like a questionable technology decision, it sounds like a great one.
Re: A database for 2022
#238This thread has (at least at the moment) serious Bob Martin Sudoku Solver energy to it. Tailscale has solved an infamously complicated problem using, for the most part, simple tools. They're not just successful; they're remarkably successful, spookily successful, upsettingly successful. Consider whether the secret sauce here might not be au courant database choices, but rather something much harder for random teams t…
+1 on good design and programming. My go to language of choice is PHP/MySQL. I've never not been able to scale a project. At one point while working at Comcast I prototyped a system for every error code from every cable set top box (150 million) in the country on a MySQL server instance running on a MacBook Pro pulling data from Splunk in real time. All so I could generate some png charts and embed them in Slack. I k…
This comment makes me unreasonably giddy.
The replies are focused on MySQL (and to a lesser extent, PHP). None of that matters.
The hero here is the logging system serving as seekable stream of collected events. (I’m trying hard here to not say K-fka).
By consuming events from that source and being able to perform time-based seeking, you’ve offloaded (or dare I say, eliminated) your scaling pressures.
Even if you commit the “sins” of no indexes on heavy queries, using MyISAM, or building a giant, memory-intensive associative array… none of it matters.
You can choose to wipe your local dataset, start again in a different language, switch to another data store model, and none of your downstream consumers will know nor care (unless you blog about it).
*This* type of design, that prioritizes consumer-facing resiliency and continuity, is what I look for in systems that scale.
Re: A database for 2022
#239Earlier quoted context omitted.
Tailscale employee here. Our database needs are tiny, as explained in the earlier post. So we optimize for things like: "can we run all our tests quickly and easily in many environments without containers and VMs?" All three of our storage schemes have had that property. We have MySQL and PostgreSQL veterans on the team. We know those options well.
> without containers Why? The official docker postgres package weighs in at 100MB. Assuming you standardized on that, then every environment would end up having a fresh postgres image just waiting to be startup. Meaning, the actual cost is the memory for the server and startup time, not the image itself. Mount the data on tmpfs and you can startup and setup a db in very little time (I know, because that's what we do)…
[deleted]
Re: A database for 2022
#240Earlier quoted context omitted.
It would be interesting to known why a standard boring RDS setup wouldn’t solve their problem completely. In fact I would be more interested to understand that than the actual details of sqlite tailing. (I think the reason they gave was vendor lock-in, but apart from that, I didn’t understand why it wouldn’t be adequate)
One obvious reason not to use RDS is wanting disk-local caches of information replicated from a single leader, rather than having every single machine in your fleet calling out to an external service on every read . That's certainly why we're not considering Postgres in our infrastructure, even though managed Postgres is a product we in fact offer. We use Postgres! It's the backing state for our API, and an important…
Sounds like postgres listen/notify could be viable for your high-read-low-write use case? Or it is not scalable enough for the fleet size?