If you liked this I highly recommend "Clean Architecture" by Uncle Bob. He has a great story of how they kept putting off switching to a SQL DB on a project and then never needed to and it was a big success anyway.
An Unlikely Database Migration
41–50 of 190 posts
Re: An Unlikely Database Migration
#42> The file reached a peak size of 150MB
is this a typo? 150MB is such a minuscule amount of data that you could do pretty much anything and be OK.
Re: An Unlikely Database Migration
#43Earlier quoted context omitted.
SQL is fine. We use it for some things. But not writing SQL is easier than writing SQL. Our data is small enough to fit in memory. Having all the data in memory and just accessible is easier than doing SQL + network round trips to get anything. ORMs: consider yourself lucky. They try to make SQL easy by auto-generating terrible SQL. Testing latency: we want to run many unit tests very quickly without high start-up co…
Ah, we don't use Docker or any other container technology. Maybe that is why we aren't seeing the latency issues you are referring to.
Docker itself doesn't add much latency. It just makes getting MySQL and PostgreSQL easier. If anything, it helps with dependencies. The database server startup still isn't great, though.
Re: An Unlikely Database Migration
#44Interesting choice of technology, but you didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica. (You could probably start with either one and easily migrate to the other one if needed.) In particular you've designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-…
> and a tailetc cluster What do you mean by this part? tailetc is a library used by the client of etcd. Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. (I previously made LiveJournal and ran its massively sharded HA MySQL setup)
> Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config.
What if you used one of the managed RDBMS services offered by the big cloud providers? BTW, if you don't mind sharing, where are you hosting the control plane?
Re: An Unlikely Database Migration
#45> The file reached a peak size of 150MB is this a typo? 150MB is such a minuscule amount of data that you could do pretty much anything and be OK.
Not a typo. See why we're holding it all in RAM?
But writing out 150MB many times per second isn't super nice when both 150MB and the number of times per second are both growing.
Re: An Unlikely Database Migration
#46Unfortunately this company is known for its shady agressive marketing on hacker news. The upvotes count is really suspicious and this is not the first time. EDIT: Why is the downvoting? the post was given like 9 upvotes in the first 5 minutes. I frequently go to "new" and this is a highly suspicious behaviur.
Its ok im part of audience that doesnt know much about databases its fun to read discussions about them once in a while.
I learned a lot about postgresql redis clickhouse and elasticsearch here, people's perspectives here are great to learn from, they tell you which to avoid and which to try.
Re: An Unlikely Database Migration
#47Earlier quoted context omitted.
> and a tailetc cluster What do you mean by this part? tailetc is a library used by the client of etcd. Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. (I previously made LiveJournal and ran its massively sharded HA MySQL setup)
> Running an etcd cluster is much easier than running an HA PostgreSQL or MySQL config. What if you used one of the managed RDBMS services offered by the big cloud providers? BTW, if you don't mind sharing, where are you hosting the control plane?
> What if you used one of the managed RDBMS services offered by the big cloud providers?
We could (and likely would, despite the costs) but that doesn't address our testing requirements.
The control plane is on AWS.
We use 4 or 5 different cloud providers (Tailscale makes that much easier) but the most important bit is on AWS.
Re: An Unlikely Database Migration
#48Unfortunately this company is known for its shady agressive marketing on hacker news. The upvotes count is really suspicious and this is not the first time. EDIT: Why is the downvoting? the post was given like 9 upvotes in the first 5 minutes. I frequently go to "new" and this is a highly suspicious behaviur.
I upvote literally everything I see from Tailscale, because I am hugely impressed with Avery Pennarun and David Crawshaw. Seeing Fitz is there too takes it to an even higher level. I want them to succeed, and I think they create good HN content. Major kudos.
Re: An Unlikely Database Migration
#49The post touches upon it, but I didn't really understand the point. Why doesn't synchronous replication in Postgres work for this use case? With synchronous replication you have a primary and secondary. Your queries go to the primary and the secondary is guaranteed to be at least as up to date as the primary. That way if the primary goes down, you can query the secondary instead and not lose any data.
That would have been considerably less scalable. etcd has some interesting scaling characteristics. I posted some followup notes on twitter here: https://twitter.com/apenwarr/status/1349453076541927425
How is PostgreSQL (or MySQL) "considerably less scalable" exactly? etcd isn't particularly known for being scalable or performant. I'm sure it's fast enough for your use-case (since you've benchmarked it), but people have been scaling both PostgreSQL and MySQL far beyond what etcd can achieve (usually at the cost of availability of course).
Re: An Unlikely Database Migration
#50This post touches on "innovation tokens". While I agree with the premise of "choose boring technology", it feels like a process smell, particularly of a startup whose goal is to innovate a techology. Feels demotivating as an engineer if management says our team can only innovate an arbitrary N times.
What is the purpose of a startup? Is it to put keywords on your resume, or is it to create a product?