Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

An Unlikely Database Migration

tailscale.com

Post reply on HN
Print view View original

21–30 of 190 posts

Re: An Unlikely Database Migration

#21

Post by bradfitz »

marcinzm wrote: post #19
crawshaw wrote: post #6

Earlier quoted context omitted.

It is a tricky tradeoff for startups. On the one hand, a startup has very limited resources and so has to focus on the business. On the other hand, a startup has to experiment to find the business. I don't think there's an easy answer. In our case, the control plane data store really should be as boring as possible. It was real stretch using anything other than MySQL. We tried to lay out the arguments in the post, bu…

I'm used to Scala for these things and it seemed fairly easy to do. Docker DB container would be spun up at the start of testing (using a package such as testcontainers-scala). Then the DB would be setup using the migration scripts (using Flyway) so it has a known state. Every test (or set of tests if they're linked) would have a pre-hook which nuked the DB to a clean state. Then the docker container would be shut do…

I wrote about that in the article. Search for "dockertest".

We considered that option but didn't like it. It still wasn't fast enough, and placed onerous (or at least annoying) dependencies on future employees.

Re: An Unlikely Database Migration

#22

Post by malisper »

The post touches upon it, but I didn't really understand the point. Why doesn't synchronous replication in Postgres work for this use case? With synchronous replication you have a primary and secondary. Your queries go to the primary and the secondary is guaranteed to be at least as up to date as the primary. That way if the primary goes down, you can query the secondary instead and not lose any data.

Re: An Unlikely Database Migration

#23

Post by nwsm »

npiit wrote: post #2

Unfortunately this company is known for its shady agressive marketing on hacker news. The upvotes count is really suspicious and this is not the first time. EDIT: Why is the downvoting? the post was given like 9 upvotes in the first 5 minutes. I frequently go to "new" and this is a highly suspicious behaviur.

Is tailscale downvoting you as well?

Re: An Unlikely Database Migration

#24

Post by bradfitz »

malisper wrote: post #22

The post touches upon it, but I didn't really understand the point. Why doesn't synchronous replication in Postgres work for this use case? With synchronous replication you have a primary and secondary. Your queries go to the primary and the secondary is guaranteed to be at least as up to date as the primary. That way if the primary goes down, you can query the secondary instead and not lose any data.

We could've done that. We could've also used DRBD, etc. But then we still have the SQL/ORM/testing latency/dependency problems.

Re: An Unlikely Database Migration

#25

Post by marcinzm »

jamescun wrote: post #3

This post touches on "innovation tokens". While I agree with the premise of "choose boring technology", it feels like a process smell, particularly of a startup whose goal is to innovate a techology. Feels demotivating as an engineer if management says our team can only innovate an arbitrary N times.

I mean, what do you propose in terms of getting a diverse group of engineers to trade off innovation versus choosing boring technologies? Keeping in mind that how much risk the company is willing to take is not the decision of engineers but the executive team. Innovation tokens convey the level of risk the executive team is willing to take in terms of technologies. The alternative I've often seen is a dictatorial CTO (or VP of Eng) who simply says NO a lot which is a lot more demotivating. A large company may do detailed risk analyses but those are too cumbersome for a startup.

Re: An Unlikely Database Migration

#26

Post by [deleted] »

bob1029 wrote: post #5

I do like putting .json files on disk when it makes sense, as this is a one-liner to serialize both ways in .NET/C#. But, once you hit that wall of wanting to select subsets of data because the total dataset got larger than your CPU cache (or some other step-wise NUMA constraint)... It's time for a little bit more structure. I would have just gone with SQLite to start. If I am not serializing a singleton out to disk,…

[deleted]

Re: An Unlikely Database Migration

#28

Post by apenwarr »

malisper wrote: post #22

The post touches upon it, but I didn't really understand the point. Why doesn't synchronous replication in Postgres work for this use case? With synchronous replication you have a primary and secondary. Your queries go to the primary and the secondary is guaranteed to be at least as up to date as the primary. That way if the primary goes down, you can query the secondary instead and not lose any data.

That would have been considerably less scalable. etcd has some interesting scaling characteristics. I posted some followup notes on twitter here: https://twitter.com/apenwarr/status/1349453076541927425

Re: An Unlikely Database Migration

#29

Post by lima »

npiit wrote: post #9
hfern wrote: post #4

Earlier quoted context omitted.

I don't think it's shady marketing. The company is composed of several prominent software engineers (eg Brad Fitzpatrick of memcached fame). I think many folks on HN (myself included) are interested to see what they are working on. Especially as Brad left Google to work on it.

I understand. But given the product features compared to the rest of the industry, Tailscale brings no real value compared to Zero-tier hen it comes to meshes, it's not zerotrust like Cloudflare, Twingate, and many others, it claims to be open source while only the client is and it cannot be used without their closed source control plane where most of the feature are behind paywall, it's way more expensive than reput…

It's more zerotrust-y than Cloudflare et al since it's entirely P2P, with only the control plane running in the cloud.

Compared to ZeroTier, the Tailscale client has a permissive license, the mesh is fully routed (vs. a L2 network with unencrypted broadcasts), is written in a memory-safe programming language, integrates with company SSO, and uses the Wireguard protocol (i.e. sane, audited crypto instead of a DIY protocol).

Re: An Unlikely Database Migration

#30

Post by judofyr »

Interesting choice of technology, but you didn't completely convince me to why this is better than just using SQLite or PostgreSQL with a lagging replica. (You could probably start with either one and easily migrate to the other one if needed.)

In particular you've designed a very complicated system: Operationally you need an etcd cluster and a tailetc cluster. Code-wise you now have to maintain your own transaction-aware caching layer on top of etcd (https://github.com/tailscale/tailetc/blob/main/tailetc.go). That's quite a brave task considering how many databases fail at Jepsen. Have you tried running Jepsen tests on tailetc yourself? You also mentioned a secondary index system which I assume is built on top of tailetc again? How does that interact with tailetc?

Considering that high-availability was not a requirement and that the main problem with the previous solution was performance ("writes went from nearly a second (sometimes worse!) to milliseconds") it looks like a simple server with SQLite + some indexes could have gotten you quite far.

We don't really get the full overview from a short blog post like this though so maybe it turns out to be a great solution for you. The code quality itself looks great and it seems that you have thought about all of the hard problems.

Post reply on HN