Looks like a clear case of migration to postgres. A single table with jsonb. You can do indexes on jsonb and a few plpgsql for partial updates, and forget about it for a while.
An Unlikely Database Migration
101–110 of 190 posts
Re: An Unlikely Database Migration
#102Earlier quoted context omitted.
If you don't use Docker, you can just leave the database server running in the background, which removes the startup latency (you can of course do this with Docker too, but Docker has a tendency to use quite a few resources when left running in the background, which a database server on it's own won't).
So then every engineer needs to install & maintain a database on their machines. (Hope they install the right version!) I mean, that's what my old company did pre-Docker. It works, but it's tedious.
There is a middle path -- use lxd. This way, the mysqld/postgres process can always be running, while the dev machine remains clean.
(But of course the battery will drain a little faster if it is a laptop)
Re: An Unlikely Database Migration
#103I use the same system (a JSON file protected with a mutex) for an internal tool I wrote, and it works great. For us, file size or request count is not a concern, it's serving a couple (internal) users per minute at peak loads, the JSON is about 150 kb after half a year, and old data could easily be deleted/archived if need be. This tool needs to insert data in the middle of (pretty short) lists, using a pretty compli…
As long as you're guaranteeing correctness[0], it's hard to disagree with the "simple" approach. As long as you don't over-promise or under-deliver, there's no problem, AFAICS.
[0] Via mutex in your case. Have you thought about durability, though. That one's actually weirdly difficult to guarantee...
Re: An Unlikely Database Migration
#104Earlier quoted context omitted.
How is PostgreSQL (or MySQL) "considerably less scalable" exactly? etcd isn't particularly known for being scalable or performant. I'm sure it's fast enough for your use-case (since you've benchmarked it), but people have been scaling both PostgreSQL and MySQL far beyond what etcd can achieve (usually at the cost of availability of course).
[I work at Tailscale] I only mean scalable for our very specific and weird access patterns, which involves frequently read-iterating through a large section of the keyspace to calculate and distribute network+firewall updates. Our database has very small amounts of data but a very, very large number of parallel readers. etcd explicitly disclaims any ability to scale to large data sizes, and probably rightly so :)
Not OP, but if I could ask further... How much consistency can you tolerate on your reads? From the use case you mention I imagine... quite a lot, but you could risk locking yourself out of networks/systems if you get it wrong?
EDIT: Negation is important
Re: An Unlikely Database Migration
#105Earlier quoted context omitted.
If you don't use Docker, you can just leave the database server running in the background, which removes the startup latency (you can of course do this with Docker too, but Docker has a tendency to use quite a few resources when left running in the background, which a database server on it's own won't).
So then every engineer needs to install & maintain a database on their machines. (Hope they install the right version!) I mean, that's what my old company did pre-Docker. It works, but it's tedious.
Do the developers not have an internet connection? Why can’t they just hit a database running in the cloud or your own server somewhere?
Re: An Unlikely Database Migration
#106This reminds me of this post from the hostifi founder, sharing the code they used for the first 3 years:
https://twitter.com/_rchase_/status/1334619345935355905
It’s just 2 files.
Sometimes it’s better to focus on getting the product working, and handle tech debt later.
Re: An Unlikely Database Migration
#107Earlier quoted context omitted.
zerotrust has nothing to do with p2p, zero-trust is about making sure that this user is authorized to access that application at the resource level not using some decades old segmentation/network level policies. Zerotier also claims to be zerotrust but it's technically not. Cloudflare, Citrix, PulseSecure have zerotrust offerings, but many others sadly just claim to be either by ignorance or dishonesty.
Yes, and implementing that is exactly the point of Tailscale, with the added advantage of not relying on a centralized proxy.
You seem to be confused between zerotrust and encryption. Zerotrust is about auhtentication/authorization at the application level. Also tailscale is as centralized as Cloudflare et al. What happens when tailscale servers go down? Can 2 peers behind NAT still be able to connect to each other? can they synchronize each other's public endpoint and public key?
Re: An Unlikely Database Migration
#108Earlier quoted context omitted.
Why are your unit tests touching a database? I’m a real stickler about keeping unit tests isolated, because once I/O gets involved, they invariably become much less reliable and as you mention, too slow.
Sorry, I should've just said tests. Our tests overall are a mix of unit tests and integration tests and all sizes in between, depending on what they want to test.
I see. That makes more sense.
Re: An Unlikely Database Migration
#109Earlier quoted context omitted.
Neat. This is very similar to [0], which is _not_ a cache but rather a complete mirror of an Etcd keyspace. It does Key/Value decoding up front, into a user-defined & validated runtime type, and promises to never mutate an existing instance (instead decoding into a new instance upon revision change). The typical workflow is do do all of your "reads" out of the keyspace, attempt to apply Etcd transactions, and (if nee…
Drat! I went looking for people doing something similar when I sat down to design our client, but did not find your package. That's a real pity, I would love to have collaborated on this. I guess Go package discovery remains an unsolved problem.
> I guess Go package discovery remains an unsolved problem.
Or did you just not really search, like most of us excited to DIY? :-D
Godoc is pretty good, the package shows up for the searches I'd probably do in a similar situation.
Re: An Unlikely Database Migration
#110Earlier quoted context omitted.
[I work at Tailscale] I only mean scalable for our very specific and weird access patterns, which involves frequently read-iterating through a large section of the keyspace to calculate and distribute network+firewall updates. Our database has very small amounts of data but a very, very large number of parallel readers. etcd explicitly disclaims any ability to scale to large data sizes, and probably rightly so :)
Not OP, but if I could ask further... How much consistency can you tolerate on your reads? From the use case you mention I imagine... quite a lot, but you could risk locking yourself out of networks/systems if you get it wrong? EDIT: Negation is important
I've always found it hard to reason about relaxing consistency, and I think people underestimate how much complexity they take on by moving away from serializable isolation towards something looser. (Fun fact! Many databases out of the box don't correctly handle the classic transaction example -- read an account balance, subtract an amount from the balance, and then add the amount to another account's balance.)
Usually people design their app with the expectation of strict serializable isolation, relax it because of some production emergency, and then deal with the business consequences of the database doing the wrong thing until the company goes out of business (usually not due to database isolation levels, to be fair).