Ahh, a classic case of "if you don't understand it, you are bound to reimplement it".
An Unlikely Database Migration
171–180 of 190 posts
Re: An Unlikely Database Migration
#172> "Attempts to avoid this with ORMs usually replace an annoying amount of typing with an annoying amount of magic and loss of efficiency." People seem to keep using poorly-designed ORMs or are stuck with some strange anti-ORM ideology. Modern ORMs are fast, efficient, and very productive. If you're working with relational databases then you're using an ORM. It's a question of whether you use something prebuilt or wri…
> Modern ORMs are fast, efficient, and very productive. The author seems to be using Go, which honestly could use work in that area. gorm is the biggest / most popular ORM out there, but it looks like a one-person project, the author seems well worn-out already, and it kinda falls apart when you work with a data model more than one level deep. Plus broadly speaking, there seems to be a bit of an aversion to using lib…
GORM is nice for beginners because it's pretty easy to get started with, but in my experience starts to fall apart when you need to do anything more complex or start to scale up. I've had good experiences with SQLBoiler [0] on the other hand. I haven't used it in for anything in production, but it's been a breeze to use in a couple of personal projects, and it handles complex SQL queries much better compared to GORM.
Re: An Unlikely Database Migration
#173Earlier quoted context omitted.
I mean that's an `apt install postgres` or `brew install postgres` away. Takes about 5 minutes. I guess it could become a pain if you need to work with multiple different versions at once.
Being deep in the cloud world right now, with aws and terraform and kubernetes cli tools, etc, not having to install third party tools on my machine does sound pretty great, but also entirely unrealistic. Managing local DBs once new versions are out and your server isn't upgraded yet is irritating, but when I'm using a Mac I'd still rather use a native DB than Docker because of the VM overhead, since I've not yet run…
> I've not yet run into a bug caused by something like "my local postgres was a different version than the server was."
Ran into that at a recent place - the code was doing "= NULL" in a bunch of places (before my time) and PG12 treated that differently than PG11 did which broke a bunch of tests.
Re: An Unlikely Database Migration
#174Earlier quoted context omitted.
Running all of our control server tests (including integration tests) right now takes 8 seconds, and we're not even incredibly happy with that. There's no reason it should even be half that. So it's not really in our patience budget for adding a mysqld or postgres start up (possible docker pull, create its schema, etc).
>right now takes 8 seconds, and we're not even incredibly happy with that With the amount of explaining and skepticism you're having to deal with in most of the threads here (plenty of reasonable questions, some seem to approach the question with the assumption that your approach is totally wrong) I feel compelled to comment on how nice such a fast feedback loop would be just so it's known that you're not listing the…
Yeah, I'm inspired to reduce the cost of my tests to get to a better place as a result of this.
Mind you, I build ML/statisical models, so my integration/e2e tests are definitely not going to get down to 8 seconds.
Re: An Unlikely Database Migration
#175Earlier quoted context omitted.
if you want a distributed key/value data store, you want to use what's already out there and vetted. It use to be zookeeper, but etcd is much simpler and that's what Kubernetes uses and it has been a big success and proved itself out there in the field. Definitely easier than a full SQL database which is overkill and much harder to replicate especially if you want to have a cluster of >= 3. Again, key is "distributed…
It's overkill until it's not. We chose etcd initially but after a while we started wanted to ask questions about our data that weren't necessarily organised in the same way as the key/value hierarchy. That just moved all the processing to the client, and now I just wish we used a SQL database from the beginning.
Yeah, but for their use case it's just KV and also ability to link directly in go
Re: An Unlikely Database Migration
#176I am missing a lot of context from this post because this just sounds nonsensical. First they're conflating storage with transport. SQL databases are a storage and query system. They're intended to be slow, but efficient, like a bodybuilder. You don't ask a bodybuilder to run the 500m dash. Second, they had a 150MB dataset, and they moved to... a distributed decentralized key-value store? They went from the simplest…
I'd answer questions but I'm not sure where to start. I think we're pretty well aware of the pros and cons of all the options and between the team members designing this we have pretty good experience with all of them. But it's entirely possible we didn't communicate the design constraints well enough. (More: https://news.ycombinator.com/item?id=25769320 ) Our data's tiny. We don't want to do anything to access it. I…
You say you want to have a small database just for each control process deployment to be independent. But you need multiple nodes for etcd... So you currently have either a shared database for all control processes, or 3 nodes per control process, or 3 processes per control node, etc. Either way it seems weird.
I get that SQLite wouldn't work, but it also doesn't make sense to have one completely independent database per process. So I imagine you're using a shared database, at which poitlnt etcd starts to make more sense. It's just not that widely understood in production as sql databases, and has limitations which you might reach in a few years.
Re: An Unlikely Database Migration
#177Earlier quoted context omitted.
Yes, and implementing that is exactly the point of Tailscale, with the added advantage of not relying on a centralized proxy.
You seem to be confused between zerotrust and encryption. Zerotrust is about auhtentication/authorization at the application level. Also tailscale is as centralized as Cloudflare et al. What happens when tailscale servers go down? Can 2 peers behind NAT still be able to connect to each other? can they synchronize each other's public endpoint and public key?
It's amazing that people are downvoting simple technical facts to appeal to others without even providing a single technical argument. It's as if I stated a political statement or something.
Re: An Unlikely Database Migration
#178I do like putting .json files on disk when it makes sense, as this is a one-liner to serialize both ways in .NET/C#. But, once you hit that wall of wanting to select subsets of data because the total dataset got larger than your CPU cache (or some other step-wise NUMA constraint)... It's time for a little bit more structure. I would have just gone with SQLite to start. If I am not serializing a singleton out to disk,…
I've seen the same when at one point we decided to just store most data in a JSON blob in the database, since "we will only read and write by ID anyway". Until we didn't, sigh. At least Postgres had JSON primitives for basic querying. The real problem with that project was of course trying to set up a microservices architecture where it wasn't necessary yet and nobody had the right level of experience and critical th…
Storing JSON blobs in the database can be the best option if you are careful with your domain modeling.
Re: An Unlikely Database Migration
#179Re: An Unlikely Database Migration
#180Earlier quoted context omitted.
Docker itself doesn't add much latency. It just makes getting MySQL and PostgreSQL easier. If anything, it helps with dependencies. The database server startup still isn't great, though.
If you don't use Docker, you can just leave the database server running in the background, which removes the startup latency (you can of course do this with Docker too, but Docker has a tendency to use quite a few resources when left running in the background, which a database server on it's own won't).
Would love to know what resources you are speaking of here.
We've definitely done some whack-a-mole with allocations in the engine, and of course there's always things getting changed/added still.