I worked at a company for a while that used QLDB as the primary system of record. The idea is great but the problem is that due to performance and other QLDB limitations all data had to be mirrored to an RDBMS via a streaming/queuing system, and there always were programmatic errors in interpreting data arriving for import into the RDBMS ... text field too long for RDBMS field; wrong data type or overflowing integer; invalid text encoding; ... Etc. These errors had to be noticed, debugged, fixed, and data had to be re-streamed. In the meantime official transactions were missing from the RDBMS side, which was used for reporting, driving the UI, deriving monetary obligations, etc. it was not worth the trouble. (I was lucky to not be involved in that design or implementation.)
Databases in 2024: A Year in Review
131–140 of 220 posts
Re: Databases in 2024: A Year in Review
#132Wow, the reasons why Redis commands API suck in Andy's video (linked in the post) are the weakest ever. It is possible to make a case against the Redis API (I would not agree of course but... it's totally legitimate), but you gotta have stronger arguments than those, particularly if you are a teacher of some kind. Especially: you need to be somewhat fluent in Redis and how developers use Redis in order to understand…
> Wow, the reasons why Redis commands API suck in Andy's video (linked in the post) are the weakest ever. In my example, the API on a key changes based on its value type. And the same collection can have different value types mixed together. You've recreated the worst parts of IBM IMS from the 1960s. However, the original version of IMS only changed the API when a collection's backing data structure changed. Redis ca…
Hmmm, this is a subtler issue than you make it out to be, I think, though I generally agree with you. The quality issues with Redis's technical design here interrelate substantially with user expectations/perceptions/squishier stuff.
The term "transaction" is anchored in most users' minds to a typical RDBMS transactional model assuming: a) some amount of state capture (e.g. snapshot) at the beginning of the transaction and b) "atomicity" of commit being interpreted as "all requested changes are atomically performed (or none are)" rather than "all requested changes are atomically attempted".
Redis has issues with both of those, so I'm sympathetic to your statement that what they call "transactions" is mis-characterized and would be better described as "best-effort command batching".
It's poor naming/branding to call it "transactions", and I don't think it had to be this way: MULTI/EXEC "transactions" should have been deprecated long ago--in favor of Redis scripts and other changes that should have been made in the Redis engine.
First, a defense of scripts: Redis scripts are, to a certain variety of user who wants transaction-esque functionality, not ideal. Those users may be reluctant to engage with a full procedural programming language rather than the database's query language. However, there's substantial overlap between those users and the ones who will be extremely confused by and unhappy with the existing MULTI/EXEC model--they're the folks with the most specific (wrong, in Redis) assumptions of how transactions should work, and suffer the most from them not working that way. Lua scripts, unfamiliar or not, are likely less troublesome in the long run for this cohort. Specifically, requiring users to be explicit about failure behavior of specific commands via call() vs. pcall() would remove one of the worst sharp edges of the MULTI/EXEC world.
Scripts can't answer other transaction-related needs, though. Ideally, I would have preferred that Redis go in the direction of a uniform set of conditions that can be applied to most write commands. There already are conditions in Redis, but they're not uniformly available: SET + NX/XX conditions single-key writes; WATCH semantically/implicitly conditions later EXEC commands with "if version of $key matches the version retrieved by the WATCH statement", etc. If that type of functionality were made explicit and uniformly available to all or most write operations, a further chunk of transaction-related needs could be addressed. When making single commands conditional isn't enough, scripts used to atomically batch-attempt commands could be invoked with parameters used to conditionalize those scripts' internal commands, and so on.
A final simple affordance in support of transaction-ish behavior would be a connection-scoped value type: either a modifier for arbitrary commands to have them operate on an empty database scoped to the connection, or a simple list-like value for connections to "stash" arbitrary data. This wouldn't fundamentally change any semantics, but would, at the cost of some indirection, marginally reduce the need for clients complexity when buffering conditions/commands for later flush via a pseudo-"commit"-script. This is somewhat hair-splitting, though: MULTI/EXEC is already such a connection-scoped buffer, just one that stages commands and not data. My hunch is that a data-only buffer to be consumed by scripts instead of "EXEC" would be an improvement here, but I may well be wrong.
Now, the system that results from these changes is still not as ergonomic/low friction as traditional transactions, and is especially unergonomic when users have to manually capture undo state and decide on rollback semantics during the failure of script execution. As Antirez mentioned in an adjacent comment, AOF can help ensure appropriate conistency in the face of database crashes during script execution, but database level reconciliation--aka "what is the equivalent of 'rollback' for a given script"--is still on the user to work out.
But that's what we're really talking about here, isn't it? That lack of undo (that is: the ability to capture and discard transactional state a la MVCC) is at the root of most of the weird and not-quite-transactional capabilities of Redis in this era.
Antirez is totally right that adding those capabilities would have substantially complicated the Redis engine, and I believe him when he says that made it not worth it to do so. Given that, I'd have vastly preferred a Redis which embraced providing tools that work in spite of/with full acknowledgement of that lack, rather than concealing it/confusing users by mis-branding MULTI/EXEC as "transactions".
Re: Databases in 2024: A Year in Review
#133Earlier quoted context omitted.
With all due respect, the linked video was pretty fair. It didn't imply not to use Redis, just not as a primary datastore. I don't think folks work with Redis out of fondness for the model, but because it's the least worst datastore for caching, lightweight message broker, and simple realtime things like counters.
Talking about the broken API argument here. Also Redis is particularly useful exactly in other situations compared to what OP says. Leaderboards style use cases with sorted sets are killer applications (super hard to model with SQL) of the data structure server thing. Apparently OP does not understand this and says "simple GET/SET" is what you should use Redis for. Redis has probabilistic data structures, the ability…
My naive view is that you create a sorted set every time you define an index. that is, the opposite of "super hard to model with SQL"
Re: Databases in 2024: A Year in Review
#134Re: Databases in 2024: A Year in Review
#135A couple of spicy things: > OtterTune. Dana, Bohan, and I worked on this research project and startup for almost a decade. And now it is dead. I am disappointed at how a particular company treated us at the end, so they are forever banned from recruiting CMU-DB students. They know who they are and what they did. Ouch. > Lastly, I want to give a shout-out to ByteBase for their article Database Tools in 2024: A Year in…
How do they enforce the ban? Do universities have non-compete clauses for PhD students?
Re: Databases in 2024: A Year in Review
#136Earlier quoted context omitted.
Are people choosing SQL Server independently of the Microsoft ecosystem? My understating is that you typically use it because you’re forced to choose a MS product.
SQL Server is a terrific product. And I detest most things Microsoft.
Re: Databases in 2024: A Year in Review
#137Re: Databases in 2024: A Year in Review
#138Earlier quoted context omitted.
Except when you need to scale.
this of course is false… it scales fine if you know what you are doing.
How much is out of the box or simple easy to access configuration not magic incantations either you need expensive courses to know or be battle hardened with years of experience is the question really
Re: Databases in 2024: A Year in Review
#139Earlier quoted context omitted.
SRE who deals with some .Net stuff that uses MSSQL but is converting to MySQL. so I feel somewhat qualified to talk about MSSQL. TL;DR: Nothing interesting going on. There is nothing to talk about here. It's boring database engine that powers boring business applications. It's pretty efficient and can scale vertically pretty well. With state of modern hardware, that vertical limit is high enough most people won't enc…
I'll probably come across as a shill here, but there is a lot going on with SQL Server, all included in your license (Standard Edition has limitations on scaling). Some of these things are merely passable, some are great, but it's all included. The key takeaway is that SQL Server is a full data platform, not just an RDBMS. - RDBMS: very solid, competitive in features - In-memory OLTP: (really a marketing name for a w…
Licensing isn't cheap. For anyone wondering, before discount, it's 876/yr per core for Standard and 3288/yr per core for Enterprise. Also note that Standard is limited to 24 cores and 128GB of RAM, if you want to unlock more of that, you must move to Enterprise.
Re: Databases in 2024: A Year in Review
#140Wow his database startup that raised 12M died this year after only three years. If anything this shows how insanely difficult it must be to succeed as a database startup (when was the most recent startup success in this space?), as the founding team is stellar. On the other hand I am surprised it died this quick and interested to know if they did a proper postmortem. Not only did they raise way more than is needed to…