11–20 of 29 posts
Sounds pretty much what Postgres is doing since v11
Actually this is quite a bit different. This is more a columnar-store version of SQLite, basically an embedded OLAP database, which is pretty cool. I’m not aware of other column-stores in this niche, most are distributed systems meant for big data and so are much more complicated to setup and manage.
Sounds pretty much what Postgres is doing since v11
Sounds pretty much what Postgres is doing since v11
Actually this is quite a bit different. This is more a columnar-store version of SQLite, basically an embedded OLAP database, which is pretty cool. I’m not aware of other column-stores in this niche, most are distributed systems meant for big data and so are much more complicated to setup and manage.
It would be amazing if the world of open-source column stores matured a little bit relative to where we are today...
I understand SQL, and I kind of understand LLVM, but I don't understand why SQL on LLVM?
Most SQL engines implement the query engine as a virtual machine that traverses the result on a row by row basis, under the presumption that I/O is the bottleneck. In other words, this is very slow on modern hardware.
Earlier quoted context omitted.
Most SQL engines implement the query engine as a virtual machine that traverses the result on a row by row basis, under the presumption that I/O is the bottleneck. In other words, this is very slow on modern hardware.
Ok you have to elaborate a bit further here. Does this mean they are actually leveraging llvm to, at runtime, compile queries?
Sounds pretty much what Postgres is doing since v11
Actually this is quite a bit different. This is more a columnar-store version of SQLite, basically an embedded OLAP database, which is pretty cool. I’m not aware of other column-stores in this niche, most are distributed systems meant for big data and so are much more complicated to setup and manage.
I understand SQL, and I kind of understand LLVM, but I don't understand why SQL on LLVM?
Sounds pretty much what Postgres is doing since v11
since llvm is so slow you really have to validate if jit does work for your queries (obviously like for anything... duh) in my case i managed to slow the DB to a crawl with queries that were estimated to be super expensive but 95% of the plan wasn't actually ever executed.
Look forward to hearing more about it in the future.
Sounds pretty much what Postgres is doing since v11
since llvm is so slow you really have to validate if jit does work for your queries (obviously like for anything... duh) in my case i managed to slow the DB to a crawl with queries that were estimated to be super expensive but 95% of the plan wasn't actually ever executed.
1) I'd hoped to get caching for JITed queries into 13 (or at least the major prerequisite), but that looks like it might miss the mark (job changes are disruptive, even if they end up allowing for more development time). The nicest bit is that that the necessary changes also result in significantly better generated code.
2) Background JIT compilation. Right now the JIT compilation happens in the foreground. We really ought to only do the IR generation in foreground, and then do the compilation in the background, while continuing with interpreted execution. Only once codegen is done, we'd redirect to the JITed program (there'd be a bit higher overhead during the interpreted phase, rechecking whether to now redirect, but not that large).
3) Improve costing logic. E.g. we don't take the size of the necessary generated code into account at the moment, and we should. The worker count isn't taken into account either.
4) Improve optimization pipeline. There's plenty cases where we don't run beneficial and cheap-ish optimization passes, and there's plenty cases where we run unlikely to be helpful and really expensive optimization passes.
Edit: Added 4).