q is the programming language underpinning kdb+ by Kx Systems - often claimed (by Kx) to be "the fastest database in the world". Possibly also the most expensive?
L is an independent implementation of a q interpreter. To those who reluctantly commercially license kdb+ - this will be a welcome alternative.
l: A new runtime for k and q
31–40 of 118 posts
Re: l: A new runtime for k and q
#32Not sure why all the hate (sure site may be vibecoded, not all of us are front-end masters and it's at least not spartan)... I've always found k to be fascinating, and cool idea to try and roll a new one. Wish it were wholly open source, but cool to have a new variant that seems to bench well.
Sure the site, the articles, the program. But sure lets just trust that the author who can't write knows how to benchmark software
Not a web dev - but have some experience in benchmarking these types of workloads e.g. https://www.mcobject.com/press/november19-2014/
L used the two open ones that are easy to replicate: H2O.ai (great bench) https://github.com/l-labs/db-benchmark TSBS (less great but useful) https://github.com/l-labs/tsbs
If there are others (will do ClickBench) they'll go there as well
Re: l: A new runtime for k and q
#33q is the programming language underpinning kdb+ by Kx Systems - often claimed (by Kx) to be "the fastest database in the world". Possibly also the most expensive? L is an independent implementation of a q interpreter. To those who reluctantly commercially license kdb+ - this will be a welcome alternative.
Absolutely and given Kx is now private equity owned and are in the customer-squeezing phase of the acquisition, this is very welcome indeed.
Re: l: A new runtime for k and q
#34l must be named 1 because I is taken
next letter after k
Re: l: A new runtime for k and q
#35q is the programming language underpinning kdb+ by Kx Systems - often claimed (by Kx) to be "the fastest database in the world". Possibly also the most expensive? L is an independent implementation of a q interpreter. To those who reluctantly commercially license kdb+ - this will be a welcome alternative.
Absolutely and given Kx is now private equity owned and are in the customer-squeezing phase of the acquisition, this is very welcome indeed.
and fusion (e.g. f g h x has no intermediates e.g. mutates in place) is new
as is compute on compressed vectors (very helpful performance unlock) https://lv1.sh/blog/compute-on-compressed/
Re: l: A new runtime for k and q
#36As someone who does not know what k4, qSQL, or q are, reading through the landing page of this website was giving me mild schizophrenia. And then I tried to search for these things in the old way, and received incredibly dry technical sites that still don’t tell me what it is, and all these names are wildly SEO unfriendly. So I had Claude give me context and it’s apparently the database Wall Street uses for tick data…
I believe k and q are member of the "array programming"/APL family of languages who are exceptionally terse/information-dense https://en.wikipedia.org/wiki/Array_programming EDIT: Did someone downvoted this because it is wrong or is someone I am arguing with rage-downvoted this ?
[deleted]
Re: l: A new runtime for k and q
#37As someone who does not know what k4, qSQL, or q are, reading through the landing page of this website was giving me mild schizophrenia. And then I tried to search for these things in the old way, and received incredibly dry technical sites that still don’t tell me what it is, and all these names are wildly SEO unfriendly. So I had Claude give me context and it’s apparently the database Wall Street uses for tick data…
K4 is the K programming language, Q is a language built on top of K. They are the practical over achieving members of the array family of programming languages.
shoutout to bryan @bcantrill who I think explained K best at https://youtu.be/2wZ1pCpJUIM?si=y4ugbFXroTZc22AY&t=471 and https://queue.acm.org/detail.cfm?id=1531242
Re: l: A new runtime for k and q
#38Earlier quoted context omitted.
Absolutely and given Kx is now private equity owned and are in the customer-squeezing phase of the acquisition, this is very welcome indeed.
and fusion (e.g. f g h x has no intermediates e.g. mutates in place) is new as is compute on compressed vectors (very helpful performance unlock) https://lv1.sh/blog/compute-on-compressed/
yeah, that is super neat! Very innovative to bring it to q.
Re: l: A new runtime for k and q
#39yeah the site's clearly vibecoded and isn't opensource, but i also think this is a genuinely interesting design space and more people should be building in it. APL ( https://www.dyalog.com/ ), BQN ( https://mlochbaum.github.io/BQN/ ), J/Jd ( https://code.jsoftware.com/wiki/Jd/Overview ), Klong ( https://news.ycombinator.com/item?id=10586872 ), Kerf ( https://news.ycombinator.com/item?id=9782520 ), RayforceDB ( https:…
I've recently tested BQN on ClickBench (a benchmark for OLAP databases), and the results are not great: https://benchmark.clickhouse.com/#system=+N|liH&type=-&machi... If anyone is curious how queries in this language look, you can see it here: https://github.com/ClickHouse/ClickBench/pull/939/changes#di...
FYI, your "8‿64 •bit._cast -⟜@ raw" in bqn/util.bqn results in passing to •bit._cast integers in the range 0..255, whereas it expects integers ¯128..127 (and does arbitrary platform-specific things on integers outside that for the int16→int8 narrow); removing the "-⟜@" makes it work properly, and 2.5-5x faster while at it from not going to intermediate int16 (allowing the bitcast to be O(1) by itself). (•FBytes → •file.MapBytes also probably improves things, at the cost of the obvious mmap problem of issues if the file changes, as BQNs arrays are immutable). This is also what caused the problem at https://github.com/ClickHouse/ClickBench/pull/939/changes/40... (there's been some discussion about what to do about invalid •bit._cast before, considering it's rather perf-sensitive; it's the only place in CBQN with such quiet very-wrong results on invalid inputs)
The default CBQN "make o3" on x86-64 also results in it only using SSE2 (utilizing function multiversioning is on the ever-infinite TODO list, though somewhat-low on it considering it's strictly-unnecessary in any specific situation; there's also AVX-512 usage on a branch, but mostly only AVX2 on mainline; and no arm SVE)
That all said, CBQN doesn't currently do any loop fusion, so being significantly-slower for sequences of operations over larger-than-cache arrays would kinda just be expected. BQN also just isn't particularly intended for database work anyway.
(didn't look much at the specific query impls, though "Pair" in utils.bqn is at least an overlong version of "Pair ← ⋈¨"; and some if not all of those Pairs would be better as "≍˘" to avoid nested arrays and ensuing pointer chasing; and, of course, if some of the columns are bools/int8/int16/int32, it'd be beneficial to store & load them as such instead of float64)
Re: l: A new runtime for k and q
#40Quite cool, but for a new runtime of an existing language it might make sense to compare to, y'know, the other[0] runtimes of that language? Even if one has to omit the best, closed ones for lack of access / permission to benchmark? [0]: https://k.miraheze.org/wiki/Running_K
https://github.com/l-labs/master-benchmark
ah. cheeky.