21–30 of 116 posts
What sort of things are people doing in their SQL queries that make them CPU bound? Admittedly I'm a meat-and-potatoes guy, but I like mine I/O bound. Really amazed to see not one but several generic JIT frameworks though, no idea that was a thing.
What sort of things are people doing in their SQL queries that make them CPU bound? Admittedly I'm a meat-and-potatoes guy, but I like mine I/O bound. Really amazed to see not one but several generic JIT frameworks though, no idea that was a thing.
Earlier quoted context omitted.
Two things are holding back current LLM-style AI of being of value here: * Latency. LLM responses are measured in order of 1000s of milliseconds, where this project targets 10s of milliseconds, that's off by almost two orders of magnitute. * Determinism. LLMs are inherently non-deterministic. Even with temperature=0, slight variations of the input lead to major changes in output. You really don't want your DB to be n…
> LLMs are inherently non-deterministic. This isn't true, and certainly not inherently so. Changes to input leading to changes in output does not violate determinism.
From what I understand, in practice it often is true[1]:
Matrix multiplication should be “independent” along every element in the batch — neither the other elements in the batch nor how large the batch is should affect the computation results of a specific element in the batch. However, as we can observe empirically, this isn’t true.
In other words, the primary reason nearly all LLM inference endpoints are nondeterministic is that the load (and thus batch-size) nondeterministically varies! This nondeterminism is not unique to GPUs — LLM inference endpoints served from CPUs or TPUs will also have this source of nondeterminism.
[1]: https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
We have everything optimized, and yet somehow DB queries need to be "interpreted" at runtime. There's no reason for DB queries to not be precompiled.
Postgresql uses a process per connection model and it has no way to serialize a query plan to some form that can be shared between processes, so the time it takes to make the plan including JIT is very important. Most other DB's cache query plans including jitted code so they are basically precompiled from one request to the next with the same statement.
https://www.postgresql.org/docs/current/parallel-query.html
"PostgreSQL can devise query plans that can leverage multiple CPUs in order to answer queries faster."
Earlier quoted context omitted.
Two things are holding back current LLM-style AI of being of value here: * Latency. LLM responses are measured in order of 1000s of milliseconds, where this project targets 10s of milliseconds, that's off by almost two orders of magnitute. * Determinism. LLMs are inherently non-deterministic. Even with temperature=0, slight variations of the input lead to major changes in output. You really don't want your DB to be n…
> LLMs are inherently non-deterministic. This isn't true, and certainly not inherently so. Changes to input leading to changes in output does not violate determinism.
"But why aren’t LLM inference engines deterministic? One common hypothesis is that some combination of floating-point non-associativity and concurrent execution leads to nondeterminism based on which concurrent core finishes first."
From https://thinkingmachines.ai/blog/defeating-nondeterminism-in...
Earlier quoted context omitted.
Postgresql uses a process per connection model and it has no way to serialize a query plan to some form that can be shared between processes, so the time it takes to make the plan including JIT is very important. Most other DB's cache query plans including jitted code so they are basically precompiled from one request to the next with the same statement.
> and it has no way to serialize a query plan to some form that can be shared between processes https://www.postgresql.org/docs/current/parallel-query.html "PostgreSQL can devise query plans that can leverage multiple CPUs in order to answer queries faster."
If process based then they can send small parts of plan across processes.
What sort of things are people doing in their SQL queries that make them CPU bound? Admittedly I'm a meat-and-potatoes guy, but I like mine I/O bound. Really amazed to see not one but several generic JIT frameworks though, no idea that was a thing.
Earlier quoted context omitted.
Postgresql uses a process per connection model and it has no way to serialize a query plan to some form that can be shared between processes, so the time it takes to make the plan including JIT is very important. Most other DB's cache query plans including jitted code so they are basically precompiled from one request to the next with the same statement.
What do you mean ? Cause the obvious thing is a shared cache and if there is one thing the writers of a db know it is locking
Earlier quoted context omitted.
> and it has no way to serialize a query plan to some form that can be shared between processes https://www.postgresql.org/docs/current/parallel-query.html "PostgreSQL can devise query plans that can leverage multiple CPUs in order to answer queries faster."
Nothing to do with plan caching, thats just talking about plan execution of parallel operations which is that thread or process based in PG? If process based then they can send small parts of plan across processes.
Plans for prepared statements are cached though.
Earlier quoted context omitted.
What do you mean ? Cause the obvious thing is a shared cache and if there is one thing the writers of a db know it is locking
Sharing executable code between processes it not as easy as sharing data. AFAIK unless somethings changed recently PG shares nothing about plans between process and can't even share a cached plan between session/connections.
I was actually confused by this submission as it puts so much of an emphasis on initial compilation time, when every DB (apparently except for pgsql) caches that result and shares it/reuses it until invalidation. Invalidation can occur for a wide variety of reasons (data composition changing, age, etc), but still the idea of redoing it on every query, where most DBs see the same queries endlessly, is insane.