Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

PG-Strom: SSD-To-GPU Direct SQL Execution

github.com

Post reply on HN
Print view View original

21–30 of 38 posts

Re: PG-Strom: SSD-To-GPU Direct SQL Execution

#21

Post by arnon »

throwaway91111 wrote: post #14
raverbashing wrote: post #5

Wondering how much the "moving data to the GPU" overhead won't help to make things significantly faster

Really depends on the shape of the data and query at hand. If you're doing aggregate computations over million/billions of scalar-sized rows, it could be quite fast. However joining two cold tables will still be more limited by schema and disk read than anything.

Unless you're doing skipping and indexing properly....

Re: PG-Strom: SSD-To-GPU Direct SQL Execution

#22

Post by ccrush »

sixdimensional wrote: post #15

I still am hoping for affordable SSD to FPGA direct execution someday!

Direct execution? With GPUDirect from NVIDIA you could have an FPGA get a memcpy() from graphics RAM to its data has. Still, not sure how you plan on using this. I would hope that NVIDIA could give us programmable fabric on the GPU silicon, and it could somehow access each compute units instruction execution pipeline. Would be interesting indeed. Or just massive, fast FPGAs with included highly parallel compute IP built in. Or all new NVIDIA cards just ship a very fast FPGA that runs 3D graphics by default, but can be reprogrammed to do CUDA compatible computation and its open source so users can add new instructions or modify existing ones or even reconfigure the number or arrangement of cores.

Re: PG-Strom: SSD-To-GPU Direct SQL Execution

#23

Post by nikanj »

ricw wrote: post #7

I love the concept.p But it's hard to tell whether this is ready for production. Anyone have any hands on experience with it?!

This is the first time the concept has ever been shown publicly. You must be exceptionally brave to use it in production. Most CTOs are cynical, and always wait for the .1 version

Re: PG-Strom: SSD-To-GPU Direct SQL Execution

#25

Post by carbocation »

microcolonel wrote: post #11
redm wrote: post #4

It's great to see this type of evolution in database performance, maybe this will become the standard someday. The only thing missing is that there isn't a comparable project for MySQL.

Not trying to be a snoot here, but what is the point of using MySQL today? I can't think of something that I'm aware it does better than Postgres.

Postgres can have some write-amplification issues for update-heavy loads where at least one of the updated fields is covered by an index. Famously, this triggered Uber to switch to MySQL. Its impact is noticeable in one of my projects as well -- and it isn't that large (a few dozen million rows).

Re: PG-Strom: SSD-To-GPU Direct SQL Execution

#26

Post by anarazel »

carbocation wrote: post #25
microcolonel wrote: post #11

Earlier quoted context omitted.

Not trying to be a snoot here, but what is the point of using MySQL today? I can't think of something that I'm aware it does better than Postgres.

Postgres can have some write-amplification issues for update-heavy loads where at least one of the updated fields is covered by an index. Famously, this triggered Uber to switch to MySQL. Its impact is noticeable in one of my projects as well -- and it isn't that large (a few dozen million rows).

Not trying to doubt your experience, just collecting datapoints for improvements. I'd appreciate if you could describe what your problems around write amplification are. Is it

- heap bloat

- index bloat

- CPU overhead of index manipulations

- Write-ahead-log overhead of updates

- increased overhead / frequency of VACUUM

- total amount of disk writes

Thanks!

Edit: formatting.

Re: PG-Strom: SSD-To-GPU Direct SQL Execution

#28

Post by jeltz »

nikanj wrote: post #23
ricw wrote: post #7

I love the concept.p But it's hard to tell whether this is ready for production. Anyone have any hands on experience with it?!

This is the first time the concept has ever been shown publicly. You must be exceptionally brave to use it in production. Most CTOs are cynical, and always wait for the .1 version

Hm? I have not followed PGStorm but the project has been around for at least 5 years, probably more.

Re: PG-Strom: SSD-To-GPU Direct SQL Execution

#29

Post by carbocation »

anarazel wrote: post #26
carbocation wrote: post #25

Earlier quoted context omitted.

Postgres can have some write-amplification issues for update-heavy loads where at least one of the updated fields is covered by an index. Famously, this triggered Uber to switch to MySQL. Its impact is noticeable in one of my projects as well -- and it isn't that large (a few dozen million rows).

Not trying to doubt your experience, just collecting datapoints for improvements. I'd appreciate if you could describe what your problems around write amplification are. Is it - heap bloat - index bloat - CPU overhead of index manipulations - Write-ahead-log overhead of updates - increased overhead / frequency of VACUUM - total amount of disk writes Thanks! Edit: formatting.

This is a great question. I have to say I haven't profiled Postgres to that level of granularity, so I am making an assumption about why my updates are slow. What I notice is relatively slow UPDATE performance, with high total disk writes and a write-ahead-log that seems to be large. The bloat on disk isn't an issue for me (or, rather, I have started using lower fillfactors to reduce the variability in DB size as the rows are updated).

Do you have a favored resource that you can direct me to that would help me identify the factors that you cited and ultimately allow me to make more nuanced criticisms (or to find a fixable error in my approach, which would of course be the ideal result)?

Post reply on HN