Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Continuous Nvidia CUDA PC Sampling Profiler

polarsignals.com

Post reply on HN
Print view View original

1–7 of 7 posts

Re: Show HN: Continuous Nvidia CUDA PC Sampling Profiler

#2

Post by killamdiaz »

Very cool project.

Curious whether the biggest value has been performance debugging itself or helping developers understand system behavior they otherwise wouldn't have visibility into.

Sometimes the observability layer ends up being more valuable than the optimization layer.

Re: Show HN: Continuous Nvidia CUDA PC Sampling Profiler

#3

Post by gnurizen »

killamdiaz wrote: post #2

Very cool project. Curious whether the biggest value has been performance debugging itself or helping developers understand system behavior they otherwise wouldn't have visibility into. Sometimes the observability layer ends up being more valuable than the optimization layer.

Thanks! I think most performance debugging happens during development, what we're bringing to the table is exposure of system behavior in production which often diverges because of changes in the shape of workloads from dev, which are often simplistic and synthetic. So I'd say its late-stage performance debugging and production observability combined that makes this useful.

Stay tuned for a follow on post where we show how we used this to optimize an FSST decompression kernel for vortex (https://github.com/vortex-data/vortex).

Re: Show HN: Continuous Nvidia CUDA PC Sampling Profiler

#4

Post by saagarjha »

Honest question, I feel like kernels are usually short enough that you can fully understand their performance in the development cycle before you even deploy them. If you get different results in production this seems to me that you didn’t spend enough time understanding what’s going on earlier. Are there things you genuinely can’t get from this workflow?

Re: Show HN: Continuous Nvidia CUDA PC Sampling Profiler

#5

Post by SyzygyRhythm »

saagarjha wrote: post #4

Honest question, I feel like kernels are usually short enough that you can fully understand their performance in the development cycle before you even deploy them. If you get different results in production this seems to me that you didn’t spend enough time understanding what’s going on earlier. Are there things you genuinely can’t get from this workflow?

Sometimes you have to optimize other people's code. Also, sometimes code behaves unexpectedly depending on the data, say over a certain size threshold. And sometimes it behaves differently on different hardware. You don't always find these things out until production.

Re: Show HN: Continuous Nvidia CUDA PC Sampling Profiler

#6

Post by saagarjha »

SyzygyRhythm wrote: post #5
saagarjha wrote: post #4

Honest question, I feel like kernels are usually short enough that you can fully understand their performance in the development cycle before you even deploy them. If you get different results in production this seems to me that you didn’t spend enough time understanding what’s going on earlier. Are there things you genuinely can’t get from this workflow?

Sometimes you have to optimize other people's code. Also, sometimes code behaves unexpectedly depending on the data, say over a certain size threshold. And sometimes it behaves differently on different hardware. You don't always find these things out until production.

I don't really understand how PC sampling is going to help you with this

Re: Show HN: Continuous Nvidia CUDA PC Sampling Profiler

#7

Post by gnurizen »

saagarjha wrote: post #6
SyzygyRhythm wrote: post #5

Earlier quoted context omitted.

Sometimes you have to optimize other people's code. Also, sometimes code behaves unexpectedly depending on the data, say over a certain size threshold. And sometimes it behaves differently on different hardware. You don't always find these things out until production.

I don't really understand how PC sampling is going to help you with this

A lot of times the performance is data dependent, like randomized data can result in poor warp utilization where as sorted data can be much more efficient because you have less warps sitting around idle. We have an upcoming blog post on decompression kernels that show this clearly, stay tuned!
Post reply on HN