Viewing profile — roanakb

Hey HN! We're Roanak and Andrew from Trainy (YC S23). We build GPU infrastructure for ML teams (scheduling multi-node training jobs on Kubernetes). When Neptune announced they're s…

Unfortunately, SM efficiency is not accessible via nvidia-smi. The best methods to track it would be to: 1. Profile your model with Pytorch Profiler 2. Export metrics with Nvidia D…

oh this looks great, thank you for bringing this up! I'll have to give it a try, but seems like the FSDP limitation on torch.compile might carry over?

Yup, you'll see 100% utilization on a kernel over a time period if it's considered active, which includes just having a single thread executing [1]. SM occupancy is great but can b…

Nice, seems like ML Productivity Goodput is a pretty well thought-out metric to understand the overall efficiency of your cluster. I'll consider adding this into our cluster manage…

Agreed, roofline plots would be quite powerful in this context. From a quick search, seems like the only way to create a roofline plot for your model would be to use Nsight [1]? Wo…

Yup, similar to SM efficiency in that sense too. If you aren't seeing >80%, there is certainly time left on the table. But getting a high SM efficiency value doesn't guarantee you'…

this is a good one for debugging rdma: https://docs.redhat.com/en/documentation/red_hat_enterprise_...

Looks great, you guys made it really easy to integrate!

docs: https://konduktor.readthedocs.io/en/latest/admin/controller.... github: https://github.com/Trainy-ai/konduktor

Thanks! Let me know if there are any features you'd like to see added.

Looks really cool! Nice work.

Location: Fremont, CA Remote: Yes Willing to relocate: No Technologies: Python, R, Pytorch, Keras, SciPy, Machine Learning, Deep Learning, Computer Vision, NLP, Javascript, MERN St…