Viewing profile — varunshenoy

I've been playing with AI agents for months, and most of them are pretty bad. They often get stuck in loops, which is frustrating. This happens in MultiOn, AutoGPT, and others. I'v…

Hey guys! Just wanted to share a fun side project. Code is here: https://github.com/varunshenoy/latentverse

Slightly different set of trade-offs, but similar mental model. You always use large batch sizes (compute bound) and the bottleneck usually ends up communication between GPUs/nodes…

Good question. Yes, the 10GB available for batching is in the HBM. In a single forward pass, you move the entire model from HBM -> SRAM exactly once. In a batched forward pass, thi…

Thanks! vLLM for quick set up, TRT-LLM for best performance. Both available on https://baseten.co/ .

Absolutely. Looks like the M1 Ultra has 800GB/s of memory bandwidth and ~20 TFLOPS of compute. The same calculations from the post should hold, except with these new values.

:)

Happy Halloween HN! Excited to share Misgif, a new AI powered app I've been working on over the last few weeks. Put yourself in your favorites movies, TV shows, and more. Under the…

Awesome job guys, and thank you for creating it. Curious if you guys have any insights on long-term memory and if there are better ways to do retreivel apart from top-k. Seems weir…

You can write an extension to support LoRA (~10 lines of Python HF Diffusers code). If you get to this before me, please create a PR!

LoRAs can be handled as a straight-forward Python extension!

If anyone wants to add SDXL support, all you have to do is create a new extension with the correct SDXL logic (loading from HF diffusers, etc.). You could parameterize `num_inferen…

done :)