Viewing profile — t-vi

Not sure about the link, the official landscape URL seems to be this (same data probably): https://landscape.pytorch.org/

We do try to check the pulse of projects there with a checklist, but I guess it started in a more uncoordinated way. If you file an issue here, I think it would work to update thin…

They have some pretty cool people, though, no reason not to think they'll catch up soon enough.

- I don't think it hurts to learn PyTorch (and having learned JAX is good, too). I don't know if JAX + triton is as impossible as you make it out, but it seems that PyTorch integra…

Note that the NVIDIA container uses CUDA+cuBLAS 13.0.2 which cites "Improved performance on NVIDIA DGX Spark for FP16/BF16 and FP8 GEMMs", which seems to be your use-case. In gener…

It seems to me that in 2016 people did (have to) play a lot more tricks with the backpropagation than today. Back then it was common to meddle with gradients in between the gradien…

Note that PyTorch's kernels are somewhat generic in shape. It has always been relatively easy to get speedups by specializing the shape, e.g. Apache TVM had that (back before it wa…

If you like JIT wrappers and Python interpreters: In Thunder[1], a PyTorch to Python JIT compiler for optimizing DL models, we are maintaining a bytecode interpreter covering 3.10-…

When I did a similar thing (but with less LLM) I liked https://github.com/coqui-ai/TTS but back then I needed to cut out the conversion step from tensor to a list of numbers to mak…

The subtraction is because "is an example of constructing the “Inner Product” distance" per the text above it. That ymmone might not be one could be because they only need that up …

> Is avoiding CF potentially just a matter of sheer scale ? My intuition would be that you get more orthogonal directions to the gradient (of previous samples) if you have larger m…

After the first epoch, the average time since the present data item was last used for during training is small at the beginning of an epoch grows during the epoch. I'd expect that …

My former neighbors run https://justanotherfoundry.com/ and I like their work and bought some.

To be honest, I never get what people want with all that business and wonder if it is because the abstraction ("ordered derivatives") implied is not ideal. If we follow the ordinar…

The PEP says that a naïve implementation would cause a 4% performance hit and have a list of various ways of speeding up cases where immortality makes sense / is guaranteed that th…

This special casing just had not been implemented yet. But as it is an interesting optimization, more so with multi-interpreter or no-GIL Python, the developers will actually intro…

Not saying that the vector space bit isn't neat, but it's called functional analysis because you can take limits of various forms and define (semi-) continuity, have completions of…

It knows nothing of the modern stuff (because MacKay died too early), but skipping the first parts of David MacKay: Information Theory, Inference, and Learning Algorithms you get a…

The logic error seems to be that you have the second for loop where you only want to destructure the splitted line into a tuple. Now writing that decomposition as a for loop over a…

I use Debian's system Python 3.10 install for most of my stuff and it works really well for me. Some things I install via pip but the key libraries (e.g. PyTorch) from source.

There is A. Karpathy's recipe for training NNs but it is not a walkthrough with an example: https://karpathy.github.io/2019/04/25/recipe/ but the general idea of "get something tha…