181–189 of 189 posts
Earlier quoted context omitted.
Speed of hardware did. Back in 80a they already knew the principles of llm training. It only took one week to train 10.000 tokens.
Got a reference on that claim?
Earlier quoted context omitted.
Sure neural networks in of themselves are conceptually simple, and not difficult to code. Andrew Ng's original Coursera class is all you need to go from zero knowledge to building MATLAB based neural nets in this same hard coded style. However, there is a huge difference in functionality (hence complexity) in a framework such as PyTorch vs hardcoding a single NN. It's a bit like the difference between writing a toy c…
I know all this things. Again: look at MLX.
I'm still not sure MLX vs PyTorch is really a fair comparison though since PyTorch is a much more mature framework and of course supports many different backends, as opposed to MLX which is just Metal or CPU.
Comparing these is a bit like comparing the new programming language of the day vs older ones that have accumulated a log of cruft/complexity. The shiny new language will likely look just as crufty after it has accumulated all the additional functionality of the older one.
If I was starting from scratch, what resources should I start with to build up an understanding of what this code does and how to read it? It's quite dense and my knowledge of LLMs is quite minimal. Are these terse variable names standard in LLM-land?
https://karpathy.ai/zero-to-hero.html
Do those, then come back to this and it will make more sense.
https://www.thebugger.us/exploring-karpathys-llm-c-a-lightwe...
Candle is a minimalist ML framework for Rust with a focus on performance (including GPU support) and ease of use https://github.com/huggingface/candle
Vision Model https://huggingface.co/spaces/radames/Candle-Moondream-2 Blip Image Captioning https://huggingface.co/spaces/radames/Candle-BLIP-Image-Capt... Microsoft Phi 2 https://huggingface.co/spaces/radames/Candle-phi1-phi2-wasm-...
Is this able to replace PyTorch, ... in normal practice? No. Does this show that in general the most used ML frameworks are a mess? Yes.
Furthermore pytorch made learning very approachable. It would have been impossible for me to learn c and deep learning.