[flagged]
Tinygrad
91–100 of 159 posts
Re: Tinygrad
#92I'd like to ship a video game that does machine learning and trains on its experiences with the player. Yes, I know there are many potential problems with this. What is the best way to ship training code in a game? Do I embed Python and PyTorch or something? Do I code my own NN training algorithm? Do I use a library such as Tinygrad?
Typically, these libraries are unusable for this purpose because you don't want to ship a Python interpreter with your video game.
I usually prefer to to rewrite my training step as a pure function, so the model weights are just inputs and the gradient updates are outputs.
You need to serialize your computation graph in some way, so it can be run in C++ or some other low-level language. TensorFlow is known for doing this well since it was original design goal of the project. Some of the other frameworks that originally targeted researchers make this harder. Most mature frameworks have some way of doing this now, though, and projects like https://onnx.ai/ may solve this in general.
It gets more complicated if your model has dynamic control flow, but you get the idea.
Re: Tinygrad
#93Earlier quoted context omitted.
No offense taken and no offense intended at all friend. In my opinion we are down the rabbit hole of arguing style over substance. I just think you are grasping at straws. Tools like black (if there was a just god it would just be part of the language, like gofmt) can easily settle stylistic choices. Really not worth endlessly debating, what next, tabs vs spaces? This commit seems fine to me as well. Consider the gol…
I think I've proven my point well enough here by making it clear that in order for them to stay under 1kLOC they have had to make golfing-style changes that cannot be automated. If I were hell-bent on convincing you, I could fork their repo and add literally any feature requiring more than ten LOC and win. They would have to bend over backwards to get enough lines back to be able to merge whatever it is I wrote, with…
That's just like, your opinion, man.
And it isn't a rhetorical strategy. It is socratic questioning.
Should you sit down and attempt to write something out of spite (an excellent motivator, as good as any) it would go slightly differently than how you theorize.
In order to add a feature requiring more than ten LOC you'd actually have to come up with one in the first place :)
That would require reading Tinygrad, which shouldn't take very long for obvious reasons. Probably we've been arguing about this for longer.
At which point you'd realize there is not much to add to it and you are splitting hairs over whether it is 1200 lines or 999 lines.
In which case you could just run black on it and leave it at whatever line count it spits out. That wouldn't even require any additional coding on your part!
Then you can compare the black'd version to theirs and realize they made a very very small sacrifice as a lark. Thereby finally understanding the difference between the principal of "code golf bad" and the reality of "tinygrad l33t demo" you ol' fuddy duddy.
Re: Tinygrad
#94I watched a few of George's live streams, and I'm pretty impressed with his coding skills and determination to solve a problem.
In the spirit of learning, anyone else on his level do live streams or has a youtube channel?
Here's his last 7 hour stream coding Tinygrad.
Re: Tinygrad
#95Even more tiny: https://github.com/breandan/picograd/blob/main/picograd.py
Look man, this is just too tiny. People are upset in this thread about the excessively short line count of OP as it is. What is this, some sort of autograd for ants?
It is freaking me out, remove this link.
Re: Tinygrad
#96Earlier quoted context omitted.
> not a normal guy I assure you some people try themselves - and I do not see what is not "normal" about it. To experience, voluntary, then grow, is the norm.
I don't know. I ran the idea of purposely getting addicted to opioids to see if i can quit by my wife and she assured me i was crazy.
I'm in a similar situation, but not with opioids.
I don't think that this is a crazy idea at all. We're just testing our limits and seeing if we as smart as we say.
Re: Tinygrad
#97Earlier quoted context omitted.
I'm getting distinct uses Eclipse to write vogon poetry and Jira Master vibes from you. Any notes on the actual code in the OP beyond the import lines formatting?
I'm sorry to have offended you, but I don't feel personal insults over the README of an open source project are particularly reasonable. This affects the actual code, too! See, for example, https://github.com/geohot/tinygrad/commit/cfb7a4c41a2b6bcc09... , which includes this gem: diff --git a/tinygrad/ops/ops_cpu.py b/tinygrad/ops/ops_cpu.py index a454f56f..0686f810 100644 --- a/tinygrad/ops/ops_cpu.py +++ b/tinygrad…
It could be written:
def log(x): return np.log(x)
def exp(x): return np.exp(x)
...
Re: Tinygrad
#98Earlier quoted context omitted.
Having gone to college and knowing this guy personally from then, you would have a much different opinion... Maybe he has grown up, but back then... yeeesh
Jealousy tends to make you react that way
Envy. Some changes in language are fine, but these are very different things and it's nice to have words for both of them.
Re: Tinygrad
#99I find this related page more interesting: A Breakdown of AI Chip Companies https://geohot.github.io/blog/jekyll/update/2021/06/13/a-bre... I especially like that he outlines an actual plan for an AI chip startup that he thinks will work, and has an update explaining why he was subsequently convinced that it wouldn't work.
I read this article vs. the actually posted one, it has a lot of good points but also get a fair amount wrong, and the correction on the power usage is just the tip of the iceberg. That's why WaferScale do what they do, the power cost of off die vs. on die is massive, and enabling everything to be on chip means you can feed the design more easily and with less power.
For example, Nvidia's compute and consumer GPU line diverged a long time ago. Modern A100s have literally only one SM capable of doing normal GPU tasks, probably to support running a display on whatever Quadro version they end up increasing. They diverged in really specific ways, for example the P100 has hardware scheduling, where as the 1080 does not (in the same way at least).
Another issue is the author spends a long time talking about how important software and ecosystem is, then completely misses that point when talking about their own CHIP - just because it is RISCV and compilers exist for that arch does not equal CUDA. Also, big re-order buffers cost area and heat that could be spent on more SMs. That's why in order to beat Nvidia you must get more specialized, they've picked their niche on the CPU-GPU-ASIC continuum, beating them at the same process node requires ditching some stuff of the stuff an Nvidia GPU. Which is why they've been specializing their arch with tensor cores.
It just also turned out those are useful for gaming with deep learning to upres the graphics, as that's easy to accelerate than driving quadraticlly higher resolutions.
Re: Tinygrad
#100I'd like to ship a video game that does machine learning and trains on its experiences with the player. Yes, I know there are many potential problems with this. What is the best way to ship training code in a game? Do I embed Python and PyTorch or something? Do I code my own NN training algorithm? Do I use a library such as Tinygrad?
You probably want a lightweight reinforcement learning library in C/C++ e.g. something from here https://github.com/search?l=C%2B%2B&q=reinforcement+learning...
You could also compile a neural net into a less python-tied format e.g. ONNX or torchscript. In general a siloed pytorch env would be massive, I'm assuming at least a gig or two.