Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Five Years of Tinygrad

geohot.github.io

Post reply on HN
Print view View original

81–90 of 167 posts

Re: Five Years of Tinygrad

#81

Post by xiphias2 »

rent0pat wrote: post #56

I've looked at the "only 18,935 lines of code" python code and it made me want to poke my eyes out. Not sure what's the point of this extreme code-golfing.

I think it's an amazing experiment.

You can look at the PyTorch code base and understand a small local function instantly, but if I would have a task to have a deep understanding of either PyTorch with all kernel code and low level code, whole CUDA code base + LLVM compilation code or Tinygrad I would pick Tinygrad in an instant.

The code looks hard because what it is doing is hard, but all its layers can be debugged.

Re: Five Years of Tinygrad

#82

Post by zephen »

tucnak wrote: post #50
zephen wrote: post #46

Earlier quoted context omitted.

I dunno. This sort of thing gives me pause: https://danluu.com/julialang/ But the first thing that gave me pause about Julia? They sort of pivoted to say "we're general purpose" but the whole index-starting-at-one thing really belies that -- these days, that's pretty much the province of specialty languages.

You're not supposed to admit it, but I never cared for Dijkstra's arguments on the matter. The same goes for his GOTO tirade, although that has been distorted by time somewhat. Pascal is using 1-ord, Fortran, R, Mathematica. If anything, it seems there's a longer tradition of 1-ord in scientific computing. In this view, I must agree insofar I don't think Julia people are serious about their "general purpose" stance w…

> I never cared for Dijkstra's arguments on the matter.

It resonates with me, as does little-endian. There are evergreen arguments about both of these, but they stand out to me as two of the few places where I appear to be in sync with the majority.

> Pascal is using 1-ord, Fortran, R, Mathematica. If anything, it seems there's a longer tradition of 1-ord in scientific computing.

Actually, Pascal let you define the starting point, but, historically, COBOL (business) and FORTRAN started at one, and these days, it's Matlab (engineering), as well as R, Mathematica, and Julia. But, really, my point was that Julia is hyped to take over from Python, and it seems if that's the focus, then you should think long and hard about what features of Python to change, and, well, obviously that wasn't really the direction they came from.

> semantics like that, without unnecessarily sacrificing performance—I don't think they get enough credit for that from programming guys.

It's good work, but at the same time, a bit overblown IMO. For example, the creators claim homoiconicity, but that stretches the definition of that word to where Python would be homoiconic as well, and it's obviously not.

As far as the multiple dispatch goes, that's actually not that difficult of a puzzle piece when you are compiling ahead of time with unboxed objects.

Re: Five Years of Tinygrad

#83

Post by alienthrowaway »

octoberfranklin wrote: post #72
Barrin92 wrote: post #58

Earlier quoted context omitted.

yes, it's really crazy, if people think you're exaggerating look at this: https://github.com/tinygrad/tinygrad/blob/master/tinygrad/co... One of the cases why I think obsession with lines of code is one of the most counterproductive metrics, it always produces code like this.

Looks fine to me. I'm going to guess that you're used to codebases that use an autoformatter. There's definitely a case to be made for autoformatters in projects with a very large (>100) number of developers, but the price you pay for that is much poorer information density per screenful of code. For things like tinygrad or sqlite with a small dev team it's possible to use whitespace, style, and formatting to communi…

> I'm going to guess that you're used to codebases that use an autoformatter.

I'm used to Python code being "Pythonic" - which is one of those "I know it when I see it" terms.

Re: Five Years of Tinygrad

#84

Post by webdevver »

rent0pat wrote: post #56

I've looked at the "only 18,935 lines of code" python code and it made me want to poke my eyes out. Not sure what's the point of this extreme code-golfing.

Down with large line counts!

LLMs have done two things for us: made lines of code a commodity, but arguably even more important: made understanding an arbitrary amount of lines of code a commodity too.

with this in hand, large codebases that were designed in the primordial era of requiring human cognition to grok are emabarassingly antiquated.

"umm what does this mean?" -> right click -> explain with AI -> "got it".

I literally reddit-soyjacked when i saw "tackle LLVM removal". I would argue that LLVM was/is the most well thought out and well-architected piece of software that exists today... but in the age of AI... who could possibly care now? when I say "well thought out" and "well-architected", that is in the context of human eyeballs. But who needs human eyeballs when we have much more powerful computer eyeballs? who grok code way better and way faster than we do.

now, the massive line count of projects like LLVM, Linux, and Chrome become much harder to defend in the context of code re-use. Why drag in millions of code when you can get an LLM to write just the bits you care about? LLVM has a ton of optimisations that are all well and good, but how many of them actually move the needle on HPC platforms that do an awful lot of instruction re-ordering and re-scheduling anyway?

Could we get away with 5% of the codebase to do "just the bits we care about"? in the past the answer might be, "well yes but its still a lot of (manual) coding", but now - why not? Let an LLM loose on the design doc and fill in the gaps!

Re: Five Years of Tinygrad

#85

Post by sod22 »

ramesh31 wrote: post #35
spiderfarmer wrote: post #7

Earlier quoted context omitted.

I hate it when ‘inspirational’ quotes are attributed to the person with the largest audience and not the people who came up with it, like in this case, the engineers at Lockheed’s Skunk Works.

It's an apocryphal quote. "A designer knows he has achieved perfection not when there is nothing left to add, but when there is nothing left to take away." - Antoine de Saint-Exupéry

I like this. A lot.

Re: Five Years of Tinygrad

#87

Post by kaashif »

geremiiah wrote: post #36

The risk for Tinygrad is that PyTorch will create a new backend for Inductor, plug in their AMD codegen stuff and walala, PyTorch still king. I mean, they could have easily just taken that route themselves instead of bothering with a new ML framework and AD engine. 99% of the work is just the AMD codegen part of the compiler. Either way, super cool project and I wish them the best.

> walala

Is this like voila or something else?

Re: Five Years of Tinygrad

#88

Post by taneq »

kaashif wrote: post #87
geremiiah wrote: post #36

The risk for Tinygrad is that PyTorch will create a new backend for Inductor, plug in their AMD codegen stuff and walala, PyTorch still king. I mean, they could have easily just taken that route themselves instead of bothering with a new ML framework and AD engine. 99% of the work is just the AMD codegen part of the compiler. Either way, super cool project and I wish them the best.

> walala Is this like voila or something else?

I thought it was “wololo” (the sound made by the priest in Age of Empires when it converts an enemy unit to a friendly one.)

Re: Five Years of Tinygrad

#89

Post by measurablefunc »

fragmede wrote: post #54
measurablefunc wrote: post #19

Is it really "Complex"? Or did we just make it "Complicated"? - https://www.youtube.com/watch?v=ubaX1Smg6pY

If you think you can do better, the code's right there and you're welcome to try! There's a giant pile of money at the end of the rainbow for you if you can uncomplexify it. (Love that talk tho)

I don't care for money but you seem to so you should focus on making as much as possible.

Re: Five Years of Tinygrad

#90

Post by ssivark »

octoberfranklin wrote: post #72
Barrin92 wrote: post #58

Earlier quoted context omitted.

yes, it's really crazy, if people think you're exaggerating look at this: https://github.com/tinygrad/tinygrad/blob/master/tinygrad/co... One of the cases why I think obsession with lines of code is one of the most counterproductive metrics, it always produces code like this.

Looks fine to me. I'm going to guess that you're used to codebases that use an autoformatter. There's definitely a case to be made for autoformatters in projects with a very large (>100) number of developers, but the price you pay for that is much poorer information density per screenful of code. For things like tinygrad or sqlite with a small dev team it's possible to use whitespace, style, and formatting to communi…

Even with autoformatters, it's a choice of what criteria you pick. It's possible to have consistency without enforcing low information density.
Post reply on HN