Viewing profile — tehsauce

For the purpose of disclosure, it should say “Warning: AI generated code” in the commit message, not an advertisement for a specific product. You would never accept any of your oth…

“888 KiB Assistant” but the assistant itself is a multi terabyte rental-only model stored in some mysterious data center.

How does it do on gold stake?

Right, and when they compare to floating point accuracy they seem to be using the number of decimals supported by the mantissa, but the exponent is important no?

“A transformer predicts the next token” Nope. A transformer is much more general than that. A GPT predicts the next token.

I was excited to try it out so I downloaded the repo and ran the build. However there were 100+ compilation errors. So I checked the commit history on github and saw that for at le…

I love this! Your results seem comparable to the counter strike or minecraft models from a bit ago with massively less compute and data. It's particularly cool that it uses real wo…

There has been some good research published on this topic of how RLHF, ie aligning to human preferences easily introduces mode collapse and bias into models. For example, with a pr…

We have a shared community map where you can watch hundreds of agents from multiple peoples training runs playing in real time! https://pwhiddy.github.io/pokerl-map-viz/

It's impossible to beat with random actions or brute force, but you can get surprisingly far. It doesn't take too long to get halfway through route 1, but even with insane compute …

Anyone interested in watching lots of reinforcement agents playing pokemon red at once, we have a website which streams hundreds of concurrent games from multiple people’s training…

the metal backend does currently generate quite a lot of unnecessary command buffers, but in general performance seems solid.

I haven’t gone through the paper in detail yet but maybe someone can answer. If you remove the hidden state from an rnn as they say they’ve done, what’s left? An mlp predicting fro…

The water consumed to produce a single hamburger is over 2000 liters, and the power likely well over 100 watt-hours. That means gpt can write >1000 emails using the resources of fe…

Awesome article! Something slightly misleading though - the first image shows the intersection of a non-convex shape, but it isn't revealed until much later that the algorithm only…

Grokking is a sudden huge jump in test accuracy with increasing training steps, well after training accuracy has fully converged. Double descent is test performance increasing, dec…

If cpu softmax were limited by memory bandwidth, then these vectorization optimizations wouldn't improve performance.

+1 for vast. they usually are the cheapest and have the most supply. some instances can be less reliable at the low end though

It's possible you might not need direct access to wave/subgroup ops to implement efficient stream compaction. There's a great old Nvidia blog post on "warp-aggregated atomics" http…

500GB/s is going to limit it to at best 1/4 the DL performance of an nvidia gpu. I’m not sure what the floating point perf of these FPGAs are but I imagine that also might set a fu…

Systolic arrays are essentially how matmul is implemented in tensor cores in GPUs and TPUs.

Not sure if he still is, but definitely was. Many of his videos are filmed in the Amazon NYC office.

If you have written something like this somewhere, or know someone that has, please share!

ASI is nothing like a corporation