Viewing profile — eugenhotaj

This is because everyone is training with synchronous sgd. all gpus need to synchronize on each gradient step so tail latency will kill you.

This is pretty cool. I had the same idea but in zig: https://github.com/EugenHotaj/zig_gpt2 Not fully finished yet, haven't gotten around to implementing bpe encoding/decoding and …

In my experience processes like these rarely work out as intended and usually add layers of bureaucracy for marginal benefit. It’s usually senior engineers or middle managers looki…

This post would sound so dumb if it didn’t come from the almighty pg.

The issue is not that ChatGPT will kill things off, the issue is that ChatGPT 4.0 will kill things off. If you don’t think that’s a real possibility, you’re sleeping.

Some of these are so odd. When Kylie Jenner launched her company she could have sold celery and still made $1B+. It wasn’t because of the effectiveness of her small team.

My experience is exactly the opposite in almost all cases. Most software is much more complicated than it needs to be. Reads like the author is just butthurt at feedback they recei…

I’ve likely had it all my life, but really started noticing about two years ago during the pandemic. Now I can’t unhear it. Went to a doctor a couple of times but nothing they trie…

The network communication overhead would be way too high to make this useful. At least for current methods of training large models.

Source: trust me bro

Now someone just needs to pipe the output into stable diffusion.

Just take a look at big tech salaries on levels.fyi. Very senior engineers (L8/L9) easily make 10x the salary of entry to mid-level engineers (L3/L4/L5). For a lot of tech companie…

Let me guess, someone pushed a bad config.

This is neat for toy problems but I don't see it working well for "real" pipelines. The magical DAG creation is going to be super hard to wrap your head around and even worse to de…

> Since the bigger network contains the smaller network, it is perfectly capable of achieving the same performance, so the only reason why this does not happen is that SGD cannot f…

If your pipeline uses only “classic” ml models, then this won’t make too much sense. It’s mostly applicable to NNs.

I also don’t have any data, but I feel like these types of tactics don’t account for higher order effects of pushing people to more extreme platforms, and in general seem extremely…

For kernel SVMs, one needs to keep around part of the training data (the support vectors) right? With DNNs, after training, all you need are the model parameters. For very large da…

True, but it sounds like you’re just shifting computation from training to inference. And I’m not sure that’s a very good trade off to make, you’re likely to predict on much more d…

Fair enough, but the number of support vectors for non trivial problems is still pretty large (as I understand but could be wrong), e.g. 20-30% of the dataset. Having to iterate ov…