111–120 of 127 posts
Hi Andrej, Huge fan of all the work you do. Wanted to understand something fundamental and whom better to ask than you: Whats so special about the transformer architecture that its able to predict the next token so beautifully understanding all the intricate previous token relationships? I understand Attention but what so special about this architecture that no other architectures are able to "attend" appropriately t…
In theory, if you grew the RNN's state quadratically vs sequence length, you could likely achieve comparable performance to transformers, but it would likely be less efficient than transformers.
Is this the sort of thing that a person with curiosity and a 4090 could do? It says he used 8xA100s in the cloud to do this but is it just a matter of the 4090 going 8x slower or will memory constraints kill the whole endeavour?
Earlier quoted context omitted.
Fun trivia: downloading 45TB costs about $60, according to Cloudflare.
That's what Cloudflare charges. It costs them around 6 cents.
Hi HN the main (more detailed) article is here https://github.com/karpathy/llm.c/discussions/481 Happy to answer questions!
Would you consider switching your interest to protein structure prediction? In particular, the current most advanced model is a closed-source, closed-weights system that was trained on a proprietary hardware. It is intentionally kept that way for now to enable deepmind to commercialize their product. The goal here isn't to make the best performing model: it's ablation. How much can we remove from protein structure pr…
That's not usually how it works.
> Just continue to show that all that's required to predict novel structures is the existing PDB.
Sounds like you know a lot about this topic. You should do it!
Earlier quoted context omitted.
It's almost as if people believe in fairness and compensating people for their work. Also, it's worth noting that this is only true as long as we're stuck in the "must train on the entire sum total of human output ever created" local minimum for machine learning. Given that most biological entities learn with much less data, this might well be the thing that prods ML research to using an approach that isn't "IDK, buy…
> It's almost as if people believe in fairness and compensating people for their work. Yet in this case we are talking about compensating the compilers/massagers/owners of the datasets, not the original authors from wherever the data was originally scraped.
That theory's a fantasy, because extractive systems involving gatekeepers get established, but in this specific case, enforcing copyright would make things fairer for authors. There's no extractive copyright-taking gatekeeper for websites: scrapers don't get copyright, so can't re-license the material they've scraped (unless it's permissively-licensed or something).
Earlier quoted context omitted.
Would you consider switching your interest to protein structure prediction? In particular, the current most advanced model is a closed-source, closed-weights system that was trained on a proprietary hardware. It is intentionally kept that way for now to enable deepmind to commercialize their product. The goal here isn't to make the best performing model: it's ablation. How much can we remove from protein structure pr…
> switching your interest That's not usually how it works. > Just continue to show that all that's required to predict novel structures is the existing PDB. Sounds like you know a lot about this topic. You should do it!
In your opinion is it important for ML engineers to know C?
Spend one year to study multiple languages - bash, C, C++, Go, Python ... and even Mojo or Rust. 10-20 hours a week. Being able to read top programming languages is the best investment I ever made. You will become fearless and can see the matrix ;)
https://mode80.github.io/7-langs-in-12-months.html
I don't regret it. But if ML is your main goal, Python is where you will end up because it's where the libraries are.
Earlier quoted context omitted.
Fun trivia: downloading 45TB costs about $60, according to Cloudflare.
That's what Cloudflare charges. It costs them around 6 cents.
Where are you getting 6 cents from?