Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: AI/ML papers to catch up with current state of AI?

news.ycombinator.com

Post reply on HN
Print view View original

31–40 of 52 posts

Re: Ask HN: AI/ML papers to catch up with current state of AI?

#31

Post by davikr »

jpdus wrote: post #4

Hey, imho best overall technical intro to LLMs (I guess that´s your main interest as you mentioned qlora + llama) is by Simon Willis [1]. Additionally or if you prefer videos, the recent 1h "busy persons intro" by Andrei Karpathy is great + dense as well [2]. [1] https://simonwillison.net/2023/Aug/3/weird-world-of-llms/ [2] https://youtu.be/zjkBMFhNj_g?si=M6pRX66NrRyPM8x- EDIT: Maybe I misunderstood as you asked abou…

This, but I'd replace Reddit with 4chan. There is a lot more information on how to build, finetune and run models there, compared to Reddit.

Re: Ask HN: AI/ML papers to catch up with current state of AI?

#32

Post by andyjohnson0 »

lysecret wrote: post #27

The good (and some might say bad thing) is that when it comes to fundamental technologies there are only 2 that are relevant: 1. Transformers 2. Diffusion The benefit is that, focus on understanding them both reeaaalllyy well and you are at the forefront of research;) Also, what is the reason you want to do this? If it is about building some kind of AI enabled app, you don't have to read anything. Get an API key and…

> there are only 2 that are relevant: 1. Transformers 2. Diffusion

I'd argue that there are plenty of less sexy, non-unicorn uses for AI/ML - particularly in industrial applications. SVMs, DNNs, etc are still very relevant. As is GOFAI in some domains.

Re: Ask HN: AI/ML papers to catch up with current state of AI?

#33

Post by hapanin »

Since nobody is actually recommending papers, here's an incomplete reading list that I sent out to some masters students I work with so they can understand the current research (academic) my little team is doing:

Paper reference / main takeaways / link

instructGPT / main concepts of instruction tuning / https://proceedings.neurips.cc/paper_files/paper/2022/hash/b...

self-instruct / bootstrap off models own generations / https://arxiv.org/pdf/2212.10560.pdf

Alpaca / how alpaca was trained / https://crfm.stanford.edu/2023/03/13/alpaca.html

Llama 2 / probably the best chat model we can train on, focus on training method. / https://arxiv.org/abs/2307.09288

LongAlpaca / One of many ways to extend context, and a useful dataset / https://arxiv.org/abs/2309.12307

PPO / important training method / idk just watch a youtube video

Obviously these are specific to my work and are out of date by ~3-4 months but I think they do capture the spirit of "how do we train LLMs on a single GPU and no annotation team" and are frequently referenced simply by what I put in the "paper reference" column.

Re: Ask HN: AI/ML papers to catch up with current state of AI?

#34

Post by magoghm »

bilsbie wrote: post #26
magoghm wrote: post #16

The book that just came out, "Understanding Deep Learning", is an excellent overview of the current state of AI: https://udlbook.github.io/udlbook/ Read that first, then to keep up to date you can follow up with any papers that seem interesting to you. A good way to be aware of the interesting papers that come out is to follow @_akhaliq on X: https://twitter.com/_akhaliq

What do you think of this book? https://fleuret.org/francois/lbdl.html I like that it’s formatted for the phone.

I think it is quite good if what you need is to get very quickly a simple overview of the current state of AI.

Re: Ask HN: AI/ML papers to catch up with current state of AI?

#35

Post by sgt101 »

No emergence

[2304.15004] Are Emergent Abilities of Large Language Models a Mirage? - arXiv https://arxiv.org/abs/2304.15004

Can't plan

https://openreview.net/forum?id=X6dEqXIsEW

No compositionality https://openreview.net/forum?id=Fkckkr3ya8

Apart from that it's great

Re: Ask HN: AI/ML papers to catch up with current state of AI?

#36

Post by magoghm »

magoghm wrote: post #16

The book that just came out, "Understanding Deep Learning", is an excellent overview of the current state of AI: https://udlbook.github.io/udlbook/ Read that first, then to keep up to date you can follow up with any papers that seem interesting to you. A good way to be aware of the interesting papers that come out is to follow @_akhaliq on X: https://twitter.com/_akhaliq

Another book which also seems to be very good is "Deep Learning, Foundations and Concepts". It is coming out soon and you can already preview it on-line at https://www.bishopbook.com/

Re: Ask HN: AI/ML papers to catch up with current state of AI?

#37

Post by cs702 »

soerxpso wrote: post #28
cs702 wrote: post #23

Earlier quoted context omitted.

I'd be wary of programmatic lists that claim to track the most important/popular recent papers. There's a ridiculous amount of hype/propaganda and citation hacking surrounding new AI research, making it hard to discern what will truly stand the test of time. Tomas Mikolov just posted about this: https://news.ycombinator.com/item?id=38654038

It doesn't claim to track the most important recent papers. It's very clear and upfront that it aims to track the most trending recent papers. It's even in the title of the website. There's no claim of permanent importance.

Thank you. You're right. I edited my comment so it refers to "important/popular" instead.

Re: Ask HN: AI/ML papers to catch up with current state of AI?

#38

Post by meltyness »

lysecret wrote: post #27

The good (and some might say bad thing) is that when it comes to fundamental technologies there are only 2 that are relevant: 1. Transformers 2. Diffusion The benefit is that, focus on understanding them both reeaaalllyy well and you are at the forefront of research;) Also, what is the reason you want to do this? If it is about building some kind of AI enabled app, you don't have to read anything. Get an API key and…

To that, what can these express, precisely, is an interesting question; so for transformer encoders:

https://arxiv.org/abs/2301.10743

Another interesting research topic is the trusted generation of tasks for finetuning

https://arxiv.org/abs/2306.08568

And I suppose too running these at the edge is terribly interesting too, if you can find analyses of "quantization" this is a highly active research are, and results are pretty incredible since it cuts resources by huge factors and no one quite knows why.

This is one that's easy to dive into with consumer hardware, but don't know any great papers myself

Run locally: https://github.com/ggerganov/llama.cpp

Quantized models: https://huggingface.co/TheBloke

Explainability is under research, though I haven't seen any good solutions.

This nay arise from skeptics who are calling the things stochastic parrots, incapable of reason, without a world model, etc.

Re: Ask HN: AI/ML papers to catch up with current state of AI?

#39

Post by gurovich »

This resource has been invaluable to me: https://paperswithcode.com/

From the past examples you give it sounds like you were into computer vision. There’s been a ton of developments since then, and I think you’d really enjoy the applications of some of those classic convolutional and variational encoder techniques in combination with transformers. A state of the art multimodal non-autoregressive neural net model such as Google’s Muse is a nice paper to work up to, since it exposes a breadth of approaches.

Re: Ask HN: AI/ML papers to catch up with current state of AI?

#40

Post by pomatic »

kevindamm wrote: post #20
pomatic wrote: post #19

Posted in another thread, but sadly I got no replies... Related question: how can I learn how to read the mathematical notation used in AI/ML papers? Is there a definitive work that describes the basics? I am a post-grad Engineer, so I know the fundamentals, but I'm really struggling with a lot of the Arxiv papers. Any pointers hugely appreciated.

I particularly enjoyed Kevin Murphy's book [0] for being just rigorous enough to satisfy but not too dry, but also not trying to add humor unnecessarily. It's not the best introduction text but it's great for someone with a little familiarity in the field who wants to broaden their understanding. There are proofs to rationalize some approaches, but not to the degree that would satisfy a hardcore mathematicians maybe,…

Thank you!
Post reply on HN