Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Long Live the 'GPU Poor' – Open-Source AI Grants

a16z.com

Post reply on HN
Print view View original

81–90 of 124 posts

Re: Long Live the 'GPU Poor' – Open-Source AI Grants

#81

Post by tikkun »

What are other open source LLM projects that people rely on? As a way to come up with ideas for future grant recipients.

This is a good initiative, and an excellent initial batch. A great use of management fees!

I'm struggling to think of additional ones because I feel like this covers everyone I would've suggested (particularly TheBloke, vLLM and oobabooga).

Edit, some ideas for future grant recipients would be:

* MLC LLM

* ExLlama

* An open source fork of text-generation-inference

* AutoGPTQ

Re: Long Live the 'GPU Poor' – Open-Source AI Grants

#82

Post by ForHackernews »

Kon-Peki wrote: post #32

I'm glad you guys are sharing some money around, but I really have a hard time thinking of a person with "Dual 4090s" as "GPU Poor". LOL

It's like how you're house poor if you only have one brownstone in Brooklyn and a pied-à-terre in Palo Alto.

Re: Long Live the 'GPU Poor' – Open-Source AI Grants

#83

Post by kcorbitt »

samstave wrote: post #76
rajko_rad wrote: post #25

Earlier quoted context omitted.

in most cases the teams already had their own source or platform of choice!

Maybe build a FAQ for others on how to accomplish, or resources you can point people at?

A lot of people are using RunPod for experimental/small-scale workloads. They have good network and disk speeds and you can generally find availability for a latest-gen GPU like an L40 or 4090 if your workload can fit on a single GPU. One GPU is plenty for fine-tuning Llama 2 7B or 13B with LoRA or QLoRA. They also sometimes have availability for multi-GPU servers like 8xA100s, but that's more hit-or-miss.

If you want to go even cheaper vast.ai is a popular option. It's a P2P marketplace for individuals to rent out their GPUs. You can generally get a ~20-30% discount vs RunPod prices by using Vast, but network speeds and perf are much more variable and there's always the possibility that the host will just shut you off without warning. I also wouldn't recommend using it if you're training with proprietary data since they can't guarantee the host isn't logging it, but most of the OSS fine-tuning community publishes their datasets anyway.

Re: Long Live the 'GPU Poor' – Open-Source AI Grants

#84

Post by rajko_rad »

thewataccount wrote: post #80
rajko_rad wrote: post #4

Hi HN, we are super excited to announce this new initiative!! Please note, this program was designed to support individuals, teams, and hackers who are not pursuing commercial companies. Nevertheless, these projects push the state of the art in open source AI and help provide us with a more robust and comprehensive understanding of the technology as it is developed. We are really proud to be contributing in this smal…

Hello, I think this is a great project! If an individual is looking to contribute to the field with different training data ideas - would they need to first establish themselves and get your attention, or would there be a way to submit a proposal? For myself compute is rather expensive, I could likely afford a few test runs for proofs of concept but beyond that it would be difficult. I've got a single 4090 so I can't…

Thank you so much for your work and enthusiasm here!! Unfortunately, that’s roughly the best way to approach it and how most of these have worked so far, demonstrations on smaller models (e.g. 7/13B), with slightly smaller datasets, catching the eye of the community, etc. in general it’s not a bad approach to research too, prove out concepts at smaller scale before scaling up!

Re: Long Live the 'GPU Poor' – Open-Source AI Grants

#86

Post by syntaxing »

Whoa this is honestly awesome, a lot of tools I see on the list are ones I’ve been using so it’s great to see they’re getting financial support. Really hoping Ollama joins the list soon. I have some fun ideas I’ve been wanting to test out but probably requires a couple hours on 8X A100s + EvolveInstruct style data that cost upward of $1K+. Gonna start preparing some code to see if I can join grant #2!

Re: Long Live the 'GPU Poor' – Open-Source AI Grants

#87

Post by thewataccount »

rajko_rad wrote: post #84
thewataccount wrote: post #80

Earlier quoted context omitted.

Hello, I think this is a great project! If an individual is looking to contribute to the field with different training data ideas - would they need to first establish themselves and get your attention, or would there be a way to submit a proposal? For myself compute is rather expensive, I could likely afford a few test runs for proofs of concept but beyond that it would be difficult. I've got a single 4090 so I can't…

Thank you so much for your work and enthusiasm here!! Unfortunately, that’s roughly the best way to approach it and how most of these have worked so far, demonstrations on smaller models (e.g. 7/13B), with slightly smaller datasets, catching the eye of the community, etc. in general it’s not a bad approach to research too, prove out concepts at smaller scale before scaling up!

Totally understand! I'm sure you'd get swarmed with requests.

Are you interested in non-LLMs as well? Stable diffusion for example?

Re: Long Live the 'GPU Poor' – Open-Source AI Grants

#88

Post by rajko_rad »

syntaxing wrote: post #86

Whoa this is honestly awesome, a lot of tools I see on the list are ones I’ve been using so it’s great to see they’re getting financial support. Really hoping Ollama joins the list soon. I have some fun ideas I’ve been wanting to test out but probably requires a couple hours on 8X A100s + EvolveInstruct style data that cost upward of $1K+. Gonna start preparing some code to see if I can join grant #2!

Love it!! Good luck with the mvps!

And thank you so much for the input - we are meeting the folks at ollama!!

Re: Long Live the 'GPU Poor' – Open-Source AI Grants

#89

Post by rajko_rad »

thewataccount wrote: post #87
rajko_rad wrote: post #84

Earlier quoted context omitted.

Thank you so much for your work and enthusiasm here!! Unfortunately, that’s roughly the best way to approach it and how most of these have worked so far, demonstrations on smaller models (e.g. 7/13B), with slightly smaller datasets, catching the eye of the community, etc. in general it’s not a bad approach to research too, prove out concepts at smaller scale before scaling up!

Totally understand! I'm sure you'd get swarmed with requests. Are you interested in non-LLMs as well? Stable diffusion for example?

Definitely. This was kicked off in the last month or two when there just seemed to be a bit more happening in OSS LLMs, but we are prioritizing diversifying this for the next cohort!
Post reply on HN