List me AI models that I can run with my hw?

#1

Post by andrea81 » Mon, Mar 11, 2024, 4:49 PM UTC

Tired of seeing AI model names with 300m of parameters.

Actually I want to see the ones compatible to run locally in my current laptop (I'm NOT going to spend money to run a model).

Is there a website where I can enter my laptop specs and list me AI models?

Re: List me AI models that I can run with my hw?

#2

Post by PaulHoule » Mon, Mar 11, 2024, 4:59 PM UTC

Use sbert.net if you believe Steven Covey and your goal is for your model to “first understand”.

Re: List me AI models that I can run with my hw?

#3

Post by pinnaclehemp » Mon, Mar 11, 2024, 5:00 PM UTC

[dead]

Re: List me AI models that I can run with my hw?

#4

Post by BillyTheMage » Mon, Mar 11, 2024, 5:05 PM UTC

pinnaclehemp wrote: post #3

[dead]

Account created 5 minutes ago spouting absolutely unrelated nonsense? Seems like a bot to me.

Re: List me AI models that I can run with my hw?

#5

Post by declaredapple » Mon, Mar 11, 2024, 5:31 PM UTC

BillyTheMage wrote: post #4

pinnaclehemp wrote: post #3

[dead]

Account created 5 minutes ago spouting absolutely unrelated nonsense? Seems like a bot to me.

In this case it appears to be for SEO spam to gain backlinks

Re: List me AI models that I can run with my hw?

#6

Post by declaredapple » Mon, Mar 11, 2024, 5:34 PM UTC

Take # parameters and divide by 2 and then add like 1 to estimate it for 4bit quantization. A 13B model will need ~7GB of memory at 4bit.

Most quantization models start falling apart rapidly after 4bit, but you can go lower...

If you can fit it in GPU memory it'll likely be usable, if you can't fit it in memory be prepared for single digit t/s or less, the only exception is for people with M1/M2/M3 cpus.