SmolLM2 • Hacker News Board

Re: SmolLM2

#11

Post by turblety » Sat, Nov 02, 2024, 9:29 AM UTC

echoangle wrote: post #10

Semi-related but is there a standard way to run this (or other models from huggingface) in a docker container and interact with them through a web API? ChatGPT tells me to write my own FastAPI wrapper which should work, but is there no pre-made solution for this?

Ollama has built in support [1] for gguf models on huggingface, and exposes a openai compatible http endpoint [2].

You can also just test it out using the cli:

ollama run hf.co/unsloth/SmolLM2-1.7B-Instruct-GGUF:F16

1. https://huggingface.co/docs/hub/ollama

2. https://github.com/ollama/ollama?tab=readme-ov-file#start-ol...

Re: SmolLM2

#12

Post by d_k_f » Sat, Nov 02, 2024, 9:30 AM UTC

ksri wrote: post #2

Is there a way to run this in the browser as yet? Transformers js doesn't seem to support this. Is there another way to run this in the browser?

They linked two examples in another blog post, only the smaller models, though:

[135M] https://huggingface.co/spaces/HuggingFaceTB/SmolLM-135M-Inst...

[360M] https://huggingface.co/spaces/HuggingFaceTB/SmolLM-360M-Inst...

Re: SmolLM2

#13

Post by jdthedisciple » Sat, Nov 02, 2024, 9:32 AM UTC

Very interesting. According to their X posts, this meme model "SmolLm" beats Meta's new 1B and 3B models across almost all metrics.

I wonder how this is possible given that Meta has been in this game for much longer and probably has much more data at their disposal as well.

Re: SmolLM2

#14

Post by cloudbonsai » Sat, Nov 02, 2024, 9:36 AM UTC

cpa wrote: post #8

What’s the context size? I couldn’t find it on the model summary page. Tangential: if it’s not on the model page, does it mean that it’s not that relevant here? If so, why?

> What’s the context size?

SmolLM2 uses up to 8192 tokens.

Re: SmolLM2

#15

Post by forrestthewoods » Sat, Nov 02, 2024, 9:39 AM UTC

Is there a good, small model that can take input images? Or are those all still larger?

Re: SmolLM2

#16

Post by stavros » Sat, Nov 02, 2024, 9:46 AM UTC

jdthedisciple wrote: post #13

Very interesting. According to their X posts, this meme model "SmolLm" beats Meta's new 1B and 3B models across almost all metrics. I wonder how this is possible given that Meta has been in this game for much longer and probably has much more data at their disposal as well.

Usually, that's because they use a groundbreaking ML method called TTDS, or "training on the test dataset".

Re: SmolLM2

#17

Post by JimDabell » Sat, Nov 02, 2024, 10:04 AM UTC

forrestthewoods wrote: post #15

Is there a good, small model that can take input images? Or are those all still larger?

I haven’t tried the smaller variants, but I’ve been very impressed with Molmo:

https://molmo.allenai.org

Re: SmolLM2

#18

Post by exe34 » Sat, Nov 02, 2024, 10:05 AM UTC

echoangle wrote: post #10

Semi-related but is there a standard way to run this (or other models from huggingface) in a docker container and interact with them through a web API? ChatGPT tells me to write my own FastAPI wrapper which should work, but is there no pre-made solution for this?

llama.cpp in a docker container (Google for the gguf version)

Re: SmolLM2

#19

Post by exe34 » Sat, Nov 02, 2024, 10:06 AM UTC

stavros wrote: post #16

jdthedisciple wrote: post #13

Very interesting. According to their X posts, this meme model "SmolLm" beats Meta's new 1B and 3B models across almost all metrics. I wonder how this is possible given that Meta has been in this game for much longer and probably has much more data at their disposal as well.

Usually, that's because they use a groundbreaking ML method called TTDS, or "training on the test dataset".

meta or smol?

Re: SmolLM2

#20

Post by stavros » Sat, Nov 02, 2024, 10:12 AM UTC

exe34 wrote: post #19

stavros wrote: post #16

Earlier quoted context omitted.

Usually, that's because they use a groundbreaking ML method called TTDS, or "training on the test dataset".

meta or smol?

Smol