Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GLM-4-9B: open-source model with superior performance to Llama-3-8B

github.com

Post reply on HN
Print view View original

11–18 of 18 posts

Re: GLM-4-9B: open-source model with superior performance to Llama-3-8B

#11

Post by czl »

abrichr wrote: post #10

> GLM-4V-9B possesses dialogue capabilities in both Chinese and English at a high resolution of 1120*1120. In various multimodal evaluations, including comprehensive abilities in Chinese and English, perception & reasoning, text recognition, and chart understanding, GLM-4V-9B demonstrates superior performance compared to GPT-4-turbo-2024-04-09, Gemini 1.0 Pro, Qwen-VL-Max, and Claude 3 Opus. But according to their ow…

Based on size (parameter count) they are in different categories. GLM-4V-9B is in the "light-weight" competition. gpt-4o-2024-05-13 is in the medium or heavy "weight" competition.

Re: GLM-4-9B: open-source model with superior performance to Llama-3-8B

#12

Post by refulgentis »

Ehhh man this is frustrating, 7B was a real sweet spot for hobbyist. 8B...doable. I've been joking to myself/simultaneously worried that Llama 3 8B and Phi-3 "3B" (3.8B) would start a "ehhh, +1, might as well be a rounding error" thing. It's a big deal! I measure a 33% decrease just going from 3B to 3.8B when inferencing on CPU.

Re: GLM-4-9B: open-source model with superior performance to Llama-3-8B

#13

Post by reaperman »

norwalkbear wrote: post #7

Isnt 3-70b so good, reddit llamaers are saying people should buy hardware to run it? Llama-3-8b was garbage for me but damn 70b is good enough

The unquantized llama 70B requires 142GB of VRAM. Some of the quantized versions are quite decent but they do tend to get overquantized below around 26.5GB of VRAM (~3 bits per weight).

So you’d at minimum be looking at dual 3090 with NVLink for about $4000 or so. Or for the highest performing non-quantized model, you’d be spending about $40,000 for two A100’s.

Re: GLM-4-9B: open-source model with superior performance to Llama-3-8B

#14

Post by Manabu-eo »

reaperman wrote: post #13
norwalkbear wrote: post #7

Isnt 3-70b so good, reddit llamaers are saying people should buy hardware to run it? Llama-3-8b was garbage for me but damn 70b is good enough

The unquantized llama 70B requires 142GB of VRAM. Some of the quantized versions are quite decent but they do tend to get overquantized below around 26.5GB of VRAM (~3 bits per weight). So you’d at minimum be looking at dual 3090 with NVLink for about $4000 or so. Or for the highest performing non-quantized model, you’d be spending about $40,000 for two A100’s.

No need for NVLink just for inference, not even with tensor parallelism. And you can get used 3090 much cheaper than that.

Re: GLM-4-9B: open-source model with superior performance to Llama-3-8B

#15

Post by wkat4242 »

Manabu-eo wrote: post #14
reaperman wrote: post #13

Earlier quoted context omitted.

The unquantized llama 70B requires 142GB of VRAM. Some of the quantized versions are quite decent but they do tend to get overquantized below around 26.5GB of VRAM (~3 bits per weight). So you’d at minimum be looking at dual 3090 with NVLink for about $4000 or so. Or for the highest performing non-quantized model, you’d be spending about $40,000 for two A100’s.

No need for NVLink just for inference, not even with tensor parallelism. And you can get used 3090 much cheaper than that.

True, you can buy 2x 4090 FE brand new for $4000

Re: GLM-4-9B: open-source model with superior performance to Llama-3-8B

#16

Post by mikeqq2024 »

ilaksh wrote: post #3

Looks like terrific technology. However, the translation says that it's an "irrevocable revocable" non-commercial license with a form to apply for commercial use.

"non-exclusive, worldwide, irrevocable, non-sublicensable, revocable, photo-free copyright license."

Translation error? output from GPT: "non-exclusive, global, non-transferable, non-sublicensable, revocable, royalty-free license."

Re: GLM-4-9B: open-source model with superior performance to Llama-3-8B

#17

Post by wtarreau »

freeqaz wrote: post #6
great_psy wrote: post #2

I’m excited to hear work is being done on models that support function calling natively. Does anybody know if performance could be greatly increased if only a single language was supported ? I suspect there’s a high demand for models that are maybe smaller and can run faster if the tradeoff is support for only English. Is this available in ollama ?

Are there any other models that support function calling?

I ran some tests on phi-3 and mistral-7b and it's not very hard to teach them to use tools, even though they were not designed for this. It turns out these models obey their instructions quite well and when you explain them that if they need to look up data on the net or to perform a calculation, they must formulate this demand with a specific syntax, they do a pretty good job. You just have to enable reverse-prompting so that the evaluation stops after their demand, your tools do the job (or you simulate it manually) and their task continues.

Re: GLM-4-9B: open-source model with superior performance to Llama-3-8B

#18

Post by norwalkbear »

reaperman wrote: post #13
norwalkbear wrote: post #7

Isnt 3-70b so good, reddit llamaers are saying people should buy hardware to run it? Llama-3-8b was garbage for me but damn 70b is good enough

The unquantized llama 70B requires 142GB of VRAM. Some of the quantized versions are quite decent but they do tend to get overquantized below around 26.5GB of VRAM (~3 bits per weight). So you’d at minimum be looking at dual 3090 with NVLink for about $4000 or so. Or for the highest performing non-quantized model, you’d be spending about $40,000 for two A100’s.

So a MacBook m series is a decent buy
Post reply on HN