Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

InternLM – new open source 7B LLM

github.com

Post reply on HN
Print view View original

11–20 of 93 posts

Re: InternLM – new open source 7B LLM

#11

Post by airgapstopgap »

Note that this is apparently a 7B version of a 104B model trained with the intention of competing with OpenAI offerings on the Chinese market [1]. There is a number of those projects: Baichuan, ChatGLM2, InternLM and some more iirc, and they all have small-scale opensource versions.

For what it's worth, I've tried out ChatGLM2-6B and Baichuan converted to LLaMA (the architecture is literally identical in that case). They're okay, though underwhelming given their reported benchmarks; probably the main point of creating them is gaining experience for engineers, and feedback from the wider community that has less incentive to downplay their shortcomings.

Surprisingly, they do not appear censored in any particularly "Chinese" political direction, but they share sensibilities of ChatGPT and Claude.

1. https://github.com/InternLM/InternLM-techreport

Re: InternLM – new open source 7B LLM

#12

Post by freediver »

Tried it with a few PyLLMs benchmarks [1] and it did not do very well.

Q. Use m to substitute p, a to substitute e, n to substitute a, g to substitute c, o to substitute h, how to spell peach under this rule?

A.

m: p

a: e

n: a

g: c

o: h

how: peach

Q. write three sentences each ending with the word apple

A.

1. The apple tree was covered in apples.

2. She sliced the apple with a knife.

3. The apple juice was too sweet.

Wonder if the synthetic benchmarks are very biased and not that useful.

[1] https://github.com/kagisearch/pyllms/blob/6c11cd51b099d33e61...

Re: InternLM – new open source 7B LLM

#13

Post by exizt88 »

> The code in this repository is open-source under the Apache-2.0 license. The InternLM weights are fully open for academic research and also allow commercial use with written permission from the official team. For inquiries about commercial licenses and collaborations, please contact internlm@pjlab.org.cn.

This makes me much less excited about this model.

Re: InternLM – new open source 7B LLM

#14

Post by rfoo »

tyfon wrote: post #4

> trust_remote_code=True This is a hard no from me, anyone know why this is so common in models from China? I'm not getting into conspiracies or anything here, but I've seen it in quite a few others from there. I wouldn't run a model with this requirement from anyone else for that matter.

That's because the model architecture hasn't been added to huggingface/transformers yet, because it literally was just published today.

    >>> from transformers import AutoTokenizer, AutoModel
    >>> model = AutoModel.from_pretrained("internlm/internlm-chat-7b", trust_remote_code=True, device='cuda')
Here, the "trust_remote_code=True" means "download the model code from huggingface repo 'internlm/internlm-chat-7b'", along with the weight, and run it. If it's False, the library would use builtin model architectures hardcoded in huggingface/transformers and only download the weight.

The scary flag is here because, of course, newcomers may not realize that model == code and if you load arbitrary model you are likely executing arbitrary code.

Wonder why, for example, you don't remember seeing LLaMA had this on release day? Because they don't use huggingface transformers library and don't use huggingface to distribute their model. You just clone and run their code from GitHub, and... how is this not "trust_remote_code"?

Re: InternLM – new open source 7B LLM

#15

Post by rfoo »

jabbany wrote: post #10
tyfon wrote: post #4

> trust_remote_code=True This is a hard no from me, anyone know why this is so common in models from China? I'm not getting into conspiracies or anything here, but I've seen it in quite a few others from there. I wouldn't run a model with this requirement from anyone else for that matter.

Seems pretty common though, for defining custom architecture configs whatnot? AFAIK the "remote code" is still openly hosted on huggingface so you can audit it if you like. Seems no more dangerous than things like `pip install some_random_library`?

I like this pip metaphor. If we had required `--trust-remote-code` for every `npm install` we could have avoided left-pad and most of the software supply chain drama in the past years.

Re: InternLM – new open source 7B LLM

#16

Post by hmottestad »

exizt88 wrote: post #13

> The code in this repository is open-source under the Apache-2.0 license. The InternLM weights are fully open for academic research and also allow commercial use with written permission from the official team. For inquiries about commercial licenses and collaborations, please contact internlm@pjlab.org.cn. This makes me much less excited about this model.

Is that even valid? This seems to be the only place where they’ve made this exemption, it’s not written in the license. Even the weights on hugging face are licensed under apache 2.0.

Doesn’t apache 2.0 allow for fairly unrestricted commercial use? Isn’t that the whole point of using that license?

Re: InternLM – new open source 7B LLM

#17

Post by ohgodplsno »

exizt88 wrote: post #13

> The code in this repository is open-source under the Apache-2.0 license. The InternLM weights are fully open for academic research and also allow commercial use with written permission from the official team. For inquiries about commercial licenses and collaborations, please contact internlm@pjlab.org.cn. This makes me much less excited about this model.

Less excited that you can't freely take work from academics to resell it in a shitty SaaS company ?

Re: InternLM – new open source 7B LLM

#18

Post by speedgoose »

ohgodplsno wrote: post #17
exizt88 wrote: post #13

> The code in this repository is open-source under the Apache-2.0 license. The InternLM weights are fully open for academic research and also allow commercial use with written permission from the official team. For inquiries about commercial licenses and collaborations, please contact internlm@pjlab.org.cn. This makes me much less excited about this model.

Less excited that you can't freely take work from academics to resell it in a shitty SaaS company ?

In Europe we call that a success story.

Re: InternLM – new open source 7B LLM

#19

Post by moffkalast »

Tepix wrote: post #6

I welcome new models! The more, the merrier. That said, this model has been tailored but they are comparing it to non-finetuned LLaMA-7B in their benchmark? That seems a bit fainthearted.

> HumanEval: InternLM-7B: 10.4, LLaMA-7B: 14.0

The funny part is that the base model apparently outperforms the fine tune.

So far the HumanEval benchmark seems to be the only one that can objectively compare overall model performance despite being a coding-only benchmark, the rest mostly just give a "99.7% chatgpt" bullshit results. Turns out you can't compare creative writing because all outputs are basically valid.

Re: InternLM – new open source 7B LLM

#20

Post by tyfon »

rfoo wrote: post #14
tyfon wrote: post #4

> trust_remote_code=True This is a hard no from me, anyone know why this is so common in models from China? I'm not getting into conspiracies or anything here, but I've seen it in quite a few others from there. I wouldn't run a model with this requirement from anyone else for that matter.

That's because the model architecture hasn't been added to huggingface/transformers yet, because it literally was just published today. >>> from transformers import AutoTokenizer, AutoModel >>> model = AutoModel.from_pretrained("internlm/internlm-chat-7b", trust_remote_code=True, device='cuda') Here, the "trust_remote_code=True" means "download the model code from huggingface repo 'internlm/internlm-chat-7b'", along…

I've only used llama via llama.cpp.

In general I think the python ML stuff is a mess. But I still won't execute code that recommend me to trust arbitrary remote code as the remote code can change at any time, it would be better to wait with the release until it was published to the transformers library or just include it in a clonable repo without the trust_remote_code flag.

It is much better to just be able to clone the code and have it locally so you can verify it once and not trust that it won't download any new code suddenly that you haven't been able to look at.

trust_remote_code means you have no control really, cloning a repo means you control when new code is added yourself.

Post reply on HN