> We trained on 320M total tokens, which is This isn't training on top of existing weights from Llama-3, it's training using their own long context data, and it such a tiny set I wondering how strong its reasoning capability is.
[deleted]
Gradient AI Releases 1M Context Llama 3 8B
11–20 of 32 posts
Re: Gradient AI Releases 1M Context Llama 3 8B
#12[deleted]
Re: Gradient AI Releases 1M Context Llama 3 8B
#13Mike @ Crusoe AI (crusoe.ai/cloud) here: we're a compute sponsor for this effort and have been working with other fine-tunes (e.g. https://x.com/erhartford/status/1783273948022755770 , https://x.com/winglian/status/1778777261568606703 ). If you're pushing the boundaries of AI/ML and need compute to help get there, shoot us an email at sponsorship at crusoecloud dot com and we'd love to chat!
All (training / evals / inference) performed on their L40s clusters. These machines are underrated but capable of serious work
Re: Gradient AI Releases 1M Context Llama 3 8B
#14> We trained on 320M total tokens, which is This isn't training on top of existing weights from Llama-3, it's training using their own long context data, and it such a tiny set I wondering how strong its reasoning capability is.
The table at the bottom says they initialized the 65K version from "LLaMA-3 7B"? (Assuming the 7B is a typo and they meant 8B.)
And each successive version with a larger window was initialized on the previous smaller one (65K -> 262K -> 524k -> 1048k).
Re: Gradient AI Releases 1M Context Llama 3 8B
#15Can someone test this with ruler please? https://github.com/hsiehjackson/RULER In practice all of these long contexts show degraded performance (there's a table on the repo). For my NLP work I find that GPT-4-turbo is much worse after 32k-ish.
Hi, Leo, chief scientist @ Gradient, here. We've been eagerly awaiting the release of RULER's code ourselves! As mentioned below, we wanted to release a model to the community asap, and have plans already for further fine-tuning & more sophisticated evals.
If you have other suggestions, I'd be happy to chat further.
Re: Gradient AI Releases 1M Context Llama 3 8B
#16> We trained on 320M total tokens, which is This isn't training on top of existing weights from Llama-3, it's training using their own long context data, and it such a tiny set I wondering how strong its reasoning capability is.
The table at the bottom says they initialized the 65K version from "LLaMA-3 7B"? (Assuming the 7B is a typo and they meant 8B.) And each successive version with a larger window was initialized on the previous smaller one (65K -> 262K -> 524k -> 1048k).
Right. We are sleep deprived -- couldn't stop over the weekend. Please forgive the typos
Re: Gradient AI Releases 1M Context Llama 3 8B
#17Will you also do a 70B finetune? Thanks.
Re: Gradient AI Releases 1M Context Llama 3 8B
#18Can someone test this with ruler please? https://github.com/hsiehjackson/RULER In practice all of these long contexts show degraded performance (there's a table on the repo). For my NLP work I find that GPT-4-turbo is much worse after 32k-ish.
Hi, Leo, chief scientist @ Gradient, here. We've been eagerly awaiting the release of RULER's code ourselves! As mentioned below, we wanted to release a model to the community asap, and have plans already for further fine-tuning & more sophisticated evals. If you have other suggestions, I'd be happy to chat further.
Hi!
Unless I'm missing something, they did add the eval scripts to that repo 4 days ago.
Re: Gradient AI Releases 1M Context Llama 3 8B
#19Pretty cool, but no one can run 1M context at home.
Re: Gradient AI Releases 1M Context Llama 3 8B
#20Wow there was already a 256k version (dolphin). 1M is insane. Be aware you need a lot of memory though
With 144gb of GPU memory, The most I can load for llama3 is 232k.