The fine tuning endpoint is deprecated according to the API docs. Is this the replacement? https://docs.mistral.ai/api/endpoint/deprecated/fine-tuning
Interesting to see. I thought they were promoting fine tuning
Mistral AI Releases Forge
21–30 of 210 posts
Re: Mistral AI Releases Forge
#22How does this compare to fine tuning?
Re: Mistral AI Releases Forge
#23I am rooting for Mistral with their different approach: not really competing on the largest and advanced models, instead doing custom engineering for customers and generally serving the needs of EU customers.
their ocr model is goated
Re: Mistral AI Releases Forge
#24They mention pretraining too, which surprises me. I thought that was prohibitively expensive? It's feasible for small models but, I thought small models were not reliable for factual information?
Typical stages of training for these models are:
Foundational:
- Pretraining - Mid/post-training (SFT) - RLHF or alignment post-training (RL)
And sometimes...
- Some more customer-specific fine-tuning.
Note that any supervised fine-tuning following the Pretraining stage is just swapping the dataset and maybe tweaking some of the optimiser settings. Presumably they're talking about this kind of pre-RL fine-tuning instead of post-RL fine-tuning, and not about swapping out the Pretraining stage entirely.
Re: Mistral AI Releases Forge
#25[flagged]
Re: Mistral AI Releases Forge
#26Earlier quoted context omitted.
[flagged]
[flagged]
I have no interest in anything crypto, but they are making a proposal about NFTs tied to AI (LLMs and verifiable machine learning) so they can make ownership decisions.
So it'd be alive in the making decisions sense, not in a "the technology is thriving" sense.
Re: Mistral AI Releases Forge
#27The future of AI is specialization, not just achieving benevolent knowledge as fast as we can at the expense of everything and everyone along the way. I appreciate and applaud this approach. I am looking into a similar product myself. Good stuff.
Re: Mistral AI Releases Forge
#28> Pre-training allows organizations to build domain-aware models by learning from large internal datasets.
> Post-training methods allow teams to refine model behavior for specific tasks and environments.
How do you suppose this works? They say "pretraining" but I'm certain that the amount of clean data available in proper dataset format is not nearly enough to make a "foundation model". Do you suppose what they are calling "pretraining" is actually SFT and then "post-training" is ... more SFT?
There's no way they mean "start from scratch". Maybe they do something like generate a heckin bunch of synthetic data seeded from company data using one of their SOA models -- which is basically equivalent to low resolution distillation, I would imagine. Hmm.
Re: Mistral AI Releases Forge
#29> Pre-training allows organizations to build domain-aware models by learning from large internal datasets. > Post-training methods allow teams to refine model behavior for specific tasks and environments. How do you suppose this works? They say "pretraining" but I'm certain that the amount of clean data available in proper dataset format is not nearly enough to make a "foundation model". Do you suppose what they are…
I can imagine that, as usual, you start with a few examples and then instruct an LLM to synthesize more examples out of that, and train using that. Sounds horrible, but actually works fairly well in practice.
Re: Mistral AI Releases Forge
#30> Pre-training allows organizations to build domain-aware models by learning from large internal datasets. > Post-training methods allow teams to refine model behavior for specific tasks and environments. How do you suppose this works? They say "pretraining" but I'm certain that the amount of clean data available in proper dataset format is not nearly enough to make a "foundation model". Do you suppose what they are…
I think they are referring to “continued pretraining”.