Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mistral AI Releases Forge

mistral.ai

Post reply on HN
Print view View original

191–200 of 210 posts

Re: Mistral AI Releases Forge

#191

Post by sisve »

Aerroon wrote: post #121
sisve wrote: post #106

Earlier quoted context omitted.

My _feeling_ is that a lot of EU/European politicians has talked a lot more about the need to be independent from the US after Trump threaten Greenland. At least in the nordic countries. Not only concerning data & privacy, but defence, communications, space etc. All areas. The wheel has started to turn. You will not see it if you look around. But in 10 years time, maybe more, Europe will have stopped depending on the…

The politicians can talk, but they needed to set up an environment that would've let a European company have a decent shot at competing with the best AI models. But they didn't. Should've thought of that before being proud of setting up those strict tech regulations.

That is not how EU does things. If you want no regulation and access to capital you should go to the US.

AI will take over a lot and the biggest AI company will be in US and China. But there will be room for Europe also on the top 10 list.

But there will be an environment that is creating sovereignty from US much more the before. We have learned our lesson

Re: Mistral AI Releases Forge

#192

Post by andai »

zby wrote: post #77
mhl47 wrote: post #60

Earlier quoted context omitted.

"External Storage" whatever that is can not be the same as continous learning as it does not have the strong connections/capture the interdepencies of knowledge. That said I think we will see more efforts also on the business side to have models that can help you build a knowledge base in some kind of standardized way that the model is trained to read. Or synthesize some sort on instructions how to navigate your know…

The key to make Copilot useful is to take the limited context problem seriously enough. There are many dimensions to it: https://zby.github.io/commonplace/notes/context-efficiency-i... and it should be the starting point for designing the systems that extensively use llms.

Well, did you solve the problem of deciding what to remember? (And I suppose, how to retrieve it? i.e. LLM can retrieve info, but if it doesn't know about something because it hasn't been retrieved yet...)

Re: Mistral AI Releases Forge

#193

Post by upghost »

qntty wrote: post #126
upghost wrote: post #28

> Pre-training allows organizations to build domain-aware models by learning from large internal datasets. > Post-training methods allow teams to refine model behavior for specific tasks and environments. How do you suppose this works? They say "pretraining" but I'm certain that the amount of clean data available in proper dataset format is not nearly enough to make a "foundation model". Do you suppose what they are…

Pre-training mean exposing an already-trained model to more raw text like PDF extracts etc (aka continued pre-training). You wouldn't be starting from scratch, but it's still pre-training because the objective is just next token prediction of the text you expose it to. Post-training means everything else: SFT, DPO, RL, etc. Anything that involves things like prompt/response pairs, reward models, or benefits from huma…

Yeah, this checks out. I wonder what they are doing to prevent semantic collapse. Also, I wonder if the base model would already be instruct and RLHF tuned or only pre-trained. Trying to do additional training without semantic collapse in a way that is meaningful would be interesting to understand. Presumably they are using adapters but I've never had much luck in stacking adapters.

i.e.:

1. Do I start with an RLHF tuned model, "pretrain" on top of that (with adapter or by freezing weights?), then SFT on top of that (stack another adapter, or add layer(s) and freeze weights?) (and where did I get the dataset? synthetic extraction from corpus?), then RL (adapter, add layer(s) and freeze?)

2. or do I start at SF tuned model, ...

3. or do I start at raw pre-trained model, ...

Would love to know what the matrix used was.

Re: Mistral AI Releases Forge

#194

Post by menaerus »

charcircuit wrote: post #168
menaerus wrote: post #85

Earlier quoted context omitted.

Sorry but who mentioned coding as a use-case? My comment was general and not specific to the coding use-case, and I don't understand where did you get the idea from that I am arguing that building a similarity search engine would be a substitute to the symbol-search engine or that symbol-search is inferior to the similarity-search? Please don't put words into my mouth. My question was genuine without making any presu…

I mentioned coding as a use case in my comment you replied to. You were asking for an example for when one wouldn't use vector search and I provided one. I did not say similarity search would be a substitute. I said that for the coding case you do not need it. >you would still likely want to build a similarity search engine In practice tools like Claude Code, Codex, Gemini, Kimi Code, etc are getting away with search…

In your point you said "There is more to searching than building a basic similarity search." which assumed and implied all kinds of things and which was completely unnecessary.

> In practice tools like Claude Code, Codex, Gemini, Kimi Code, etc are getting away with searching for code with grep / find and understanding code by loading a sufficient amount of code into the context window

Getting away is the formulation I would use as well. "Sufficient amount" OTOH is arguable and subjective. What suffices in one usage example, it does not in another, so the perception of how sufficient it really is depends on the usage patterns, e.g. type and size of the codebases and actual queries asked.

The crux of the problem is what amount and what parts of the codebase do you want to load into the context while not blowing up the context and while still maintaining the capability of the model to be able to reason about the codebase correctly.

And I find it hard to argue that building the vector database would not help exactly in that problem.

Re: Mistral AI Releases Forge

#195

Post by rldjbpin »

ogou wrote: post #40

Don't sleep on Mistral. Highly underrated as a general service LLM. Cheaper, too. Their emphasis on bespoke modelling over generalized megaliths will pay off. There are all kinds of specialized datasets and restricted access stores that can benefit from their approach. Especially in highly regulated EU. Not everyone is obsessed with code generation. There is a whole world out there.

> Not everyone is obsessed with code generation.

They (still) are. https://news.ycombinator.com/item?id=47404796

Re: Mistral AI Releases Forge

#196

Post by philipallstar »

hermanzegerman wrote: post #183
wqaatwt wrote: post #173

Earlier quoted context omitted.

Well yes, but because there are approximately zero EU tech companies that can be affected by these fines and regulations there is very little political pushback against them. In a certain sense it’s a way for EU to clawback at least a small slice of all that money flowing to the US.

Why should there be pushback against antitrust measures? It's what keeps markets alive

Well, not necessarily; lots of things keep markets alive, including making it easier for people to start companies. But that aside, it's the selective enforcement of antitrust measures that's my point.

Re: Mistral AI Releases Forge

#197

Post by hermanzegerman »

prmoustache wrote: post #184
hermanzegerman wrote: post #181

Earlier quoted context omitted.

Yes the farmers are a very vocal and powerful minority. They get more than 50% of their income from subsidies, are quite well off, but always find a reason to complain. I was thinking more about stuff like "Buy American"-Regulations for public tenders. Stuff like that doesn't exist here

Well I can certainly understand them. Based on price tgey would not be able to compete and have half decent living wages so protectionism AND subsidies is a decent strategy to maintain local production which I feel allow a country / area to not lose a lever in international negociations.

The quotas for these products in the trade deal would have been very limited, and the opportunities for our industry far greater.

It was a small vocal minority that avoided benefit for the majority

Re: Mistral AI Releases Forge

#198

Post by hermanzegerman »

philipallstar wrote: post #196
hermanzegerman wrote: post #183

Earlier quoted context omitted.

Why should there be pushback against antitrust measures? It's what keeps markets alive

Well, not necessarily; lots of things keep markets alive, including making it easier for people to start companies. But that aside, it's the selective enforcement of antitrust measures that's my point.

Where is the selective enforcement?

They also enforce antitrust in other sectors than big tech. You just don't read that much about it in the news

Re: Mistral AI Releases Forge

#199

Post by brabel »

prmoustache wrote: post #184
hermanzegerman wrote: post #181

Earlier quoted context omitted.

Yes the farmers are a very vocal and powerful minority. They get more than 50% of their income from subsidies, are quite well off, but always find a reason to complain. I was thinking more about stuff like "Buy American"-Regulations for public tenders. Stuff like that doesn't exist here

Well I can certainly understand them. Based on price tgey would not be able to compete and have half decent living wages so protectionism AND subsidies is a decent strategy to maintain local production which I feel allow a country / area to not lose a lever in international negociations.

Yes it's great for the EU, but not so great for the Mercosur, is it? The EU wants them to buy products from the EU, but if the Mercosur acted the same way as the EU does with regards to agriculture, would mean they would also "protect" their industries from "unfair EU competition" and no deal can be made at all.

If you want to have commerce, you need to give some to get some. Without agriculture being included, the EU would just take and not give anything. It used to work in the colonial times, but one would hope that is behind us now.

Re: Mistral AI Releases Forge

#200

Post by zby »

andai wrote: post #192
zby wrote: post #77

Earlier quoted context omitted.

The key to make Copilot useful is to take the limited context problem seriously enough. There are many dimensions to it: https://zby.github.io/commonplace/notes/context-efficiency-i... and it should be the starting point for designing the systems that extensively use llms.

Well, did you solve the problem of deciding what to remember? (And I suppose, how to retrieve it? i.e. LLM can retrieve info, but if it doesn't know about something because it hasn't been retrieved yet...)

There are many techniques - I analyse them at my knowledge base.

I wrote 'my bet is that solving ...' - if I have solved it I would not have to hedge like this.

Post reply on HN