131–140 of 366 posts
It’s been a relief to find a platform where I can ask questions without the fear of being humiliated Half joking, but I am pretty tired of SO pedantry.
Earlier quoted context omitted.
"LLMs can train on the official documentation of tools l/libraries but they can't experiment and figure out solutions to weird problems" LLMs train on way more than just the official documentation: they train on the code itself, the unit tests for that code (which, for well written projects, cover all sorts of undocumented edge-based) and - for popular projects - thousands of examples of that library being used (and…
> they train on the code itself, the unit tests for that code If understanding the code was enough, we wouldn't have any bugs or counterintuitive behaviors. > and - for popular projects - thousands of examples of that library being used (and unit tested) "in the wild". If people stopped contributing to forums, we won't have any such data for new things that are being made.
I would argue that code in GitHub is much more useful, because it's presented in the context of a larger application and is also more likely to work.
The problem is eventually what are LLMs going’s to draw from? They’re not creating new information, just regurgitating and combining existing info. That’s why they perform so poorly on code for which there aren’t many many publicly available samples, SO/reddit answers etc.
Published documentation.
I'm going to make up a number but I'll defend it: 90% of the information content of stackoverflow is regurgitated from some manual somewhere. The problem is that the specific information you're looking for in the relevant documentation is often hard to find, and even when found is often hard to read. LLMs are fantastic at reading and understanding documentation.
It’s been a relief to find a platform where I can ask questions without the fear of being humiliated Half joking, but I am pretty tired of SO pedantry.
Earlier quoted context omitted.
Everyone has a pet theory about what’s wrong with SO; but here’s the truth: Whatever they’re doing, it isn’t working. Blame mods. Blame AI. Blame askers… whatever man. That is a sinking ship. If you don’t see people complain about SO, it’s because they aren’t using it, not because they’re using the search. Pretty hard to argue at this point that the problem is with the users being too shit to use the platform. That’s…
I get good answers all the time on SO or used to. My problem is that I've been down voted several times for "stupid question" and also been down voted for not knowing what I was talking about in an area I'm an expert in. I had one question that was a bit odd and went against testing dogma that I had a friend post. He pulled it 30 minutes later as he was already down 30 votes. It was a thing that's not best practice i…
The problem is eventually what are LLMs going’s to draw from? They’re not creating new information, just regurgitating and combining existing info. That’s why they perform so poorly on code for which there aren’t many many publicly available samples, SO/reddit answers etc.
Edit: OP had actually qualified their statement to refer to only underrepresented coding languages. That's 100% true - LLM coding performance is super biased in favor of well-represented languages, esp. in public repos. Interesting - I actually think they perform quite well on code, considering that code has a set of correct answers (unlike most other tasks we use LLMs for on a daily basis). GitHub Copilot had a 30%+…
The problem is eventually what are LLMs going’s to draw from? They’re not creating new information, just regurgitating and combining existing info. That’s why they perform so poorly on code for which there aren’t many many publicly available samples, SO/reddit answers etc.
The problem is eventually what are LLMs going’s to draw from? They’re not creating new information, just regurgitating and combining existing info. That’s why they perform so poorly on code for which there aren’t many many publicly available samples, SO/reddit answers etc.
I thought synthetic data is what is partially training the new multimodal large models, i.e. AlphaGeometry, o1, etc.
E.g. you can't just synthetically generate code, something or someone needs to run it and see if it performs the functions you actually asked of it.
You need to feed the LLM output into some kind of formal verification system, and only then add it back to the synthetic training dataset.
Here, for example - dumb recursive training causes model collapse:
It’s been a relief to find a platform where I can ask questions without the fear of being humiliated Half joking, but I am pretty tired of SO pedantry.
The flipside to this is you can’t get answers to anything _recent_, since the models are trained years behind in content. My feelig is it’s getting increasingly difficult to figure out issues on the latest version of libraries & tools, as the only options are private Discords (which aren’t even googleable)
Earlier quoted context omitted.
I haven't really found stackoverflow to be that humiliating (compared to some IRC rooms or forums), basic questions get asked and answered all the time. But the worst part is when you want to do something off the beaten path. Q: how do I do thing X in C? A: Why do you need to know this? The C standard doesn't say anything about X. The answer will depend on your compiler and platform. Are you sure you want to do X ins…
The first time that I asked a question on #cpp @Freenode was a unique experience for my younger self. My message contained greetings and the question in the same message. I was banned immediately and the response from the mods was: - do not greet; we don't have time for that bullshit - do not use natural language questions; submit a test case and we will understand what you mean through your code - do not abbreviate…
Now that Rust is succesfully assimilating those communities, I have noticed the same toxicity on less well moderated forums, like the subreddit. The Discord luckily is still great.
It's probably really important to separate the curmudgeons from the fresh initiates to provide an enjoyable and positive experience for both groups. Discord makes that really easy.
In the Ruby IRC channel curmudgeons would simply be shot down instantly with MINASWAN style arguments. In the Haskell IRC channel I guess it was basically accepted that everyone was learning new things all the time, and there was always someone willing to teach at the level you were trying to learn.