Earlier quoted context omitted.
Have you tried `ss -ar`? You may have to install `ss`. It is standard on Linux.
No I was trying to see it could use tools/binaries that come with MacOS.
netstat is now considered too old to be used in new code.
Learning to Reason with LLMs
881–890 of 1001 posts
Re: Learning to Reason with LLMs
#882Student here. Can someone give me one reason why I should continue in software engineering that isn't denial and hopium?
Computer Science becomes MORE interesting as computers become more capable, not less. There are so many things we could be working on, but we still waste so much time on boring libraries, configuration, implementation details that we simply don't get to experiment enough.
Just like nobody programs on punch cards anymore, learning details of a specific technology without deeper understanding will become obsolete. But general knowledge about computer science will become more valuable.
Re: Learning to Reason with LLMs
#883This is a pretty big technical achievement, and I am excited to see this type of advancement in the field. However, I am very worried about the utility of this tool given that it (like all LLMs) is still prone to hallucination. Exactly who is it for? If you're enough of an expert to critically judge the output, you're probably just as well off doing the reasoning yourself. If you're not capable of evaluating the outp…
The utility I usually get from these kinds of tools so far is more like an extremely good reference or helper to something I could definitely figure out if given enough time. E.g. figuring out the best way to clean up a specific syntax error, setting up a class and some obvious base functions I'm going to need in it, helping me figure out where I might have went astray in solving a math problem.
The tools have not been at "and now I don't need code tests & review, mathematicians in society, or factbooks all because I have an LLM" level. While that's definitely a goal of AGI it's also definitely not my bar for weighing whether there is utility in a tool.
The alternative way to think about it: the value of a tool is in what you can figure out to do with it, not in whether it's perfect at doing something. On one extreme that means a dictionary can still be a useful spelling reference even if books have a rare typo. On the other extreme that means a coworker can still offer valuable insight into your code even if they make lots of coding errors and don't have an accurate understanding of everything there is to know about all of C++. Whether you get something out of either of these cases is a product of how much they can help you reach the accuracy you need to arrive at and the way you utilize the tool, not their accuracy alone. Usually I can get a lot out of a person who is really bad at one shot coding a perfect answer but feels like their answer seems right so I can get quite a bit out of an LLM that has the same problem. That might not be true for all types of questions though but that's fine, not all tools have utility in every problem.
Re: Learning to Reason with LLMs
#884What's the precedent set here?
Models that hide away their reasoning and only display the output, charging whatever tokens they'd like?
This is not a good release on any front.
Re: Learning to Reason with LLMs
#885If you’re using the API and are on tier 4, don’t bother adding more credits to move up to tier 5. I did this, and while my rate limits increased, the o1-preview / o1-mini model still wasn’t available.
Re: Learning to Reason with LLMs
#886Feels like a lot of commenters here miss the difference between just doing chain-of-thought prompting, and what is happening here, which is learning a good chain of thought strategy using reinforcement learning. "Through reinforcement learning, o1 learns to hone its chain of thought and refine the strategies it uses." When looking at the chain of thought (COT) in the examples, you can see that the model employs diffe…
It’s basically a scaled Tree of Thoughts
This seems most likely, with some special tokens thrown in to kick off different streams of thought.
Re: Learning to Reason with LLMs
#887This should also be good news for open weights models, right? Since OpenAI is basically saying "you can get very far with good prompts and some feedback loops".
No. It's bad news, because you can't see the rationale/search process that led to the final answer, just the final answer, and if training on the final answer were really that adequate, we wouldn't be here. It also is probably massively expensive compute-wise, much more so than simple unsupervised training on a corpus of question/answer pairs (because you have to generate the corpus by search first). It's also also bad news because reinforcement learning tends to be highly finicky and requires you to sweat the details and act like a professional, while open weight stuff tends to be produced by people for whom the phrase 'like herding cats' was coined, and so open source RL stuff is usually flakier than proprietary solutions (where it exists at all). They can do it for a few passion projects shared by many nerds, like chess or Go, but it takes a long time.
Re: Learning to Reason with LLMs
#888If you’re using the API and are on tier 4, don’t bother adding more credits to move up to tier 5. I did this, and while my rate limits increased, the o1-preview / o1-mini model still wasn’t available.
Keep us posted!
Re: Learning to Reason with LLMs
#889Just added o1 to https://double.bot if anyone would like to try it for coding. --- Some thoughts: * The performance is really good. I have a private set of questions I note down whenever gpt-4o/sonnet fails. o1 solved everything so far. * It really is quite slow * It's interesting that the chain of thought is hidden. This is I think the first time where OpenAI can improve their models without it being immediately dis…
Trying out Double now. o1 did a significantly better job converting a JavaScript file to TypeScript than Llama 3.1 405B, GitHub Copilot, and Claude 3.5. It even simplified my code a bit while retaining the same functionality. Very impressive. It was able to refactor a ~160 line file but I'm getting an infinite "thinking bubble" on a ~420 line file. Maybe something's timing out with the longer o1 response times?
> Maybe something's timing out with the longer o1 response times?
Let me look into this – one issue is that OpenAI doesn't expose a streaming endpoint via the API for o1 models. It's possible there's an HTTP timeout occurring in the stack. Thanks for the report
Re: Learning to Reason with LLMs
#890What is interesting to me is that there is no difference in the AP English lit/lang exams. Why did chain-of-thought produce negligible improvements in this area?
I would guess because there is not much problem-solving required in that domain. There’s less of a “right answer” to reason towards.
I think there may also be a lack of specification there. When you get more demanding and require more, the creative writing seems to be better. Like it does much better at things like sestinas. For all of those questions, there's probably a lot of unspecified criteria you could say makes an answer better or worse, but you don't, so the first solution appears adequate.