Until I see an AI sysadmin that can help with basic configure/make problems, I don't have high hopes for an AI developer.
need AI for ffmpeg flags
I made a program for all your Unix admin needs:
What we learned in 6 months of working on an AI Developer
21–30 of 51 posts
Re: What we learned in 6 months of working on an AI Developer
#22Until I see an AI sysadmin that can help with basic configure/make problems, I don't have high hopes for an AI developer.
That should be quite easy compared to software development, which is much more open-ended since the requirement are usually more nebulous, potentially contradictory, and at times simply wrong.
Yet, I haven't seen an AI that solves the software distribution problem. Tools like ChatGPT are often plain wrong when answering questions about basic sysadmin problems, and even make up commands.
Re: What we learned in 6 months of working on an AI Developer
#23Earlier quoted context omitted.
It is a very important bit and might be how we all code in the future.
It’s definitely not even close to bring solved either. I haven’t seen a single code generator that works (100% of the time) for anything more than a very simple one or two liner.
So far, this involved changing the input or patching the code generator. Now we can simply nudge the code generator toward the fix. And we circle back to the point TA is making: just like you can't expect a junior programmer to get it right the first time, also the code generator needs feedback. And since the code generator is not restrained by a lack of technical knowledge like a junior programmer might be, it is even more important to supervise it.
Re: What we learned in 6 months of working on an AI Developer
#24Earlier quoted context omitted.
> Pretty soon GPT-4 will not be the best in the field. The next generation will perform much better. What makes you believe that progress is linear, or at least a line forever going up? I keep seeing people predicting rapidly improving AI, based on how rapid it improved over the last x months. But why is that not an outlier? How do we know we haven't hit a ceiling and stagnating? Isn't progress typically very bumpy a…
I would say that Microsoft/OpenAI’s attacks on open source, whether it be through “AGI safety” BS as a front for regulating their way to a monopoly, or attempting Embrace, Exentend, Extinguish on companies like Mistral, and Cold War-style fear mongering about China, are the greatest near-term risks to linear progress. And it’s worth noting on that latter point that China is not similarly constrained and so could end…
It’d be a bit hypocritical on the part of Microsoft/ClosedAI if they try to restrict AI tech in the name of safety while stepping around the fact that they themselves built it while essentially ignoring copyright.
Their commercial LLMs would not be possible without original creators who are now being ripped off and squeezed out of their jobs; if they don’t keep the tech open it will be very difficult to justify.
Re: What we learned in 6 months of working on an AI Developer
#25Earlier quoted context omitted.
That should be quite easy compared to software development, which is much more open-ended since the requirement are usually more nebulous, potentially contradictory, and at times simply wrong.
Yet, I haven't seen an AI that solves the software distribution problem. Tools like ChatGPT are often plain wrong when answering questions about basic sysadmin problems, and even make up commands.
The software distribution problem is not a close-ended, technical task that we could realistically expect an LLM to have an answer for. At least not now. The latter problem could eventually be lessened by fine-tuning the LLM more specifically and by improving the tooling around it. For example, by automatically briefing it about peculiarities of the operating systems.
Re: What we learned in 6 months of working on an AI Developer
#26Earlier quoted context omitted.
> Pretty soon GPT-4 will not be the best in the field. The next generation will perform much better. What makes you believe that progress is linear, or at least a line forever going up? I keep seeing people predicting rapidly improving AI, based on how rapid it improved over the last x months. But why is that not an outlier? How do we know we haven't hit a ceiling and stagnating? Isn't progress typically very bumpy a…
I would say that Microsoft/OpenAI’s attacks on open source, whether it be through “AGI safety” BS as a front for regulating their way to a monopoly, or attempting Embrace, Exentend, Extinguish on companies like Mistral, and Cold War-style fear mongering about China, are the greatest near-term risks to linear progress. And it’s worth noting on that latter point that China is not similarly constrained and so could end…
> China is not similarly constrained and so could end up outcompeting the U.S.
With what technology ?
US has long term export controls on China and as they have demonstrated with Russia recently once you have secondary sanctions in place everyone falls into line. So it's pretty likely they will be effective.
Re: What we learned in 6 months of working on an AI Developer
#27Earlier quoted context omitted.
I would say that Microsoft/OpenAI’s attacks on open source, whether it be through “AGI safety” BS as a front for regulating their way to a monopoly, or attempting Embrace, Exentend, Extinguish on companies like Mistral, and Cold War-style fear mongering about China, are the greatest near-term risks to linear progress. And it’s worth noting on that latter point that China is not similarly constrained and so could end…
> China is not similarly constrained and so could end up outcompeting the U.S. With what technology ? US has long term export controls on China and as they have demonstrated with Russia recently once you have secondary sanctions in place everyone falls into line. So it's pretty likely they will be effective.
China actually has very advanced technology in literally every area of the sciences now. They just don’t bother talking about it in english
I recommend you follow some of the specialist chinese AI substacks to see what’s happening over there.
Esp around chip building
Re: What we learned in 6 months of working on an AI Developer
#28Even though I don't think GPT-4 is up to the task, it does seem like now is the right time to be working on these things. Pretty soon GPT-4 will not be the best in the field. The next generation will perform much better. Possibly the most frustrating thing I find about GPT-4 is how close it gets with it's wrong answers. It's easy to dismiss a lesser answer when it responds with a laughably out-of-band idea. GPT-4 oft…
Oh man. When it’s so close but wrong it’s amazing for creative endeavors! For technical ones, it is quite a bad thing. It’s like being a Star Wars fan but the AI just wants to talk about Star Trek. I think this is why the non-tech people see AI as so amazing. For anything human and non-technical, the “almost but not quite” nature is a good thing. I was using an AI to help me debug a weird thing (mainly summarizing lo…
I think it's equally shit at creative work too, that's just harder to dismiss as "wrong". It's still wrong, it's just harder to see.
Re: What we learned in 6 months of working on an AI Developer
#29Earlier quoted context omitted.
I would say that Microsoft/OpenAI’s attacks on open source, whether it be through “AGI safety” BS as a front for regulating their way to a monopoly, or attempting Embrace, Exentend, Extinguish on companies like Mistral, and Cold War-style fear mongering about China, are the greatest near-term risks to linear progress. And it’s worth noting on that latter point that China is not similarly constrained and so could end…
> China is not similarly constrained and so could end up outcompeting the U.S. With what technology ? US has long term export controls on China and as they have demonstrated with Russia recently once you have secondary sanctions in place everyone falls into line. So it's pretty likely they will be effective.
I’m guessing you don’t read many machine learning papers. Chinese research in AI coming out of their top institutions (e.g. Tsinghua University, Shanghai AI Lab), is pretty much on par with state of the art in U.S. labs.
Re: What we learned in 6 months of working on an AI Developer
#30Until I see an AI sysadmin that can help with basic configure/make problems, I don't have high hopes for an AI developer.
ChatGPT-4 is wonderful for composing regular expressions. Saves so much time when transforming various Java strings that arise in my work.
Today's big time savings came from this prompt: "Write a Java method that uses the Eclipse AST parser to create a simple markdown file showing the commented method signatures of a given Java class text file."