What we learned in 6 months of working on an AI Developer
blog.pythagora.ai
What we learned in 6 months of working on an AI Developer
1–10 of 51 posts
Re: What we learned in 6 months of working on an AI Developer
#2> Our approach is to focus on building the application layer instead of working on getting LLMs to output better results. The reasoning is that LLMs will get better,...
So more jam tomorrow then. Building the framework around the magic is the easy bit.
Re: What we learned in 6 months of working on an AI Developer
#3> Our approach is to focus on building the application layer instead of working on getting LLMs to output better results. The reasoning is that LLMs will get better,... So more jam tomorrow then. Building the framework around the magic is the easy bit.
It is a very important bit and might be how we all code in the future.
Re: What we learned in 6 months of working on an AI Developer
#4The focus on upfront specs feels a bit off. Since it's apparently cheap to generate running code, as a user, I'd much rather be able to just iterate really fast and use output to refine my requirements rather than having to laboriously state them all up front. Agile rather than waterfall if you will.
Re: What we learned in 6 months of working on an AI Developer
#5> Our approach is to focus on building the application layer instead of working on getting LLMs to output better results. The reasoning is that LLMs will get better,... So more jam tomorrow then. Building the framework around the magic is the easy bit.
It is a very important bit and might be how we all code in the future.
It’s definitely not even close to bring solved either. I haven’t seen a single code generator that works (100% of the time) for anything more than a very simple one or two liner.
Re: What we learned in 6 months of working on an AI Developer
#6Until I see an AI sysadmin that can help with basic configure/make problems, I don't have high hopes for an AI developer.
Re: What we learned in 6 months of working on an AI Developer
#7Even though I don't think GPT-4 is up to the task, it does seem like now is the right time to be working on these things. Pretty soon GPT-4 will not be the best in the field. The next generation will perform much better.
Possibly the most frustrating thing I find about GPT-4 is how close it gets with it's wrong answers. It's easy to dismiss a lesser answer when it responds with a laughably out-of-band idea. GPT-4 often shows that it has a general idea of what you want but misses a small but critical aspect which results in a solution to something else that is similar but not what you wanted.
I have mixed results on iterating on it's own mistakes. It will too often try and change the world to match it's answer, rather than fixing the answer. The best approach I have found to stop this is by getting it to create unit tests. I imagine there is a lot of training data for it to understand the intention behind fixing a failing test. It's a very specific problem for it to look at and generally changing the test is not considered the correct solution.
Re: What we learned in 6 months of working on an AI Developer
#8Maybe AI developers can make landing pages and basic APIs. But, taking front end as an example, I just don't see how an AI can reproduce exact design specifications and interactivity to the point where it wouldn't just be faster to write the code yourself or search for some human verified snippet that does what you want.
And programmers who do know how to actually write efficient code without AI seem like they'd be even more in demand than those that rely on AI. Skill + knowledge + ability to use existing resources (e.g. StackOverflow, packages, templates), as we do now, are much more predictable and faster than trying to wrangle AI to do exactly what the designer or PM wants.
When the dishwasher was invented, everyone thought the human dish washer would be obsolete. And yet, restaurants still employ dish washers because they are much more efficient and thorough than a dishwashing machine.
Re: What we learned in 6 months of working on an AI Developer
#9Until I see an AI sysadmin that can help with basic configure/make problems, I don't have high hopes for an AI developer.
need AI for ffmpeg flags