21–30 of 47 posts
Picking apart Devin based solely on the demo video while ignoring all of the primary source testimonials on Twitter as to Devin's effectiveness seems somewhat intellectually dishonest... A demo video will of course cherrypick impressive-looking moments, even if they're not really.
Faking in demo video is now an essential evil to get virality . Because social media is about virality . All commenting on the video and this thread are now curious to try Devin, to prove it works or to prove it doesnt . So now if it works faking helped it get virality , more users , more demand for product . If it doesnt work good enough still it will be good enough for some of the users who discovered it because of…
Attention is all you need and faking it in demo gets it . Even if you deliver a decent enough product it will sell now..
Picking apart Devin based solely on the demo video while ignoring all of the primary source testimonials on Twitter as to Devin's effectiveness seems somewhat intellectually dishonest... A demo video will of course cherrypick impressive-looking moments, even if they're not really.
Great video. As a counterpoint to the video author's claims, it's worth pointing out that Devin doesn't have to be anywhere close to as fast as a human software engineer to be useful. Even if it turns an hour task to a day long task, it's still going to cost a fraction of what it would to pay the engineer, so that bar it must meet is quite low.
Earlier quoted context omitted.
In other words they managed to successfully recreate most people's experience on upwork.
In other words they managed to fake it until they make! Like most visionaries in silicon valley, lie now, tweet about it, prompt it through fake influencers with their mouth open on YouTube, get that VC money without any due diligence, hire smart people and force them to do it!
Picking apart Devin based solely on the demo video while ignoring all of the primary source testimonials on Twitter as to Devin's effectiveness seems somewhat intellectually dishonest... A demo video will of course cherrypick impressive-looking moments, even if they're not really.
Hence I’m skeptical of people making claims about a product I can’t try out myself. It’s unclear if the tasks they are doing and the way they are using Agents is relevant to the work I do. Which is usually working on a team of engineers shipping code on a complex code base.
For AI I tend to put a lot more weight in benchmarks, such as SWE-bench, which is why I wrote an article about:
https://www.stepchange.work/blog/why-do-ai-software-engineer...
SWE-bench is mostly small python tasks evaluated solely by unit tests which require less than 15 line changes to a single file. Most of those it fails at and the ones it gets right it ignores all sorts of libraries and conventions used in the rest of the code base.
I’m Optimistic that agents will eventually agents will improve dramatically in a few years but today Devin is not good at making larger changes that build on one another like features.
Faking in demo video is now an essential evil to get virality . Because social media is about virality . All commenting on the video and this thread are now curious to try Devin, to prove it works or to prove it doesnt . So now if it works faking helped it get virality , more users , more demand for product . If it doesnt work good enough still it will be good enough for some of the users who discovered it because of…
Im not curious to try Devin. LLM agents suck and have sucked since the first faked demo 2 years ago. I’ll leave it to the real labs (OA, Anthropic, DeepMind, FAIR) to make real advances.
That's really all I require, just show me an agentic workflow that doesn't routinely implode at various stages and I'll buy in to some of the broader claims about the future of agents.