11–19 of 19 posts
Now do it without those pre-written tests. Spec only. Else, the writers of those tests deserve a LOT of credit.
That's probably the most important thing, actually. I've tried my hardest to get Claude to build an APL VM using only the spec and it's virtually impossible to get full compliance as it takes too many shortcuts and makes too many assumptions. That's part of the challenge though, to see how far the daffy robots have come.
It is pretty incredible to me that in the pre-LLM/agent coding world, creating a new high-quality JS engine or browser seemed like it would likely never happen again. But now, any large company could build one if they wanted to. In a single digit number of months no less.
There's many JS implementations out there. Quality kind depends on what you need, and there's some engines more or less complete in which quirks are supported. And for example, v8 doesn't make much sense in embedded contexts
Earlier quoted context omitted.
There's many JS implementations out there. Quality kind depends on what you need, and there's some engines more or less complete in which quirks are supported. And for example, v8 doesn't make much sense in embedded contexts
There are definitely plenty of other JS engines, but they aren't always up to date on newer JS features. I'm pretty sure this is the 3rd JS engine to fully support the Temporal API (even JSC hasn't shipped it yet).
Now do it without those pre-written tests. Spec only. Else, the writers of those tests deserve a LOT of credit.
Now do it without those pre-written tests. Spec only. Else, the writers of those tests deserve a LOT of credit.
> Now do it without those pre-written tests That's probably the most important thing, actually. I've tried my hardest to get Claude to build an APL VM using only the spec and it's virtually impossible to get full compliance as it takes too many shortcuts and makes too many assumptions. That's part of the challenge though, to see how far the daffy robots have come.
I gave it the proper compile flags, I gave it test cases and their expected output, and everything it would have needed. The test cases were specifically hand picked to be hard on the search algorithm. And the base program was correct and gave the correct results (I was only adding an optimization), and were what I was using as a baseline for testing my implementation. You know, with a debugger and breakpoints, printfs and all that.
In the end it couldn't get the thing to work (I asked it to compile and verify) then it proudly declared that in all of the test cases I gave it, everything was solved through constraint propagation and the search didn't even trigger. So it didn't introduce any bugs. It tried to gaslight me. Even though it got a segfault in the new code it added (which would obviously not have been triggered if the search didn't actually execute)
It is pretty incredible to me that in the pre-LLM/agent coding world, creating a new high-quality JS engine or browser seemed like it would likely never happen again. But now, any large company could build one if they wanted to. In a single digit number of months no less.
Now do it without those pre-written tests. Spec only. Else, the writers of those tests deserve a LOT of credit.
You can prompt an LLM to generate tests from the spec and I'd bet it would easily get most of the way there, especially if you give it a reference implementation to test against. I did just that, though on a small scale - just for feature tests. The last few percent would be the real challenge, you probably don't want it to just imitate another implementation's bugs.
Earlier quoted context omitted.
You can prompt an LLM to generate tests from the spec and I'd bet it would easily get most of the way there, especially if you give it a reference implementation to test against. I did just that, though on a small scale - just for feature tests. The last few percent would be the real challenge, you probably don't want it to just imitate another implementation's bugs.
Reference implementation that someone else — a human, wrote? Hm… so one way or another, some humans’ labour is laundered…
Earlier quoted context omitted.
Reference implementation that someone else — a human, wrote? Hm… so one way or another, some humans’ labour is laundered…
Don't we all stand on the shoulders of giants?