Simulacrum of Knowledge Work
blog.happyfellow.dev
Simulacrum of Knowledge Work
1–10 of 97 posts
Re: Simulacrum of Knowledge Work
#2>We've automated ourselves into Goodhart's law.
Yes.
This does not however mean that progress is not being made.
It just means the progress is happening along such dimensions that are completely illegible in terms of the culture of the early XXI century Internet, which is to say in terms of the values of the society which produced it.
Re: Simulacrum of Knowledge Work
#3Everybody's output is someone else's input. When you generate quantity by using an LLM, the other person uses an LLM to parse it and generate their own output from their input. When the very last consumer of the product complains, no one can figure out which part went wrong.
Re: Simulacrum of Knowledge Work
#4>"is the RLHF judge happy with the answer."
Reinforcement Learning with Verifiable Rewards (RLVR) to improve math and coding success rates seems like an exception.
Re: Simulacrum of Knowledge Work
#5I don't really agree with the premise of the article. Sure proxy measures are everywhere. But for knowledge work specifically you can usually check real quality. Of course it's not as extremely easy as "oh this report contains a few spelling errors", but it is doable. If you accepted work purely based on superficial proxy measures you were not fairly evaluating work at all.
Re: Simulacrum of Knowledge Work
#6>We've automated ourselves into Goodhart's law. Yes. This does not however mean that progress is not being made. It just means the progress is happening along such dimensions that are completely illegible in terms of the culture of the early XXI century Internet, which is to say in terms of the values of the society which produced it.
Feels like a parallel with https://en.wikipedia.org/wiki/Constructivism_%28philosophy_o... where "it's not valid until you checked"
Re: Simulacrum of Knowledge Work
#7Everybody's output is someone else's input. When you generate quantity by using an LLM, the other person uses an LLM to parse it and generate their own output from their input. When the very last consumer of the product complains, no one can figure out which part went wrong.
Well the last consumer is holding it wrong of course. Why? The last consumer is present, and everyone else is behind 7 proxies.
Re: Simulacrum of Knowledge Work
#8>We've automated ourselves into Goodhart's law. Yes. This does not however mean that progress is not being made. It just means the progress is happening along such dimensions that are completely illegible in terms of the culture of the early XXI century Internet, which is to say in terms of the values of the society which produced it.
Feels like a parallel with https://en.wikipedia.org/wiki/Constructivism_%28philosophy_o... where "it's not valid until you checked"
I didn't see the connection initially.
Re: Simulacrum of Knowledge Work
#9I don't really agree with the premise of the article. Sure proxy measures are everywhere. But for knowledge work specifically you can usually check real quality. Of course it's not as extremely easy as "oh this report contains a few spelling errors", but it is doable. If you accepted work purely based on superficial proxy measures you were not fairly evaluating work at all.
I think there’s a weaker claim that holds true: we were able to ignore lots of content based on the superficial (and pay proper attention to work that passed this test) and now we are overwhelmed because everything meets the superficial criteria and we can’t pay proper attention to all of it.
Re: Simulacrum of Knowledge Work
#10The article asserts that the quality of human knowledge work was easier to judge based on proxy measures such as typos and errors, and that the lack of such "tells" in AI poses a problem.
I don't know if I agree with either assertion… I've seen plenty of human-generated knowledge work that was factually correct, well-formatted, and extremely low quality on a conceptual level.
And AI signatures are now easy for people to recognize. In fact, these turns of phrase aren't just recognizable—they're unmistakable. Having worked with corporate clients for 10 years, I don't view the pre-LLM era as a golden age of high-quality knowledge work. There was a lot of junk that I would also classify as a "working simulacrum of knowledge work."