31–40 of 54 posts
Here's our "Sora's Annotated Diffusion Transformer" writeup (code+paper side-by-side) Link: https://leetarxiv.substack.com/p/the-annotated-diffusion-tra...
There is an enormous gulf between research in general and the people who should be reading it from a professional point of view. Science communication is really broken and what makes the trade press or press generally is largely about whether a papers authors manage to write a good press release and effectively writes an article themselves. We need more New Scientist type magazine like things that do decent round ups…
Science journalism has gotten a lot harder over the years simply due to how fragmented ("salami sliced" [1] as it is sometimes called) so much research is now. "Publish or perish" encourages researchers to break up a single, coherent body of research into many smaller papers ("minimum publishable units") rather than publishing a larger and easy-to-follow paper. I find it to be one of the most annoying current practices in scientific publishing, because it makes it difficult to see the bigger picture even if you are a subject matter expert. It is hard to find every piece of the research given how split up it becomes, though forward and backward citation analysis helps. That only gets worse when trying to summarize the research from a less technical perspective as science journalists do.
[1] https://en.wikipedia.org/wiki/Salami_slicing_tactics#Salami_...
It would be great to see an updated edition.
Do you know a better source of information?
Based on this and other articles (and on experience), it's an especially underutilized resource. By reading it, you would gain an advantage over competition. Why aren't you using this advantage that is there for the taking?
And why don't we see papers posted to HN?
For the audience here, the opposite side of the coin is more relevant: Why don't you read software research? Based on this and other articles (and on experience), it's an especially underutilized resource. By reading it, you would gain an advantage over competition. Why aren't you using this advantage that is there for the taking? And why don't we see papers posted to HN?
For the audience here, the opposite side of the coin is more relevant: Why don't you read software research? Based on this and other articles (and on experience), it's an especially underutilized resource. By reading it, you would gain an advantage over competition. Why aren't you using this advantage that is there for the taking? And why don't we see papers posted to HN?
I'm sure there are sub-fields where academic papers are more important -- AI research, or really anything with "research" in the name -- but if you're just building normal software, I don't think there's much there.
For the audience here, the opposite side of the coin is more relevant: Why don't you read software research? Based on this and other articles (and on experience), it's an especially underutilized resource. By reading it, you would gain an advantage over competition. Why aren't you using this advantage that is there for the taking? And why don't we see papers posted to HN?
Because it's usually not that useful. I have a friend who was software-adjacent, and would post all these exciting studies showing that this or that practice was a big deal and massively boosted productivity. And without fail, those studies were some toy experiment design that had nothing to do with actual real-world conditions, and weren't remotely strong enough to convince me to up-end opinions based on my actual e…
> those studies were some toy experiment design that had nothing to do with actual real-world conditions
Isn't that the nature of understanding and applying science? Science is not engineering: Science discovers new knowledge. Applying that knowledge to the real world is engineering.
Perhaps overcoming that barrier, to some degree, is worthwhile. In a sense, it's a well-known gap.
Earlier quoted context omitted.
Because it's usually not that useful. I have a friend who was software-adjacent, and would post all these exciting studies showing that this or that practice was a big deal and massively boosted productivity. And without fail, those studies were some toy experiment design that had nothing to do with actual real-world conditions, and weren't remotely strong enough to convince me to up-end opinions based on my actual e…
Thanks for your POV. > those studies were some toy experiment design that had nothing to do with actual real-world conditions Isn't that the nature of understanding and applying science? Science is not engineering: Science discovers new knowledge. Applying that knowledge to the real world is engineering. Perhaps overcoming that barrier, to some degree, is worthwhile. In a sense, it's a well-known gap.
Shameless plug. I run LeetArxiv. It's a successor to papers with code built on the thesis "Every research paper should be a brief blog post with relevant code". Here's our "Sora's Annotated Diffusion Transformer" writeup (code+paper side-by-side) Link: https://leetarxiv.substack.com/p/the-annotated-diffusion-tra...
For the audience here, the opposite side of the coin is more relevant: Why don't you read software research? Based on this and other articles (and on experience), it's an especially underutilized resource. By reading it, you would gain an advantage over competition. Why aren't you using this advantage that is there for the taking? And why don't we see papers posted to HN?
This was their methodology:
> we recruited 16 experienced developers from large open-source repositories (averaging 22k+ stars and 1M+ lines of code) that they’ve contributed to for multiple years. Developers provide lists of real issues (246 total) that would be valuable to the repository—bug fixes, features, and refactors that would normally be part of their regular work. Then, we randomly assign each issue to either allow or disallow use of AI while working on the issue.
Now consider the question of whether you expect this research to generalize. Do you expect that if you / your friends / your coworkers started using AI tools (or stopped using AI tools) that the difference in productivity would also be 19%? Of course not! They didn't look at enough people or contexts to get two sig figs of precision on that average, nor enough to expect the conclusion to generalize. Plus the AI tools are constantly changing, so even if the study was nailing the average productivity change it would be wrong a few months later. Plus the time period wasn't long enough for the people to build expertise, and "if I spend time getting good at this will it be worth it" is probably the real question we want answered. The study is so weak that I don't even feel compelled to trust the sign of their result to be predictive. And I would be saying the same thing if it reported 19% higher instead of 19% lower.
I don't want to be too harsh on the study authors; I have a hard time imagining any way to do better given resource constraints and real world practicalities... but that's kind of the whole problem with such studies. They're too small and too specific and that's really hard to fix. Honestly I think I'd trust five anecdotes at lunch more than most software studies (mainly because the anecdotes have the huge advantage of being from the same context I work in). Contrast with medical studies where I'd trust the studies over the anecdotes, because for all their flaws at least they actually put in the necessary resources.
To be pithy: maybe we upvote Carmack quotes more than software studies because Carmack quotes are informed by more written code than most software studies.
[1]: https://metr.org/blog/2025-07-10-early-2025-ai-experienced-o...