Designing AI for Disruptive Science

> In fact, relativity was only recently fully backed up with experimental data.

Can you elaborate on the assertion you made here? In addition to the important points @elbasti made about tests performed approximately a century ago, what does it even mean for a scientific theory to be "fully backed up"? Such theories can be tested and the tests either passed or the theory disproven but it's not possible to _prove_ such a theory. And to some extent we already know that relativity cannot be the final answer because it doesn't mesh well with quantum mechanics (which has been experimentally tested substantially, arguably even more than relativity has).

meh. I would be happier with this article if it demonstrated familiarity with the source material. "Del Rigor en la Ciencia" (On Exactitude in Science) was Borges (hilarious) investigation into Korzybski's General Semantics, a fact that was surprisingly absent from the text. Borges implied "the map is not the territory," but Korzybski actually came out in said it a decade or so before Borges wrote the story in question. Understanding the themes of Borges story is greatly informed by a passing familiarity with Korzybski's work. WorldCat tells me the 4th edition of "Science and Sanity" is widely held by libraries, and if you're interested in the assertions made in this article, you might enjoy reading (at least parts of) it.

https://search.worldcat.org/title/369632

The author completely missed the point Borges (and Korzybski) made about the utility of maps. Maps (according to both) are abstractions which allow the user to ignore irrelevant aspects of reality so other, more interesting facets come into sharper resolve. This might be why Beck's London Tube map is so well regarded. It allows the user to easily ignore aspects that are not germane to the task of deciding where and when to get on and off the tube.

But is a scientific paradigm like a map? Certainly it is an abstraction, if we take Kuhn's definition. If you're interested, I can recommend both "The Structure of Scientific Revolutions" and "The Essential Tension : Selected Studies in Scientific Tradition and Change" by Kuhn.

https://search.worldcat.org/title/4660423077

https://search.worldcat.org/title/3034084

Calling scientific paradigms maps isn't wrong, per se, but it does create more of a meta-metaphor, and a weak one at that.

Also. No. Maxwell did not replace a patchwork of equations with four short ones. That was Heaviside.

https://en.wikipedia.org/wiki/Oliver_Heaviside

Something we don't mention in polite society these days is that Maxwell proposed electromagnetic waves as propagating through an aether:

https://en.wikisource.org/wiki/A_Treatise_on_Electricity_and...

If you're going to talk about new paradigms, Maxwell is a great example, but his story is not complete without mentioning Heaviside, Michelson and Morley.

Also... I bristle at the phrase "Hypernormal Science." It's also introduced without definition or reference. Collins, et al describe it as distinct from (though seemingly related to) the word "Hypernormal" as coined by Yurchak in "Everything Was Forever Until It Was No More."

https://direct.mit.edu/posc/article-abstract/31/2/262/112751...

https://search.worldcat.org/title/1572419463

Or if you're short on time, you can get an entertaining (though not as enlightening) description from Adam Curtis' 2016 documentary HyperNormalization. You won't come away from it with a better understanding of AI, General Semantics or Popperian falsifiability, but it has a striking visual style and a very good soundtrack. And may lead to a better understanding of "hypernormal science."

https://youtu.be/u1Tp-ryQPFI

And getting back to the Michelson-Morley experiment. The author talks about how their results did not cause the scientific establishment to abandon the concept of luminiferous aether. Certainly there is conservatism in science. Gigging science-monkeys tend to want to see interesting results replicated.

And this was one of the issues with the MM experiment. It took a while to replicate. We're MUCH better at replicating it these days and I would guess that thousands (maybe hundreds) of physics undergrads did this very task last year. But we've had over a century of pedagogical experience w/ this experiment. We know how to structure it to get the results we want. This was not the case in the late 1800s and in fact, several early attempts to replicate the experiment suggested the existence of an aether which was drifting slowly towards Cleveland.

And what does it say that heat flow, fluid flow, diffusion and electrostatics share equations? Does it say there's something fundamental in reality? Or does it say there's something fundamental in the way we model reality?

That being said... I think the author has hit upon something here... people are often wary of evidence which contradicts experience, even when that evidence (and not experience) is more correct.

But each of the examples he provides glosses over the process by which new paradigms overrode the old.

I deeply appreciate the author avoiding slavish fealty to fashionable AI trends. He probably could have gone further to describe more representational weakness of ESM3 and GNoME.

I fear, however, he has missed the point. It's less interesting to describe the messy ways in which AI fails than to describe the messy ways in which humans succeed. The process by which paradigms shift is messy, social and fundamentally human. It often has more to do with qualitative explanations than quantitative science. Science, as a human endeavor, is very much a story-telling exercise.

Can you explain further how we can prevent this

Well maybe not much for the God of math but Newtonian mechanics is more practical for life, beyond just matters of taste.

Models are not trained to self-evaluate. Or only as an afterthought during tuning. So they are poor at it.

It isn’t mysterious.

Humans are trained incrementally, educated informally and formally, and along the way tested by context and classroom. Evaluated by people in their circle and strangers. The training to evaluate ourselves is near constant.

Even then, many people habitually believe they understand things they clearly don’t.

And can even be hostile to feedback.

Many forms of hallucination are canonized, or socially encouraged at varying demographic scales. While others are idiosyncratic.

Once self-examination is a first-class part of models training process, I expect they will respond like they have to being trained on vast troves of information, which is far excel human beings.

But, until then, their poor self-assessment isn’t mysterious, it is a mundane result of being trained not to do that. Or only as a tuning after thought.

Not as different from humans as we might like to think.

And models are noticeably improving on this measure. Humans in my experiencing may be regressing.