Live data from Hacker News

Lean4 helped Terence Tao discover a small bug in his recent paper

mathstodon.xyz

51–60 of 230 posts

Re: Lean4 helped Terence Tao discover a small bug in his recent paper

#51
post #44

I wonder whether we could combine formal proof checkers (like the Lean proof checker) with language models that generate synthetic conjecture-proof pairs in a formal language like Lean. The Lean proof checker could be used to automatically verify whether the synthetic proofs written by the language model are correct. This information could be used to provide an RL reward signal applied to the original language model,…

This workflow is an explicit goal of the Lean 4 devs. The official Zulip chat has a channel dedicated to the interface between the two:

https://leanprover.zulipchat.com/#streams/219941/Machine%20L...

Re: Lean4 helped Terence Tao discover a small bug in his recent paper

#52
post #35

Earlier quoted context omitted.

The issue is that modern software fails because it's part of a complex system with many moving parts, rather than, it is inherently complex at the source-level. Certain sorts of algorithmically complex development (games, cars, medical hardware, etc.) would benefit from a 'closed-world verification' -- but that's not most software, and they have alternatives. 'Code correctness', including unit testing, ends up being…

Strong disagree. > The issue is that modern software fails because it's part of a complex system with many moving parts, rather than, it is inherently complex at the source-level. The choice to run a system as many different moving parts is a decision taken by the team in order to avoid failure . > Certain sorts of algorithmically complex development -- but that's not most software It's all software. > 'Code correctn…

I'm not really sure what you're saying here -- the parent commenter is talking about closed-world source-level code verification.

My claim is that these techniques help in relatively few areas of software development. In the main, software is built with open-world interactions across a range of devices (network, disk, etc.) where those interactions dominate the net failure modes of the system.

In the latter case, source-level verification is not only of no help, but it's a huge resource misallocation. You end up writing tests that just create the illusion of success; and then the whole thing constantly falls over.

Spend all that time instead in designing tests that do fit into ram (eg., sample your datasets); and instrumentation that does reveal errors (eg., sampling of memory, requests, etc.).

One of my first software jobs was writing a reverse proxy -- one of my co-devs wrote unit tests that simply established the in/out of various functions was "as expected". Pretty useless -- the issue is whether the proxy actually worked.

Likewise most source-level correctness efforts are 'testing theatre'.

Devs are looking for cargo-cult solutions they can copy/paste. No, testing software is an actual area of development -- and you need to develop tests, not cargo-cult

Re: Lean4 helped Terence Tao discover a small bug in his recent paper

#53

Lean4 is brilliant. Worth digging into as a programmer. Coq, Lean4, Agda etc. made my brain explode in a good way. Making me a better software developer.

I'm kind of interested in how useful Lean4 is as a programming language, and if it's easy to prove things about a program written in Lean. I should probably look into that when I have a minute.

Regarding usefulness: Lean is very nice to program in, if you care about pure functional languages; its FFI allows you to incorporate fast C routines very easily if pure Lean is not performant enough or lacks features. However, in some domains, Lean is within a decimal order of magnitude of (not hand-optimized) C; some benchmarks I hand-made recently impressed me.

Regarding proving things about programs, no, it is not easy, and the developers do not seem to consider it a core goal of Lean.

Re: Lean4 helped Terence Tao discover a small bug in his recent paper

#54
post #50
post #47

That one of the brightest minds of our generation is able to increase his bandwidth with the combination of LLMs and automated proofs makes me super bullish on this tech combo in the future! It starts with bug-fixing, then supports verification, until it starts propelling new discoveries and push the envelope. We need a term when a dynamic like Moore's Law "infects" a field that had no such compounding properties bef…

Lean 4 is a programming language and theorem prover, and has nothing to do with LLMs as far as I know though.

the missing context from the previous comment is that Tao used GitHub Copilot to help with learning Lean.

He's been writing about it as he goes, most recently: https://mathstodon.xyz/@tao/111271244206606941

Re: Lean4 helped Terence Tao discover a small bug in his recent paper

#56
post #8

For people that know neither (like me 5 minutes ago): >Lean4 > Lean is a functional programming language that makes it easy to write correct and maintainable code. You can also use Lean as an interactive theorem prover. https://lean-lang.org/about/ > Terence Tao > [...] is an Australian mathematician. He is a professor of mathematics at the University of California, Los Angeles (UCLA), where he holds the James and Ca…

some claim his intelligence is the highest ever measured

I'm sure some people do indeed say that, as that has also been said about most famous smart people.

However, from my experience as someone who scored "off the charts" on a UK school Cognitive Abilities Test and 148 on an IQ test: any score over 130 is suspect, as basically nobody is collecting enough data to make the tests statistically valid once you're more than 2σ from the mean.

Re: Lean4 helped Terence Tao discover a small bug in his recent paper

#57
post #38

For people that know neither (like me 5 minutes ago): >Lean4 > Lean is a functional programming language that makes it easy to write correct and maintainable code. You can also use Lean as an interactive theorem prover. https://lean-lang.org/about/ > Terence Tao > [...] is an Australian mathematician. He is a professor of mathematics at the University of California, Los Angeles (UCLA), where he holds the James and Ca…

Kevin Buzzard (a professional mathematician) has a few good talks about Lean https://youtu.be/Dp-mQ3HxgDE from 2019 at MS Research https://www.youtube.com/watch?v=SEID4XYFN7o&t=4m35s (2022 at ICM international math congress)

Youtube has forgotten what "audio" is for me so noting the first talk is also at https://www.microsoft.com/en-us/research/video/the-future-of... - which is using youtube - but manages to make noises. Which I suppose is a pointwise instance arguing that software does not work as well as it might do.

Re: Lean4 helped Terence Tao discover a small bug in his recent paper

#58

For people that know neither (like me 5 minutes ago): >Lean4 > Lean is a functional programming language that makes it easy to write correct and maintainable code. You can also use Lean as an interactive theorem prover. https://lean-lang.org/about/ > Terence Tao > [...] is an Australian mathematician. He is a professor of mathematics at the University of California, Los Angeles (UCLA), where he holds the James and Ca…

I'm relieved that Lean4 doesn't seem to be some AI thing.

Re: Lean4 helped Terence Tao discover a small bug in his recent paper

#59
post #8

Earlier quoted context omitted.

some claim his intelligence is the highest ever measured

He's gonna fix EVERYTHING. And he's gonna do it in ONE WEEK!

You know, we have this thing in our company, when someone performs so well they can do other people's work.

Re: Lean4 helped Terence Tao discover a small bug in his recent paper

#60
I’m really excited about dependent types. I’m expecting we won’t get them for a while though. Dependent Haskell is progressing but apparently it’s hard to retrofit. Idris’ own creator has said he expects it to be a model for other languages, I don’t think it will ever have mainstream adoption. Coq and Agda, F* aren’t really designed to be general purpose.

Although the implementation for the compiler is complex, and the syntax can get complex and verbose, to me my requirement is simple: I want to encode everything about input and output that I know.

Right now in mainstream languages I often know more about my arguments or output than the type system will allow me to specify.

Post reply on HN