Live data from Hacker News

What can you confidently guarantee about your software?

queue.acm.org

51–60 of 71 posts

Re: What can you confidently guarantee about your software?

#51
post #8

Earlier quoted context omitted.

Well, I'm someone who barely knows more than jack about formal verification, but in pretty much every case you have to have some kind of model that you are actually verifying. How close that model sits to the real thing you have modeled is an important question, and you are free to be as close or distant as you want -- e.g. for verifying different properties of a programming language you might decide to not care abou…

Formal verification is a siren song. The siren sings, "bug-free code is possible in principle!" But it's a trap. Even with LLMs, bug-free code is impractical. I argued that property-based testing is mostly unhelpful for e-commerce/CRUD apps, and that formal verification is a performance improvement on property-based tests. In a property-based test, you identify some rule (an invariant) that you want to apply to your…

Check out seL4 and CompCert for real-world examples of complex software proven correct and used in industry.

Re: What can you confidently guarantee about your software?

#52

Formal verification is still too limited to be useful for most app developers. The article gives an example of an e-commerce platform using it to prove the correctness of managing refunds, but then acknowledges: > As of today, the formally verified core can handle most effect-free logic—invariants, transitions, conflict resolution. But the UI, network calls, and database interactions typically sit outside the verific…

If you are willing to relax the restrictions, and you probably should, model checking is probably worth its weight in gold for these scenarios. You won’t get proofs but you will spell out your logic in a formal language[0] and each run of the checker will exhaustively check your invariants[1]. [0] Useful because often you will learn something you hadn’t considered. [1] A proof will guarantee your statements hold over…

I’ve recently started using PBT at work and it has been more than worth it, and I want to try Model checking next.

Re: What can you confidently guarantee about your software?

#53
post #45

Earlier quoted context omitted.

> Have an LLM write a 50-100KLOC program and prove all correctness properties [...] and tell us what it cost. Assuming the 50-100KLOC program is of real-world use and not something contrived for the sake of offering something to prove, it is unlikely that proving all correctness properties will be possible, fundamentally. So costs will be nothing — or infinite if you foolishly remain determined to try the impossible.…

> I would think the cost multiplier in those cases is much lower for an LLM as compared to a human that doesn't have an inherit understanding and needs to give it thought. Wouldn't you? No. I don't see why proving would require less relative effort for an LLM. In fact, years ago, long before LLMs, I wrote about why it is relatively easy to write sort-of-correct software yet hard to write provably correct software, an…

Every time you compile a statically-typed programming language you are using formal verification, so we have all kinds of industrial scale examples. The reports suggest that outputting tokens for these languages is as easy for LLMs as Javascript. And actually, I would suggest that the reports indicate that LLMs find it easier to output tokens for those languages than Javascript. LLMs are laughably bad at writing Javascript.

On the other hand we can watch humans struggle to do the same. How often have you heard things like "I won't use Rust because it is too hard to use"? I have never seen an LLM refuse to output Rust because it thought it was too hard. So what in that suggests an equivalent multiplier?

> yet hard to write provably correct software

That isn't just hard. Proving software correct in complete generally is impossible. There are all kinds of practical and fundamental constraints that leave it to be impossible. Verification is only useful when you are acting within the scope of a compressed specification of a system's behaviour.

Re: What can you confidently guarantee about your software?

#54

Formal verification is still too limited to be useful for most app developers. The article gives an example of an e-commerce platform using it to prove the correctness of managing refunds, but then acknowledges: > As of today, the formally verified core can handle most effect-free logic—invariants, transitions, conflict resolution. But the UI, network calls, and database interactions typically sit outside the verific…

I mean the cumbersome gold standard is literally just using the program. Fictional dialogue:

  A: Is it possible to pay for a product on Firefox mobile?   
  
  B: Yeah, I manually test it for each new version  
  
Please let us not forget: any automated solution is just there to reduce the amount of manual testing needed.

Of course that is a lot of work and testing every possible combination of situations, technologies etc is probably not feasible, but you shoud have a checklist that you follow, if there are tests you can't automate.

Re: What can you confidently guarantee about your software?

#55
post #8

Earlier quoted context omitted.

Well, I'm someone who barely knows more than jack about formal verification, but in pretty much every case you have to have some kind of model that you are actually verifying. How close that model sits to the real thing you have modeled is an important question, and you are free to be as close or distant as you want -- e.g. for verifying different properties of a programming language you might decide to not care abou…

Formal verification is a siren song. The siren sings, "bug-free code is possible in principle!" But it's a trap. Even with LLMs, bug-free code is impractical. I argued that property-based testing is mostly unhelpful for e-commerce/CRUD apps, and that formal verification is a performance improvement on property-based tests. In a property-based test, you identify some rule (an invariant) that you want to apply to your…

Formal Specification and Verification of Data-Centric Web Services by Iman Saleh (PhD thesis) - https://www.researchgate.net/publication/235675652_Formal_Sp...

Excerpt:

In this thesis, we develop and evaluate a formal model and contracting framework for data-centric Web services. The central component of our framework is a formal specification of a common Create-Read-Update-Delete (CRUD) data store. We show how this model can be used in the formal specification and verification of both basic and transactional Web service compositions. We demonstrate through both formal proofs and empirical evaluations that our proposed framework significantly decreases ambiguity about a service, enhances its reuse, and facilitates detection of errors in service-based implementations.

Formalizing Data-Centric Web Services by Iman Saleh (book) - https://link.springer.com/book/10.1007/978-3-319-24678-9

Author website - https://www.imansaleh.com/

Re: What can you confidently guarantee about your software?

#56

Formal verification is still too limited to be useful for most app developers. The article gives an example of an e-commerce platform using it to prove the correctness of managing refunds, but then acknowledges: > As of today, the formally verified core can handle most effect-free logic—invariants, transitions, conflict resolution. But the UI, network calls, and database interactions typically sit outside the verific…

If you are willing to relax the restrictions, and you probably should, model checking is probably worth its weight in gold for these scenarios. You won’t get proofs but you will spell out your logic in a formal language[0] and each run of the checker will exhaustively check your invariants[1]. [0] Useful because often you will learn something you hadn’t considered. [1] A proof will guarantee your statements hold over…

I endorse this, and IME find there are major benefits to “relaxing” even moreso through reductions or decompositions into well-known models rather than defining your own.

A lot of problems in practice are just too complex, too theoretically difficult, or some combination of not-theoretically-interesting-enough for an academic or specialized worker to be attracted/assigned/exposed to it, or just too difficult/expensive/time consuming/unimportant to be worth directly formally modeling. Or, they’re big problem spaces and you want to start where you get the most bang-for-your-buck. A (partial) reduction lets you still realize practical benefits for problems that may not perfectly or entirely be amenable to simple/small models, or which themselves may be “jagged”/gnarly problem domains due to real-world constraints like backwards compatibility.

Reduction to a formal target arise naturally from defining type conversions (likely to a codomain that is either actually a strict superset of the real reduction image, ie overly generalized/non-surjective, or bijective for some partial subset/step). Mathematically or computationally, it may be less than ideal to use an overly broad/flexible/general abstraction to model something more structured. But practically, it lets you use actual software written for the target reduction and apply any results/properties of the more familiar or better studied model.

The less-than-ideal reduction may be easily verifiable/simple and so obviously safe that, in-context, it’s clearly worth choosing the technically-suboptimal/less-elegant/partial approach to free ride off existing formal verification, without needing to wade deep into theoretical territory, than to chase perfection.

For example, if you can model a data format as a Context Free Grammar, or a subset of all context free grammars, a developer could spell out that grammar or grammars and then use existing metaparsing tools to safely process input data rather than write their own parser. And they wouldn't even need the format to be a complete CFG to leverage CFG tooling! Suppose portions are context-sensitive - for example, decoding the payload requires applying some length-prefix extracted from the header - just encode and parse the header format as a CFG with an opaque payload blob suffix with a length equal to eg the max payload/packet/fragment size. Perhaps some part of the header also specifies the payload’s structure in addition to the length: you could have one CFG for reading the header, a simple wrapping CFG for a fixed-length frame containing a header followed by junk (FRAME_HEADER_CFG :=, a CFG for each payload format, and one wrapping/router function that handles the very simple non-cfg step of using the first-pass parsing output to determine the length/format of the second pass.

FRAME_HEADER_CFG := HEADER_CFG {BYTE}

...

Parse(byte[] in) {

  AST frame = ParseCFG(FRAME_HEADER_CFG, in[:FRAME_SIZE]);

  Header h, byte[] p = frame[0], frame[1];

  assert(len(p) >= h.payload_size);

  return ParseCFG(PAYLOAD_CFGS[h.payload_type], in[FRAME_SIZE-len(p):h.payload_size]);
}

Then even though your data format is not technically a CFG, the vast majority of your parsing occurs within a CFG, with only a small part that does (in pseudocode). And you did not have to do or run any formal verifications, or use special model checkers, only encode the CFG-compatible subsets of your data format into something like EBNF, then use a compatible formally-verified metaparser, and take a leap of faith that the small amount of non-formally-verified/not-nice-to-model code is safe. That's much more realistic for a developer to do than to implement a formal-verification of a non-cfg format de novo.

Re: What can you confidently guarantee about your software?

#57
post #9

Formal verification is still too limited to be useful for most app developers. The article gives an example of an e-commerce platform using it to prove the correctness of managing refunds, but then acknowledges: > As of today, the formally verified core can handle most effect-free logic—invariants, transitions, conflict resolution. But the UI, network calls, and database interactions typically sit outside the verific…

The first part of formal verification is getting a formal specification. I don't know about most developers, but I rarely get a written specification for anything I work on, and when I do, it's no where near what would be needed to turn it into a formal specification. Anyway, the specification is subject to change at the whim of a hat, so putting a lot of effort into verifying it is foolish. I do see value in formal…

Nothing stops you from writing a formal spec for an implementation or the technical parts of a problem that the product / upstream requirements don't explicitly demand. In fact I would say this is a large part of what software engineering fundamentally is; the reason specs are written in a certain way is because they model a problem domain and encode fundamental (or incidental, which could still be very well-reasoned, eg due to backwards compatibility, limitations of dependencies, or practical constraints) properties that are unlikely to change. Once you've actually written software that other systems/components are depending on, you've in practice created a spec. And it's a false dichotomy to think that just because there is no formalized spec for something before it's implemented at all, that this means that once the domain is better understood or stabilizied, you cannot use formal verification on the de-facto spec that has been arrived at/de-jure rules and structure that are prescribed but not yet formalized.

Even when you are writing something for the first time, like a parser for a certain kind of data format, you often can greatly accelerate your production-safety/security maturity by formalizing your spec enough to be fuzzed and to use existing parsing tooling that avoids the footguns most developers will introduce writing their own parsers from scratch. This is IMO the most common situation in which formal verification becomes very useful; even if the data format is purely an implementaton detail, any time parsing untrusted/user-provided input is involved, it would be much more foolish to try to just throw together some random string processing functions than to use something like flex/bison, an existing serialization format/protocol, or a metaparser (which may themselves be formally verified).

Re: What can you confidently guarantee about your software?

#59

Formal verification is still too limited to be useful for most app developers. The article gives an example of an e-commerce platform using it to prove the correctness of managing refunds, but then acknowledges: > As of today, the formally verified core can handle most effect-free logic—invariants, transitions, conflict resolution. But the UI, network calls, and database interactions typically sit outside the verific…

I have been working on metaparsing and some related tools for spec-based-development, fuzzing, and property-based-testing off and on for the past few months. I hope to make it easier for application developers to get the benefits of formal verification without having to spend inordinate amounts of time or resources writing unfamiliar, difficult formal-verification software or tests. You're right that it's not useful for most developers to concern themselves with this now, but I think that's mostly just because nothing makes it easy enough yet.

> Even property-based testing is mostly unhelpful for e-commerce apps like these.

Only in practice because of lack of application-developer oriented tooling. Why can you not have a callback assertion or database trigger that alerts or opens a ticket when a refund is not processed within a specified amount of time? Why can't you map out your payment processors' APIs and, if amenable, define a more transactional interface on top of their CRUD API to automatically rollback incomplete or partial state transitions? (Obviously, not all payment processors will expose an interface allowing you to do this in every case; but often times they do provide the fundamental pieces in-practice, just not in a way that is easy to discover or implement). Why can't you define cross-dependency static analysis guaranteeing that eg the core checkout flow continues to work before even running any integration tests? With the right tooling you could, it's just unreasonable to expect every developer working on ecommerce to get a PhD in automated theorem proving to build their own tools for that.

> tricky logic with just a bit of I/O

I actually think that IO is the #1 most important and high ROI surface to apply formal verification (and fuzzing/other related tools which are downstream of formal methods). How many ecommerce apps are secure against DOS or runtime panics from an adversarial client sending them junk data? How many write their own janky/insecure parsers against input formats or external APIs and introduce security vulnerabilities there? I would hazard a guess that most 5+ dev webapps are guilty of basic IO sins that formal verification tools could identify in short time and also prevent from being introduced if they were added to the core development process.

The approach I'm taking is to use codegen to automatically convert OpenAPI specs and grpc service definitions (and in some cases actually just methods and http handlers themselves), and wire-format data, and application-level data structures into formalized representations that I can use downstream with autofuzzing/trace/black-box integration testing/etc tools. There are certain things that only make sense to do with AI, that are very hip and envogue, eg black-box or API-surface-informed adversarial integration testing. But you can actually much more cheaply just use AI to set up deterministic tests like this once; the problem up until now is the lack of good end-user tooling and support for those tools, which are hard to write and test especially in real-world applications that don't conform to the specific industries/academic niches that formal verification are already adopted within. That's getting solved too.

Re: What can you confidently guarantee about your software?

#60
post #39
post #25

Earlier quoted context omitted.

Nitpick: You don't necessarily need any specification at all in order to reap benefits. Formal verification languages come with a lot of conditions that your program must fulfill in order to be accepted: Every loop terminates, every object you want to read/write is non-null, every list or array access is in bounds, etc. For example, if you load an arbitrary C program into Frama-C, you'll have tons of properties to pr…

> The promises you get is that the program will always terminate I don't want my programs to terminate though.

Oh no, you have just proved formal verification useless! Or, if you have an outer event loop that should potentially run forever, you mark only that one loop accordingly (details depend on the system).
Post reply on HN