Live data from Hacker News

Notes by djb on using Fil-C

cr.yp.to

231–240 of 263 posts

Re: Notes by djb on using Fil-C

#231
post #52

Earlier quoted context omitted.

If that's all you need, the state of the art is very available already through the JVM and the .NET CLR, as well as a handful others depending on your use case. Most of those also come with decent languages, and great facilities to leverage the GC to its maximum. But GCs aren't magic and you will never get rid of all the overhead. Even if the CPU time is not noticeable in your use case, the memory usage fundamentally…

I do not think this is niche in the slightest. I would very happily take a 2-4x slowdown for almost all of the web facing C software I run if I get guaranteed memory safety. I will be using at the very least fil-c openssh (and likely much more) on every machine I run.

Apologies ahead of time as this is pure FUD, That is I don't actually know what I am talking about but had an interesting thought.

Remember the Debian weak keys kerfuffle, That was caused because the Debian package maintainer saw a warning about using uninitialized memory, fixed it, and then it turned out that uninitialized memory was a critical seed for the openssl random number generator.

Anyhow my stupid FUD thought. is there a weak-key equivalent bug that shows up now that your C compiler is memory safe?

Re: Notes by djb on using Fil-C

#232
post #146

Earlier quoted context omitted.

> best case scenario since there is very little memory management involved and runtime is dominated by computation in tight loops. This describes most C programs and many, if not most, C++ programs. Basically, this is how C/C++ code is being written, by avoiding memory management, especially in tight loops.

This depends heavily on what problem domain you're talking about. For example, a DBMS is necessarily going to shuffle a lot of data into and out of memory.

I am professional database developer. We do not do what you are thinking we are doing. ;)

Re: Notes by djb on using Fil-C

#233
post #36

Building tools is one thing, building a system like Postgres or Databases is going to be another thing. Anyone really tried building PG or MySQL or such a complex system which heavily relies on IO operations and multi threading capabilities

Look at how fanatic the compatibility actually is. Building Postgres or MySQL is conceivable but probably will require some changes. (SQLite compiles and runs with zero changes right now.)

SQLite runs about 5 times faster compiled with GCC (13.3.0) than it does when compiled with FIL-C. And the resulting compiled binary from GCC is 13 times smaller.

Re: Notes by djb on using Fil-C

#234
post #35

Earlier quoted context omitted.

Note that it is a garbage collector designed and implemented by one of the most experienced GC experts on earth. He previously designed and implemented WebKit's state of the art concurrent GC, for example. So—yes, but don't dismiss it too quickly.

It's amazing how much technical discourse revolves around impressions. "Oh, it has a GC! GC bad!" "No, this GC by smart guy, so good!" "No, GC always bad!" People aren't engaging with the technical substance. GC based systems and can be plenty good and fast. How do people think JavaScript works? And Go? It's like people just absorbed from the discursive background radiation the idea GC is slow without understanding w…

> It's amazing how much technical discourse revolves around impressions.

One of the single most incisive comments in the whole discussion.

My take: people don't take the time to even try to understand some things of only moderate complexity. They dismiss it as "too hard", drop it, accept the received wisdom and move on.

This is also behind the curse of "best practice". After coming up on 40Y in the industry, my impression is that this boils down to "this is what the previous guys did and it worked". In other words, very close to "nobody ever got fired for buying IBM" as a methodology.

What it means: "you don't need to think about it -- just do this." Which quickly turns to "you don't need to understand it, just do this."

Why I am saying this: because I think you're absolutely right, much of the industry discourse runs on impressions -- but there is a second factor that matters as much.

People form impressions of things they don't understand well themselves by listening to the opinions of people they trust. The question then is: where do they find those opinions?

There are communities of like-minded folks, folks interested in particular tech or whatever. Rust folks, "plain ol' C" folks, C++ folks, "let's replace C with $something_more_modern" folks (where that's D or Nim or whatever).

But those communities group together too. They hang out in the same fora, talk on the same media, etc. Result, there are hierarchies of communities, and the result is like religions: people in one church know of other related churches fairly well, and some are siblings, relatives, whatever; others are hated enemies to be scorned.

But they know next to nothing of other religions which are too far away, too different.

So when people are comparing the offspring of C, they are probably from the Unix faith. They don't know that but everyone they ever talked to, every software they ever saw, is a Unix, so they don't realise there's anything else.

I see passionate debates about Rust vs Go and things and I strongly suspect these are problems fixed among the Wirthian languages decades ago. Walls of text, thousands of lines of code, for things fixed in Modula-2 or Ada 30 or 40 years ago and those folks moved on.

Whereas the Lisp folks never had those problems and are politely baffled by tools that still have flaws that deeply passionate 20-somethings and 30-somethings are calling each other names about and blocking each other over.

I've had people in dead seriousness tell me that ALL OTHER SOFTWARE is written in C at the lowest level. They are amazed when I laugh at them. C is a niche in a niche and the team that wrote C and Unix moved on to Aleph and Limbo and one splinter wrote Go.

The Forth people laugh at the vastly verbose Lisp folks and their forgotten million-line OSes.

The APL people smile at the painfully iterative Forth and Lisp folks.

Unix won on servers and it's pushing Windows off desktops, now relegating everything else to embedded and realtime and the handwritten-by-one-person systems, where nobody else will ever read their code.

I can't help but think that there must be a better way. Not sure what it is. Classes in comparative software religion on Youtube? Sports style competitions for the smallest/simplest/fastest ways to solve particular problems in languages people might not consider? Tools for easier linkage between less-known languages and well-known OSes?

Re: Notes by djb on using Fil-C

#235
post #233
post #36

Earlier quoted context omitted.

Look at how fanatic the compatibility actually is. Building Postgres or MySQL is conceivable but probably will require some changes. (SQLite compiles and runs with zero changes right now.)

SQLite runs about 5 times faster compiled with GCC (13.3.0) than it does when compiled with FIL-C. And the resulting compiled binary from GCC is 13 times smaller.

Interesting! I guess that's from your standard benchmark setup. Please note that Fil-C makes no secret of having a performance penalty. It's definitely a pre-1.0 toolchain and only recently starting to pick up some momentum. The author is eager to keep improving it, and seems to think that there's still plenty of low hanging and medium hanging fruit to pick.

It does (or did, at some point) pass the thorough SQLite test suite, so at least it's probably correct! The famous SQLite test coverage and general proven quality might make SQLite itself less interesting to harden, but in order to run less comprehensively verified software that links with SQLite, we have to build SQLite with Fil-C too.

Re: Notes by djb on using Fil-C

#236

Earlier quoted context omitted.

Good cryptographic code should match its algorithmic description. Rust enables abstractions that allow this. C does not. That you have some familiarity with C and not Rust should not be a contributing factor. I say this as someone who has written cryptographic code that’s been downloaded millions of times.

The problem in terms of reference implementations is exactly the abstractions. Reference implementations should be free of abstractions and should be understandable. Abstractions make code much less understandable. I say this as someone who has been involved in cryptography and has read through dozens of reference implementations. Stick to C, not Python or Rust, it is much easier to understand because the abstraction…

Abstractions are the only way to make sense of cryptography. Pretending otherwise leads to cross layer bugs and vulnerabilities. Of course bad abstractions are bad, but that doesn’t mean no abstraction is good.

Feel free to post your challenge snippet.

Re: Notes by djb on using Fil-C

#237

Earlier quoted context omitted.

> Is your intention that people use the Fil-C garbage collector instead of free()? Or is it just a backstop in case of memory leak bugs? Wow great question! My intention is to give folks powerful options. You can choose: - Compile your code with Fil-C while still maintaining it for Yolo-C. In that case, you'll be calling free(). Fil-C's free() behavior ensures no GC-induced leaks (more on that below) so code that doe…

> Nope. Reason: the Fil-C runtime itself now relies on GC, and there's some functionality that only a GC can provide that has proven indispensable for porting some complex stuff (like CPython and Perl5). What if there was a flag you could set on an allocation, “must be freed”. An app can set the “must be freed” flag on its allocations, meaning when the GC collects the allocation, it checks if free() has been called o…

Internal allocations can have pointers to user allocations

Re: Notes by djb on using Fil-C

#238

Earlier quoted context omitted.

It's not an either-or (well, except for this last item). It seems sensible to not write new software in plain C. Rust is certainly a valid choice for a safer language, but in many cases overkill wrt how painful the rewrite is vs benefits gained from avoiding a higher-level memory-safe one like OCaml. At the same time, "let's just rewrite everything!" is also madness. We have many battle-tested libraries written in C…

> the software may be bug-free, but sandboxing it may still be desirable e.g. as a matter of trust (or lack thereof) Wouldn't the only cause of mistrust be bugs, or am I missing something? If the program is malicious, sandboxing isn't the pertinent action.

If any program can potentially be malicious (which is the effectively the case today with any downloaded software), then sandboxing is exactly the pertinent action - provided that the sandbox is tight enough.

Re: Notes by djb on using Fil-C

#239

Earlier quoted context omitted.

The problem in terms of reference implementations is exactly the abstractions. Reference implementations should be free of abstractions and should be understandable. Abstractions make code much less understandable. I say this as someone who has been involved in cryptography and has read through dozens of reference implementations. Stick to C, not Python or Rust, it is much easier to understand because the abstraction…

Abstractions are the only way to make sense of cryptography. Pretending otherwise leads to cross layer bugs and vulnerabilities. Of course bad abstractions are bad, but that doesn’t mean no abstraction is good. Feel free to post your challenge snippet.

  pub fn read>(path: P) -> io::Result> {
    fn inner(path: &Path) -> io::Result> {
      let mut file = File::open(path)?;
      let mut bytes = Vec::new();
      file.read_to_end(&mut bytes)?;
      Ok(bytes)
    }
    inner(path.as_ref())
  }
"aBsTraCtiOnS aRe gOod"... Right.

Reference implementations must NOT have abstractions like this. Rust encourages it. Lots of Rust codebase is filled with them. Your feelings for Rust is irrelevant. C is simple and easy to understand, therefore reference implementations must be in C. Period.

...or Common Lisp, or OCaml... why not?!

Re: Notes by djb on using Fil-C

#240

For those who might miss it, the notes cite a new 64-bit version of cdb that supports exabyte databases https://cdb.cr.yp.to Also maybe of interest is that the new cdb subdomain is using pqconnect instead of dnscurve

The PQConnect documentation, specifically the document "INSTALL.md", describes the pq1 portion of the CNAME as a subdomain.

   Please update your DNS A/AAAA records for all domains on this server as follows:

   Existing record:
   Type    Name        Value
   A/AAAA  SUBDOMAIN   IP Address

   New Records:
   Type    Name        Value
   CNAME   SUBDOMAIN   pq1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.DOMAIN.TLD
   A/AAAA  pq1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX  IP Address
   TXT    pq1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.DOMAIN.TLD    p=42424
   TXT    ks.pq1XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX.DOMAIN.TLD    ip=IP ADDRESS;p=42425"
Post reply on HN