Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Is SQLite Coded In C

sqlite.org

Post reply on HN
Print view View original

321–330 of 411 posts

Re: Why Is SQLite Coded In C

#321

Post by MangoToupe »

1vuio0pswjnm7 wrote: post #318
jasonthorsness wrote: post #21

“None of the safe programming languages existed for the first 10 years of SQLite's existence. SQLite could be recoded in Go or Rust, but doing so would probably introduce far more bugs than would be fixed, and it may also result in slower code.” Modern languages might do more than C to prevent programmers from writing buggy code, but if you already have bug-free code due to massive time, attention, and testing, and t…

This begs the question of why Rust evangelists keep targeting existing projects instead of focusing writing new, better software. In theory these languages should allow software developers to write programs that they would not, or could not, attempt using languages without automatic memory management Instead what I see _mostly_ is re-writes and proposed re-writes of existing software, often software that has no netwo…

> This begs the question of why Rust evangelists keep targeting existing projects instead of focusing writing new, better software.

Designing new software is orders of magnitude more difficult than iterating on existing software

Re: Why Is SQLite Coded In C

#322

Post by steveklabnik »

csande17 wrote: post #267
aw1621107 wrote: post #240

Earlier quoted context omitted.

Yeah, Steve has been "just" a well-informed third party for a while now. I would be curious if he has commented on that specific issue before; usually when unsoundness comes up it's cve-rs which is mentioned. > but even current official sources like https://rust-lang.org/ say things like "Rust’s rich type system and ownership model guarantee memory-safety" that are only true of the vague-ideal "Rust language" and are…

I agree that it's a tough situation. "The type system guarantees memory safety" is an extremely important pillar of Rust's identity. They kind of have to portray all soundness issues as "more compiler bugs than something broken in the language itself" (see eg https://news.ycombinator.com/item?id=21930599 which references a GitHub label that AIUI would've included the trait coherence thing at the time) to keep making…

Yes, so there's a few things going on here: the first is, I absolutely pattern matched on the cve-rs link. Most people bringing that up are trying to bring up a quick gotcha. I did not follow the first link, I assumed it was to that. I am not educated on that specific bug at all.

I still ultimately think that the framing of Rust being any different than other languages here is actively trying to read the worst into things; Rust is working on having a spec, and formally proving things out. This takes a long time. But it's still ongoing. That doesn't mean Rust marketing relies on lying, I don't think most people even understand "soundness" at all, let alone assume that when Rust says "there's no UB in safe code" or similar that there's a promise of zero soundness bugs or open questions. That backwards incompatible changes are made in spite of breaking code at times to fix soundness issues is an acknowledgement of how sometimes there are in fact bugs, this doesn't change that for virtually all Rust users most of the time, updating the compiler is without fanfare, and so in practice, it is backwards compatible. I have heard of people struggling to update their C or C++ compilers to new standards, that doesn't mean that those languages are horribly backwards incompatible, just that there is a spectrum here, and being on one side of it as close as realistically possible doesn't mean that it's a lie.

But, regardless of all of that, it does appear that the issue you linked specifically may be not just a bug, but a real issue. That's my bad, and I'll try to remember that specific bug in the future.

Re: Why Is SQLite Coded In C

#323

Post by steveklabnik »

1vuio0pswjnm7 wrote: post #318
jasonthorsness wrote: post #21

“None of the safe programming languages existed for the first 10 years of SQLite's existence. SQLite could be recoded in Go or Rust, but doing so would probably introduce far more bugs than would be fixed, and it may also result in slower code.” Modern languages might do more than C to prevent programmers from writing buggy code, but if you already have bug-free code due to massive time, attention, and testing, and t…

This begs the question of why Rust evangelists keep targeting existing projects instead of focusing writing new, better software. In theory these languages should allow software developers to write programs that they would not, or could not, attempt using languages without automatic memory management Instead what I see _mostly_ is re-writes and proposed re-writes of existing software, often software that has no netwo…

There's no such thing as "Rust evangelists targeting existing projects" as some sort of broad strategy. What you're observing is that some people like to write programs in Rust, and so they choose to write new code in Rust. For some people, writing versions of software they understand already is a good way to learn a language, for some, it's that they don't like some aspect of the existing software and want to make something similar, but different.

That is, nobody perceives, say, "the silver searcher" as being some sort of nefarious plot to re-write grep, but they did with ripgrep, even though that's not what it is trying to do.

There are a few projects that are deliberately doing a re-write for reasons they consider important, but they're not "just because it's in Rust," they're things like "openssl has had memory safety issues, we should use tools that eliminate those" or "sudo is very important and has grown a lot of features and has had memory safety issues, we should consider cutting some scope and using tools that help eliminate the memory safety issues." But those number of projects are very small. Heck, even things like uutils were originally "I just want to learn Rust" projects and not some sort of "we must re-write the world in Rust."

Re: Why Is SQLite Coded In C

#324

Post by steveklabnik »

saalweachter wrote: post #297
Sammi wrote: post #242

Earlier quoted context omitted.

The Sqlite team in stuck in the classic dilemma. They are stuck with their existing thing because it is so big, that you can't just stop the world for you existing users and redo it. Meanwhile some small innovator comes along and builds the next thing because why not, they don't have anything holding them back. This is classic Innovator's Dilemma and Creative Destruction. This has of course not happened yet and we ha…

You say that like SQLite is a for profit company competing for market share or one of several rival projects in a corporation trying to not be canceled. It's an open sourced project! It's public domain ! If you make an open source project that is heavily used and widely lauded for a quarter century before being supplanted by a newer solution that's better, do you know what that is? A success! You did it! You made a t…

I agree with your broad point, but it's worth pointing out that SQLite very much is a product that is being sold and that aspect of things is not open source.

Re: Why Is SQLite Coded In C

#325

Post by [deleted] »

1vuio0pswjnm7 wrote: post #318
jasonthorsness wrote: post #21

“None of the safe programming languages existed for the first 10 years of SQLite's existence. SQLite could be recoded in Go or Rust, but doing so would probably introduce far more bugs than would be fixed, and it may also result in slower code.” Modern languages might do more than C to prevent programmers from writing buggy code, but if you already have bug-free code due to massive time, attention, and testing, and t…

This begs the question of why Rust evangelists keep targeting existing projects instead of focusing writing new, better software. In theory these languages should allow software developers to write programs that they would not, or could not, attempt using languages without automatic memory management Instead what I see _mostly_ is re-writes and proposed re-writes of existing software, often software that has no netwo…

[deleted]

Re: Why Is SQLite Coded In C

#326

Post by netrap »

Why not just say "because I don't want to change it"? rustaceans will always say your argument isn't valid for some reason or another. IDGAF, it's written in C because it is -- that's it!

Re: Why Is SQLite Coded In C

#327

Post by steveklabnik »

scuff3d wrote: post #191
mungaihaha wrote: post #170

Earlier quoted context omitted.

> But most of the other arguments they list can be picked apart and are only half true I'd like to see you pick the other arguments apart

> Other programming languages sometimes claim to be "as fast as C". But no other language claims to be faster than C for general-purpose programming, because none are. Not OP, And I'm not really arguing with the post, but this struck me as a really odd thing to include in the article. Of course nothing is going to be faster then C, because it compiles straight to machine code with no garbage collection. Literally any…

> Literally any language that does the same will be the same speed but not faster, because there's no way to be faster. It's physically impossible.

There is nothing special about C that makes this true. C has semantics, just like any language, that are higher level than assembly, and sometimes, those semantics make the code slower than other languages that have different semantics.

Consider this C function:

    void redundant_store(int *a, int *b) {
        int t = *a + 1;
        *a = t;
        *b = 0; // May clobber *a if (b == a)
        *a = t;
    }
Because a and b may point to the same address, you get this code (on clang trunk):

  redundant_store:
          mov     eax, dword ptr [rdi]
          inc     eax
          mov     dword ptr [rdi], eax
          mov     dword ptr [rsi], 0
          mov     dword ptr [rdi], eax
          ret
That fifth line there has to be kept in, because the final `*a = t;` there is semantically meaningful; if a == b, then a is also set to 0 on line four, and so we need to reset it to t on line five.

Consider the Rust version:

    pub fn redundant_store(a: &mut i32, b: &mut i32) {
        let t = *a + 1;
        *a = t;
        *b = 0; // a and b must not alias, so can never clobber
        *a = t;
    }
You get this output (on Rust 1.90.0):

  redundant_store:
          mov     eax, dword ptr [rdi]
          inc     eax
          mov     dword ptr [rsi], 0
          mov     dword ptr [rdi], eax
          ret
Because a and b can never alias, we know that the extra store to *a is redundant, as it's not possible for the assignment to *b to modify *a. This means we can get rid of this line.

Sure, eliminating one single store isn't going to have a meaningful difference here. But that's not the point, the point is that "it's not possible to be faster than C because C is especially low level" just simply isn't true.

Re: Why Is SQLite Coded In C

#328

Post by burntsushi »

1vuio0pswjnm7 wrote: post #318
jasonthorsness wrote: post #21

“None of the safe programming languages existed for the first 10 years of SQLite's existence. SQLite could be recoded in Go or Rust, but doing so would probably introduce far more bugs than would be fixed, and it may also result in slower code.” Modern languages might do more than C to prevent programmers from writing buggy code, but if you already have bug-free code due to massive time, attention, and testing, and t…

This begs the question of why Rust evangelists keep targeting existing projects instead of focusing writing new, better software. In theory these languages should allow software developers to write programs that they would not, or could not, attempt using languages without automatic memory management Instead what I see _mostly_ is re-writes and proposed re-writes of existing software, often software that has no netwo…

: This begs [sic] the question why copyleft evangelists keep targeting existing projects instead of focusing writing new, better software.

Few things in this life are novel. Regardless, when I wrote ripgrep, I was focusing on writing new and better software. I perceived several problems with similar tools at the time and set out to do something better.

Imagine if people actually listened to whinging like your comment. I'm glad I never have.

Re: Why Is SQLite Coded In C

#329

Post by dathinab »

uecker wrote: post #314
dwattttt wrote: post #252

Earlier quoted context omitted.

While a package with 10 million all-time downloads is nothing to sneeze at, it's had one memory corruption bug reported in its ~7 year life. It's being compared to a C library that's held to extremely high standards, yet this year had two integer overflow CVEs and two other memory corruption CVEs. SQLite is a lot more code, but it's also been around a lot longer.

The point is that matrix transpose should be trivial. But my main point really is that looking at CVEs is just nonsense. In both cases it is is a rather meaningless.

except that if you read into the actual issue you will realize that transposing matrices high performant is surprisingly not trivial, e.g. see this code code: https://github.com/ejmahler/transpose/blob/e70dd159f1881d86a...

furthermore the issue at core was an integer overflow, which is tricky in all languages and e.g. has poppet up on HN recently in context of "proven correct" code still having bugs (because the prove didn't use finit precision integers)

it's also less tricky in rust then in C due to no implicit casts and debug build checking for integer overflows and tests normally running against debug builds

Projects do sometimes enable it even on release builds for security sensitive code(1).

so if anything the linked issue is in favor of using rust over C while acting as a reminder that no solution is perfect

(1): It comes at a high performance cost, but sometimes for some things it's an acceptable cost. Also you can change such setting per crate. E.g. at a company I worked at a few years ago we did build some sensitive and iffy but not hot parts always with such checks enabled and some supper hot ML parts always with optimizations enabled even for "debug/test" builds.

Re: Why Is SQLite Coded In C

#330

Post by jandrewrogers »

wodenokoto wrote: post #161
jandrewrogers wrote: post #44

Earlier quoted context omitted.

If maximum performance is a top objective, it is probably because C++ produces faster binaries with less code. Modern C++ specifically also has a lot of nice compile-time safety features, especially for database-like code.

I can’t verify those claims one way or another, but I’m interested to hear why they were downvoted.

I've worked on a couple different projects that did substantial parallel development in C++20 and Rust, which created interesting opportunities for concrete comparison. It was performance-engineered code and we needed to validate their equivalence by testing them against each other.

The practical differences are larger than the theoretical differences, so I would expect the gap to diminish over time.

Rust reminded me of when I used to write database engines in Java. It required a lot more code, which has its own costs, but never really delivered on claims of comparable performance. The "more code" part largely comes down to the more limited ability to build good abstractions compared to C++20 and more limited composability. The "slower binaries" part comes down to worse codegen, which you can't blame on Rust per se, and a lot of extra overhead introduced in the code to satisfy the Rust safety model that would simply not be required in other systems languages.

Safety is a mixed bag. Rust can check several things at compile-time that C++20 cannot. C++20 can check several things at compile-time that Rust cannot.

For high-performance database-y code, memory is allocated at startup and is accessed via managed index handles. Rust does the same thing. In these types of memory models, i.e. no dynamic allocation and no raw pointers, both Rust and C++20 offer similar memory safety guarantees. Most high-performance software is thread-per-core that is almost purely single-threaded, so thread-safety concerns are limited.

That said, stripping away all of the above, the only real advantage that C++20 has its much more powerful toolset for building abstractions. Its performance and unique safety elements are based almost entirely on the ability to build concise, contextual, and highly composable abstractions as needed. This is not a feature that should be downplayed, I immediately miss it when I use most other languages.

Post reply on HN