Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Is SQLite Coded In C

sqlite.org

Post reply on HN
Print view View original

81–90 of 411 posts

Re: Why Is SQLite Coded In C

#81

Post by 3eb7988a1663 »

slashdev wrote: post #31

This is ignoring the elephant in the room: SQLite is being rewritten in Rust and it's going quite well. https://github.com/tursodatabase/turso It has async I/O support on Linux with io_uring, vector support, BEGIN CONCURRENT for improved write throughput using multi-version concurrency control (MVCC), Encryption at rest, incremental computation using DBSP for incremental view maintenance and query subscriptions. Time…

It should be noted that project has no affiliation with the SQLite project. They just use the name for promotional/aspirational purposes. Which feels incredibly icky.

Also, this is a VC backed project. Everyone has to eat, but I suspect that Turso will not go out of its way to offer a Public Domain offering or 50 year support in the way that SQLite has.

Re: Why Is SQLite Coded In C

#82

Post by bravura »

Jtsummers wrote: post #10

Two previous, and substantial, discussions on this page: https://news.ycombinator.com/item?id=28278859 - August 2021 https://news.ycombinator.com/item?id=16585120 - March 2018

I'm curious about tptacek's comment (https://news.ycombinator.com/item?id=28279426). 'the "security" paragraphs in this page do the rest of the argument a disservice. The fact is, C is a demonstrable security liability for sqlite.'

The current doc no longer has any paragraphs about security, or even the word security once.

The 2021 edition of the doc contained this text which no longer appears: 'Safe languages are often touted for helping to prevent security vulnerabilities. True enough, but SQLite is not a particularly security-sensitive library. If an application is running untrusted and unverified SQL, then it already has much bigger security issues (SQL injection) that no "safe" language will fix.

It is true that applications sometimes import complete binary SQLite database files from untrusted sources, and such imports could present a possible attack vector. However, those code paths in SQLite are limited and are extremely well tested. And pre-validation routines are available to applications that want to read untrusted databases that can help detect possible attacks prior to use.'

https://web.archive.org/web/20210825025834/https%3A//www.sql...

Re: Why Is SQLite Coded In C

#83

Post by hypeatei »

oconnor663 wrote: post #64
hypeatei wrote: post #48

Earlier quoted context omitted.

Couldn't a method like `get_unchecked()` be used to avoid the bounds check[0] if you know it's safe? 0: https://doc.rust-lang.org/std/vec/struct.Vec.html#method.get...

Yes. You have to write `unsafe { ... }` around it, so there's an ergonomic penalty plus a more nebulous "sense that you're doing something dangerous that might get some skeptical looks in code review" penalty, but the resulting assembly will be the same as indexing in C.

I figured, but I guess I don't understand this argument then. SQLite as a project already spends a lot of time on quality so doing some `unsafe` blocks with a `// SAFETY:` comment doesn't seem unreasonable if they want to avoid the compiler inserting a panic branch for bounds checks.

Re: Why Is SQLite Coded In C

#84

Post by wrs »

steveklabnik wrote: post #27
DarkNova6 wrote: post #19

> All that said, it is possible that SQLite might one day be recoded in Rust. Recoding SQLite in Go is unlikely since Go hates assert(). But Rust is a possibility. Some preconditions that must occur before SQLite is recoded in Rust include: - Rust needs to mature a little more, stop changing so fast, and move further toward being old and boring. - Rust needs to demonstrate that it can be used to create general-purpos…

1. Rust has had ten years since 1.0. It changes in backward compatible ways. For some people, they want no changes at all, so it’s important to nail down which sense is meant. 2. This has been demonstrated. 3. This one hinges on your definition of “obscure,” but the “without an operating system” bit is unambiguously demonstrated. 4. I am not an expert here, but given that you’re testing binaries, I’m not sure what is…

For a little more color on 5, as a user of no_std Rust on embedded processors I use crates like heapless or trybox that provide Vec, String, etc. APIs like the std ones, but fallible.

Of course, two libraries that choose different no_std collection types can't communicate...but hey, we're comparing to C here.

Re: Why Is SQLite Coded In C

#85

Post by t14n »

sema4hacker wrote: post #59

"Why SQLite is coded in C..." is an explanation, as documented at sqlite.org. "Why is SQLite coded in C and not Rust?" is a question, which immediately makes me want to ask "Why do you need SQLite coded in Rust?".

fwiw there's a project doing just that: https://github.com/tursodatabase/turso

they have a blog hinting at some answers as to "why": https://turso.tech/blog/introducing-limbo-a-complete-rewrite...

Re: Why Is SQLite Coded In C

#86

Post by unclad5968 »

selcuka wrote: post #67
lionkor wrote: post #50

Earlier quoted context omitted.

It's not like that, the compiler explicitly doesn't do compile-time checks here and offloads those to the runtime. Rust does not stop you from writing code that accesses out of bounds, at all. It just makes sure that there's an if that checks.

Ok, but you can still test all the branches in your source code and have 100% coverage. Those additional `if` branches are added by the compiler. You are responsible for testing the code you write, not the one that actually runs. Your compiler's test suite is responsible for the rest. By the same logic one could also claim that tail recursion optimisation, or loop unrolling are also dangerous because they change the…

I don't see anything wrong with taking responsibility for the code that actually runs. I would argue it's that level of accountability has played a part in Sqlite being such a great project.

Re: Why Is SQLite Coded In C

#87

Post by estebank »

Aurornis wrote: post #62
bfkwlfkjf wrote: post #16

> Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy. Huh it's not everyday that I hear a genuinely new argument. Thanks for sharing.

I guess I don’t find that argument very compelling. If you’re convinced the code branch can’t ever be taken, you also should be confident that it doesn’t need to be tested. This feels like chasing arbitrary 100% test coverage at the expense of safety. The code quality isn’t actually improved by omitting the checks even though it makes testing coverage go up.

In safety critical spaces you need to be able to trace any piece of a binary back to code back to requirements. If a piece of running code is implicit in code, it makes that traceability back to requirements harder. But I'd be surprised if things like bounds checks are really a problem for that kind of analysis.

Re: Why Is SQLite Coded In C

#88

Post by cogman10 »

anitil wrote: post #38
bfkwlfkjf wrote: post #16

> Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy. Huh it's not everyday that I hear a genuinely new argument. Thanks for sharing.

It's the sort of argument that I wouldn't accept from most people and most projects, but from Dr Hipp isn't most people and Sqlite isn't most projects.

It's a bad argument.

Certainly don't get me wrong, SQLite is one of the best and most thoroughly tested libraries out there. But this was an argument to have 4 arguments. That's because 2 of the arguments break down as "Those languages didn't exist when we first wrote SQLite and we aren't going to rewrite the whole library just because a new language came around."

Any language, including C, will emit or not emit instructions that are "invisible" to the author. For example, whenever the C compiler decides it can autovectorize a section of a function it'll be introducing a complicated set of SIMD instructions and new invisible branch tests. That can also happen if the C compiler decides to unroll a loop for whatever reason.

The entire point of compilers and their optimizations is to emit instructions which keep the semantic intent of higher level code. That includes excluding branches, adding new branches, or creating complex lookup tables if the compiler believes it'll make things faster.

Dr Hipp is completely correct in rejecting Rust for SQLite. Sqlite is already written and extremely well tested. Switching over to a new language now would almost certainly introduce new bugs that don't currently exist as it'd inevitably need to be changed to remain "safe".

Re: Why Is SQLite Coded In C

#89

Post by estebank »

selcuka wrote: post #67
lionkor wrote: post #50

Earlier quoted context omitted.

It's not like that, the compiler explicitly doesn't do compile-time checks here and offloads those to the runtime. Rust does not stop you from writing code that accesses out of bounds, at all. It just makes sure that there's an if that checks.

Ok, but you can still test all the branches in your source code and have 100% coverage. Those additional `if` branches are added by the compiler. You are responsible for testing the code you write, not the one that actually runs. Your compiler's test suite is responsible for the rest. By the same logic one could also claim that tail recursion optimisation, or loop unrolling are also dangerous because they change the…

> You are responsible for testing the code you write, not the one that actually runs.

This is not correct for every industry.

Re: Why Is SQLite Coded In C

#90

Post by NobodyNada »

jonahx wrote: post #34
bfkwlfkjf wrote: post #16

> Safe languages insert additional machine branches to do things like verify that array accesses are in-bounds. In correct code, those branches are never taken. That means that the machine code cannot be 100% branch tested, which is an important component of SQLite's quality strategy. Huh it's not everyday that I hear a genuinely new argument. Thanks for sharing.

So is the argument that safe langs produce stuff like: // pseudocode if (i >= array_length) panic("index out of bounds") that are never actually run if the code is correct? But (if I understand correctly) these are checks implicitly added by the compiler. So the objection amounts to questioning the correctness of this auto-generated code, and is predicated upon mistrusting the correctness of the compiler? But presuma…

> But (if I understand correctly) these are checks implicitly added by the compiler.

This is a dubious statement. In Rust, the array indexing operator arr[i] is syntactic sugar for calling the function arr.index(i), and the implementation of this function on the standard library's array types is documented to perform a bounds-check assertion and access the element.

So the checks aren't really implicitly added -- you explicitly called a function that performs a bounds check. If you want different behavior, you can call a different, slightly-less-ergonomic indexing function, such as `get` (which returns an Option, making your code responsible for handling the failure case) or `get_unchecked` (which requires an unsafe block and exhibits UB if the index is out of bounds, like C).

Post reply on HN