Live data from Hacker News

Hobby x86 kernel written with Zig

github.com

171–180 of 229 posts

Re: Hobby x86 kernel written with Zig

#171

Earlier quoted context omitted.

> including the vast majority of code editors by default Which ones exactly? Not really a problem I've encountered often except when someone tries to mix both spaces and tabs, and in general editors are built with the existence of this very common character in mind. People tend to hit the tab key to indent anyway, and one tab char meaning one level of indent is perfectly intuitive and allows users to individually con…

the problem i encounter is when you try to break a long line into multiple lines. if you want to use tabs and align the continuation, you should be mixing spaces and tabs. for example ('-' is tab, '.' is space): --function_with_lots_of_arguments(arg1, arg2, arg3, --................................arg4, arg5, arg6); it can be done, but a lot of editors get it wrong and it requires paying attention to the whitespace. o…

Some food for thought: https://youtu.be/ZsHMHukIlJY?t=633

For example, this is the best way to define functions with argument lists long enough not to fit on a single line:

  fn doThing(
      argument1,
      argument2,
      argument3,
  ) {
      
  }
Visually clear, doesn't have any spaces/tabs issues, produces minimal diffs when adding/removing/renaming arguments.

Re: Hobby x86 kernel written with Zig

#172
post #146
post #122

Earlier quoted context omitted.

> We have a pretty good answer to the question "is ASan sufficient to prevent memory safety problems in practice" That is not the question we're interested in answering, and elimination of all memory errors is no one's ultimate goal, certainly not at any cost. By definition, unsound techniques will let some errors through. The question is which approach leads to an overall safer program for a given effort, and soundn…

P.S. > Notice that the working approaches have something important in common: a strong system that, given certain assumptions, guarantees the lack of memory safety problems. That's a very good point and I'm not arguing against it. It's just that even if it's true -- and I'm more than willing to concede that it is -- it still doesn't answer the question, which is: what is the best approach to achieving a required leve…

> I think -- and I could be wrong -- that Rust sacrifices more than it has to just to achieve that soundness, by also paying for "zero-cost abstractions," which, for my taste, is repeating C++'s biggest mistake, namely sacrificing complexity for the appearance of high-level abstraction that may look convincing when you read the finished code (perhaps more convincing in Rust than in C++), but falls apart when you try to change it.

The argument here seems to be that there is can be no real abstraction in low-level languages, so there's no point providing language features for abstraction. The premise seems clearly false to me, because even C has plenty of abstraction. Functions are abstractions. Private symbols are abstractions. Even local variables are abstractions (over the stack vs. registers).

People often argue that Rust is too complicated for its goal of memory safety. It's easy to say that, but it's a lot harder to list specific features that Rust has that shouldn't be there. In fact, as far as I'm concerned Rust is an exercise in minimal language design, as the development of Rust from 0.6-1.0 makes clear (features were being thrown out left and right). Most of the features that look like they're there solely to support "zero-cost abstractions"—traits, for example—are really needed to achieve memory safety too. For instance, Deref is central to the concept of smart pointers, and, without smart pointers, users would have to manually write Arc/Rc/Box in unsafe code every time they wanted to heap-allocate something.

> Language simplicity goes a long way, even as far as sound formal verification is concerned. For example, there are existing sound static analysis tools that can guarantee no UB for C -- but not the complete C++, AFAIK -- with relatively little effort. It's not yet clear to me whether Zig, with its comptime, is simple enough for that, though.

The most important static analyzers used in industry today are Clang's sanitizers, which work on both C and C++. The most important such sanitizers actually work at the LLVM level, which means they work on Rust as well [1]! The days of having to write a compiler frontend for static analysis are long gone. We have excellent shared compiler infrastructure that makes it easy to write instrumentation that targets many low-level languages at once. (Even in the world of C, this is necessary. Plain old C99 is an increasingly marginal language, because the real important code, such as Windows and Linux kernels, are written in compiler-specific dialects of C, which means that a static analysis tool that isn't integrated with some popular compiler infrastructure will have limited usefulness anyway.)

> It is my great interest in software correctness, together with my personal aesthetic preferences, that has made me dislike language complexity so much and made me a believer in "when in doubt -- leave it out."

Again: easy to say, harder to specify specific Rust features you think should be removed.

[1]: https://github.com/japaric/rust-san

Re: Hobby x86 kernel written with Zig

#173

Earlier quoted context omitted.

> The ammount of unsafe blocks is pretty small compared to the rest of the code. Which means those parts can be reviewed more extensively. If anything your argument is pretty goofy. I've talked about this at length before, but just because you have 'less' `unsafe` code does not mean you code is any better off. Very few operations are actually `unsafe`, and you're still not allowed to break any of the (poorly or not d…

If you aren't making a safe abstraction around your unsafe code, you're supposed to mark the surrounding function "unsafe" as well, and document its expected constraints in a 'Safety' comment block. In fact, you're supposed to do the same for any piece of code that cannot provide safety guarantees about its use of 'unsafe' features. 'Plain' unsafe{ } blocks should only be used as part of building a safe abstraction.

> If you aren't making a safe abstraction around your unsafe code, you're supposed to mark the surrounding function "unsafe" as well, and document its expected constraints in a 'Safety' comment block.

Sure, but now we're back to the issue that it's unclear what constraints `unsafe` code actually has to hold, meaning ensuring your abstraction is safe is just about impossible to do with certainty (And OS kernel code is definitely going to hit on a lot of those corner cases). You may think you have a safe interface, only to find out later you're relying on internal details that aren't guaranteed to stay the same between compiler versions or during optimizations.

With that said, while I agree with what you're proposing about marking surrounding code "unsafe", it leads to lots of strange cases and most people get it wrong or will even disagree with this proposal completely. For example, it can lead to cases where you mark a function `unsafe` even though it contains nothing but "safe" code. And at that point, it's up to you to determine if something is actually "safe" or "unsafe", meaning the markings of "safe" and "unsafe" just become arbitrary choices based on what you think "unsafe" means, rather than marking something that actually does one of the "unsafe" operations.

One of the best examples is pointer arithmetic. It's explicitly a safe operation, but it's also the first things most people would identify as being "unsafe", even though it is dereferencing that is the "unsafe" operation. Ex. You could easily write slice::from_raw_parts without using any `unsafe` code at all, it just puts a pointer and length together into a structure (it doesn't even need to do arithmetic!). It's only marked `unsafe` because it can break other pieces of "safe" code that it doesn't use, but it itself is 100% safe. You could just as easily argue the "other code" should be the `unsafe` code, since it's what will actually break if you use it incorrectly.

Perhaps the biggest annoyance I have is that the official Rust documentation pretty much goes against the idea you presented, saying

> People are fallible, and mistakes will happen, but by requiring these four unsafe operations to be inside blocks annotated with unsafe you’ll know that any errors related to memory safety must be within an unsafe block. Keep unsafe blocks small; you’ll be thankful later when you investigate memory bugs.

Which is just incorrect - memory safety issues are likely to be due to the "safe" code surrounding your unsafe code - when you're writing C code, the bug isn't that you dereferenced NULL or an OOB pointer, the bug is the code that gave you that pointer in the first place, and in Rust that code is likely to all be "safe". Point being, most of the advice on keeping your unsafe blocks small just leads to people making silly APIs that can be broken via the safe API they wrap (Even in weird ways, like calling a method containing only safe code), and unfortunately there are lots of subtle ways you can unintentionally break you Rust code that most people aren't going to have any idea about.

Re: Hobby x86 kernel written with Zig

#174

Earlier quoted context omitted.

yep, 100% I personally think the Rust community has missed the biggest advantage of unsafe, and that's tooling. Imagine an IDE that could highlight and warn when touching state that is accessed in unsafe blocks, or when state being accessed in unsafe blocks isn't being properly tested. Or tools that kick off notification for a required code review because code that touches state used by an unsafe block got changed. T…

Those tools exist or are being worked on! Servo (last I checked) uses a “unsafe code was added or modified in this PR, please review extra carefully” bot, and Miri is on its way to becoming a fantastic tool.

That's good to know.

The Rust community can be a bit... fanatical, lets say. I've seen so many people argue that the unsafe keyword by itself makes rust so much safer than alternatives, when really safe code can cause unsafe code to explode. So you end up writing modules to protect the unsafe code, which is what you do in any other language as well.

Which means the unsafe keyword is valuable, just not nearly as valuable as I've seen a lot of people claim.

Now the unsafe keyword combined with the sort of tooling I've mentioned? That to me is a killer combination. Unsafe isn't a silver bullet, it still requires work, but it enables tools to make that work immensely easier to deal with.

Re: Hobby x86 kernel written with Zig

#175

Earlier quoted context omitted.

The Rust code you wrote seems to be extremely heavy on boilerplate. A couple of macros could have drastically reduced the line count. I don't think what Linus wrote in 2004 on C++ has much relevance for 2020 Rust, especially for a new kernel that doesn't have a ton of contributors. Most of his complaints seem to be on how C++ abstractions can make it hard to review unknown code and how 2004 C++ code bases often had e…

Small nitpick: Google isn't using Rust for the Fuchsia kernel. The kernel, Zircon, is written entirely in C and is based on Little Kernel. Google is using Rust for some userspace drivers (Fuchsia being a micro-kernel).

> The kernel, Zircon, is written entirely in C

No, it's written in C++[1]. They're very different languages.

That said, Google seems to think C++ has something to offer to all kinds of development. It seems to work for them, but they are a heavily C++ shop. I wouldn't read too much out of their use of C++ anywhere; it'd be more surprising and interesting to see them use anything else.

[1]: https://fuchsia.googlesource.com/fuchsia/+/master/zircon/ker...

Re: Hobby x86 kernel written with Zig

#176
post #146

Earlier quoted context omitted.

P.S. > Notice that the working approaches have something important in common: a strong system that, given certain assumptions, guarantees the lack of memory safety problems. That's a very good point and I'm not arguing against it. It's just that even if it's true -- and I'm more than willing to concede that it is -- it still doesn't answer the question, which is: what is the best approach to achieving a required leve…

> I think -- and I could be wrong -- that Rust sacrifices more than it has to just to achieve that soundness, by also paying for "zero-cost abstractions," which, for my taste, is repeating C++'s biggest mistake, namely sacrificing complexity for the appearance of high-level abstraction that may look convincing when you read the finished code (perhaps more convincing in Rust than in C++), but falls apart when you try…

> The argument here seems to be that there is can be no real abstraction in low-level languages, so there's no point providing language features for abstraction.

My argument is that low-level languages allow for low abstraction, i.e. there's little that they can abstract over, where by abstraction I mean hide internal implementation details in a way that when they change the consumer of the construct, or "abstraction", does not need to change; if it does, then the construct is not an abstraction. With "zero-cost abstraction," C++/Rust offer constructs that syntactically appear as if they were abstractions (e.g. static vs dynamic dispatch; subroutine vs. coroutine call), but in reality aren't. I am not aware of any other language (unless Ada has changed considerably since I last used it in the early '00s) that values this idea to such a great extent.

The things you mentioned are abstractions only in the sense that the user doesn't need to know how the compiler implements them; in that respect, every language construct, including `if` (e.g. in Java, not every if is compiled into a branch) is an abstraction. I speak of the language's ability to allow users to abstract, and in that regard all low-level languages provide for poor abstraction. Without a JIT, the caller needs to know the calling convention; without a tracing GC the caller needs to know how the memory pointed to by a returned value is to be deallocated. The question is how much you try to make sure that all this knowledge is implicit in the syntax.

> People often argue that Rust is too complicated for its goal of memory safety.

I didn't know people often say that. I said it, and I'm not at all sure that's the case. I think that Rust pays far too heavy a price in complexity. It's too heavy for my taste whether or not it's all necessary for sound memory safety, but if it isn't, all the more the shame.

> Most of the features that look like they're there solely to support "zero-cost abstractions"—traits, for example—are really needed to achieve memory safety too.

OK, so I'll take your word for it and not say that again.

> The most important static analyzers used in industry today are Clang's sanitizers

I'm talking about sound static analysis tools, like Trust-in-Soft, that can guarantee no UB in C code. I think that particular tool might support some subset of C++, but not all of it. The sanitizers you mention rely on concrete interpretation (aka "dynamic") and are, therefore, usually unsound. Sound static analysis requires abstract interpretation, of which type checking and type inference are special cases. Just as you can't make all of Rust's guarantees by running Rust's type-checker on LLVM bytecode, so too you cannot run today's most powerful sound static analysis tools -- that are already strong enough to absolutely guarantee no UB in C st little cost -- on LLVM bytecode; they require a higher-level language. Don't know about tomorrow's tools.

> Again: easy to say, harder to specify specific Rust features you think should be removed.

I accept your claim. In general, I don't like to isolate language features; it's the gestalt that matters, and it's possible that once Rust committed to sound memory safety everything else followed. But let me just ask: are macros absolutely essential?

Re: Hobby x86 kernel written with Zig

#177

Earlier quoted context omitted.

In general you'll find that zig is easier to read than Rust (see the first version of this project in Rust [0]) because it's a simpler language. For kernel programming this is even more so the case: * zig has native support for arbitrary sized integers. In Rust I used to do bitshifts, Now I just have a packed struct of u3/u5/u7 whatever (see `src/pci/pci.zig`). Of course Rust has a bitflags crate but I didn't find it…

>"zig has native support for arbitrary sized integers." I apologize if this is a naive question. Might you or someone else elaborate on where arbitrary sized integers are used or necessary in kernel programming? Don't they all ultimately need to get padded out for the CPU registers to work with them anyway?

Packed binary structures. Common in device register space (the author alludes to PCI driver(s)).

Re: Hobby x86 kernel written with Zig

#178

Earlier quoted context omitted.

If you aren't making a safe abstraction around your unsafe code, you're supposed to mark the surrounding function "unsafe" as well, and document its expected constraints in a 'Safety' comment block. In fact, you're supposed to do the same for any piece of code that cannot provide safety guarantees about its use of 'unsafe' features. 'Plain' unsafe{ } blocks should only be used as part of building a safe abstraction.

> If you aren't making a safe abstraction around your unsafe code, you're supposed to mark the surrounding function "unsafe" as well, and document its expected constraints in a 'Safety' comment block. Sure, but now we're back to the issue that it's unclear what constraints `unsafe` code actually has to hold, meaning ensuring your abstraction is safe is just about impossible to do with certainty (And OS kernel code is…

> Sure, but now we're back to the issue that it's unclear what constraints `unsafe` code actually has to hold

The Rust Nomicon actually documents these constraints for each 'unsafe' operation. Code that can ensure that these constraints hold can be regarded as 'safe' and rely on an unsafe{ } block. Code that can't, should be marked unsafe and document its own constraints in turn.

> You may think you have a safe interface, only to find out later you're relying on internal details

If you're relying on internal details, you're most likely doing it wrong, even by the standards of 'unsafe'. There are ways to nail down these details where required, at which point they're not even "internal" details anymore, but this has to be opted-into explicitly, for sensible reasons.

> It's only marked `unsafe` because it can break other pieces of "safe" code that it doesn't use, but it itself is 100% safe. You could just as easily argue the "other code" should be the `unsafe` code, since it's what will actually break if you use it incorrectly.

No, "other code" should not be marked unsafe because this would mean that relying on slices is inherently unsafe. Which is silly; the whole point of a "slice" type, contrasted with its "raw parts", is in the guarantees it provides. This is why slice::from_raw_parts is unsafe: not because if what it does, but what it states about the result.

> Which is just incorrect - memory safety issues are likely to be due to the "safe" code surrounding your unsafe code

This is a matter of how you assign "blame" for a memory safety error. It's definitely true however, that the memory unsafe operations play a key role, and I assume that's the point that these docs are making. I agree that people shouldn't be creating faulty abstractions around "unsafe" blocks, but the reason we can even identify this as an issue is because we have "unsafe" blocks in the first place!

Re: Hobby x86 kernel written with Zig

#179
post #15
post #9

Earlier quoted context omitted.

I'm curious too, I'm quite familiar with Rust and never written any Zig in my life so I went digging through the source and I find the syntax remarkably similar for the most part. The only thing that stood out is that apparently you can drop the braces for single-line `if` bodies like in C whereas Rust makes them always mandatory but I'm firmly on Rust's side on this one. The part where Rust can get really messy is w…

If you want more syntax weirdness: tab characters are illegal.

Honestly, I like tabs for indent in C, but still support this language choice by Zig. It's hard to do tabs wrong if you don't allow tabs at all. Python3 went part of the way to disallowing tabs completely, but instead only disallowed mixed tabs and spaces. (And in Python, it's a bigger problem as the indentation level is significant to the syntax.)

Re: Hobby x86 kernel written with Zig

#180
post #50
post #15

Earlier quoted context omitted.

If you want more syntax weirdness: tab characters are illegal.

Even weirder than that, it intentionally fails on \r\n newline, so windows text files straight up don't work by default.

Good! Even Windows Notepad, the extreme example of "not a code editor editor," supports \n newlines now: https://devblogs.microsoft.com/commandline/extended-eol-in-n...
Post reply on HN