Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How do I inform Windows that I'm writing a binary file?

devblogs.microsoft.com

Post reply on HN
Print view View original

61–70 of 74 posts

Re: How do I inform Windows that I'm writing a binary file?

#61

Post by pjc50 »

wjholden wrote: post #47
jcranmer wrote: post #22

Earlier quoted context omitted.

There is a very unfortunate situation in Unix systems in that the library named 'libc' is serving several simultaneous different roles. One of those roles--what it is named for--is serving as the C standard library. The more important role is that the library also provides the implementation of a different standard API, the POSIX API, which is the main API used to access system details. There's also yet another role…

Not OP, but thank you for your sharing this. If you don't mind a follow-on question, I always hear people talk about the "runtime" in languages like Go and libraries like Tokio. What is that these runtimes are doing that you cannot get from the likes of libc and these Windows DLLs?

The Go runtime provides its internal scheduler for "goroutines" (roughly "green threads", threads which are managed in-process and not by the OS for lower overhead), a garbage collector, and so on, which in turn are tightly integrated into the language.

Re: How do I inform Windows that I'm writing a binary file?

#62

Post by pjc50 »

lmm wrote: post #9
ChrisSD wrote: post #5

Earlier quoted context omitted.

The Win32 API. E.g. using WriteFile to write files ( https://learn.microsoft.com/en-us/windows/win32/api/fileapi/... ) It wasn't until fairly recently that the C runtime was stably shipped with Windows. Previously you had to install the correct version of the C library alongside your application.

> The Win32 API. E.g. using WriteFile to write files ( https://learn.microsoft.com/en-us/windows/win32/api/fileapi/ ...) Which is called from what, if not C? Does windows really offer no API for writing text (rather than bytes) to files? Or does it rely on the application developer to manage line endings in their own code? Neither of those sounds very developer-friendly.

This discussion is rather weird because hardly anyone writes raw C programs in Windows any more, and especially not Microsoft. Generally they expect developers to have moved to C++ or C#, both of which offer richer APIs.

The real fragmentation is not CRLF but the transition to system level UTF-16 support, involving all sorts of macros and duplicating almost every OS API function into FooW() and FooA() variants.

Re: How do I inform Windows that I'm writing a binary file?

#63

Post by account42 »

ChrisSD wrote: post #5
lmm wrote: post #4

The article seems to be taking the position that the C runtime library is not part of "Windows", which feels like a rather odd view to me. What is the stable API that Windows offers to application developers if not that?

The Win32 API. E.g. using WriteFile to write files ( https://learn.microsoft.com/en-us/windows/win32/api/fileapi/... ) It wasn't until fairly recently that the C runtime was stably shipped with Windows. Previously you had to install the correct version of the C library alongside your application.

Windows has shipped msvcrt.dll for a long time, even if it isn't officially supported for application use (and outdated, but that often doesn't matter).

Re: How do I inform Windows that I'm writing a binary file?

#64

Post by account42 »

Pannoniae wrote: post #17
lmm wrote: post #14

Earlier quoted context omitted.

> you can of course use non-C languages to call the Win32 API. Or even directly using assembly code. Is that a supported/official API though? On Linux you "can" put your arguments in registers and trigger the system call interrupt directly, and I think Go programs even do this, but it's not the official interface and they reserve the right to break your program in future updates, at least in theory.

Calling win32 from other languages is supported, calling it from assembly is supported (as long as you use the calling convention properly, obviously), using ntdll to bypass the win32 API is not supported. Basically on Linux the syscalls are the equivalent of Win32 except much narrower in scope.

> using ntdll to bypass the win32 API is not supported

But it is sometimes required to do things properly.

Re: How do I inform Windows that I'm writing a binary file?

#65

Post by qbane »

wahern wrote: post #6
qbane wrote: post #3

Earlier quoted context omitted.

It's C library taking care of the "b" part for you according to the article.

It's the other way around. It's the C runtime that treats text ("t") mode differently, because the C standard specifies \n as a line delimiter but the Windows convention is \r\n. In text mode C stdio translates between \n and \r\n. In binary mode it does no translation.

Note that when neither is supplied, the text mode is the default. This is why I said that it is the C library handling the "b" flag.

Re: How do I inform Windows that I'm writing a binary file?

#66

Post by satiated_grue »

Isamu wrote: post #15
criddell wrote: post #12

Earlier quoted context omitted.

Linefeed (\n) is a single byte in DOS as well. I think you are talking about carriage return linefeed pair (CRLF or \r\n), These control codes go back to line printers. Linefeed advances the paper one line and carriage return moves the print head to the left.

>Linefeed (\n) is a single byte in DOS as well. In binary mode. In text mode if you printf(“Hello World\n”) you get CRLF because that’s how text works on DOS. Unix had the convention of only requiring the LF for text. And Unix didn’t have text/binary modes. That’s the compatibility hack on DOS. >These control codes go back to line printers. Back to teletypes even. Believe me, I go back to line printers.

Yes, and the order is important.

Sending a carriage return and linefeed to a TTY 33 and then printing works fine. Doing them in the opposite order, if the carriage is to the right of the page, will result in a linefeed (platen rotation) happening quickly, then the carriage starting to move left to the beginning of the next line, and then the next printed character will print wherever the carriage happens to be at the time - not yet to the left. So you will be missing a character at the beginning of the line because it's in between the two lines in an unexpected column-ish.

I have run into (in my mind, "hipster") code where the programmer for some reason reversed the order of CR and LF.

Re: How do I inform Windows that I'm writing a binary file?

#67

Post by skissane »

Isamu wrote: post #7

Going all the way back to the earliest C compilers on DOS. There was a decision made to make “\n” just work on DOS for portability of Unix programs, and to make the examples from the C programming book just work. But in Unix “\n” is a single byte, and in DOS it is 2. So they introduced text and binary modes for files on DOS. Behind the scenes the library will handle the extra byte. This is not necessary in Unix. I us…

> So they introduced text and binary modes for files on DOS

It actually long predates DOS

C stdio is descended from Mike Lesk’s “portable IO package” (original release circa 1973). Bell Labs ported their C compiler from Unix to Honeywell GCOS and IBM S/370 mainframes. Mainframes handle text files very differently from how Unix systems do-it is much more complex than simply changing the newline character. So in Lesk’s package, the mode parameter to copen() told you whether the file was text or binary. copen() was renamed to fopen(), and the character to indicate binary mode was changed from “i” to “b”, and hence stdio

stdio has always had text-vs-binary file distinction, on some platforms (such as Unix) it has always been a no-op, on others it hasn’t

https://archive.org/details/lesk-iolib

Re: How do I inform Windows that I'm writing a binary file?

#68

Post by xenadu02 »

wjholden wrote: post #47
jcranmer wrote: post #22

Earlier quoted context omitted.

There is a very unfortunate situation in Unix systems in that the library named 'libc' is serving several simultaneous different roles. One of those roles--what it is named for--is serving as the C standard library. The more important role is that the library also provides the implementation of a different standard API, the POSIX API, which is the main API used to access system details. There's also yet another role…

Not OP, but thank you for your sharing this. If you don't mind a follow-on question, I always hear people talk about the "runtime" in languages like Go and libraries like Tokio. What is that these runtimes are doing that you cannot get from the likes of libc and these Windows DLLs?

In more general terms "runtime" means the userspace facilities above and beyond the standard system APIs (which often translate directly into syscalls into the kernel). It is a set of agreed-upon data structures and functions for operating on them and often includes facilities and patterns for extending the system with your own data structures.

For example the C runtime has a notion of what a "string" is: it's binary layout in memory and the conventions around it (e.g. an array of utf-8 bytes terminated by a null).

A runtime can be very thin or very complex. The dotnet or Java runtimes are massive by comparison. To the point they generally JIT the intermediate language to produce executable code (whether ahead of time or on-the-fly). Go's runtime has its own notion of threading built on top of the system notion of threads.

A self-contained static binary embeds any runtime implementations it needs into its own binary so it is still using runtime facilities but needs no external libraries.

A standalone or "bare" program can mean one that is built using only syscall primitives. Of course that can be taken further: you can build a true baremetal program that is designed to be copied into memory by the bootloader so it runs without a kernel or OS underneath it. This is, after all, what an OS kernel is: just code built such that the bootloader can jump to a fixed (or designated in metadata) address, handing off a pointer to info about the hardware (such as a DeviceTree) in memory and that's it.

In the early PC days BIOS was basically a set of functions built-in to the hardware (or more often flashed onto EEPROM). More or less a minimal sort of runtime + device drivers that knew how to read keyboard input, print characters to the screen, etc.

Almost everything is built on abstractions. In modern systems EFI or equivalents is a form of runtime + device drivers for early boot and the kernel. The kernel forms that for userspace. And a userspace language runtime can be something like a mini-OS for the code it runs. Going the other direction CPUs themselves are much more like a collection of networked PCs than you might expect.

Re: How do I inform Windows that I'm writing a binary file?

#69

Post by lmm »

Joker_vD wrote: post #51
lmm wrote: post #40

Earlier quoted context omitted.

> Which is a C ABI. Borland's Turbo C and C++Builder used different ABI than Microsoft C compiler did. GCC for Windows used to use a third, entirely different ABI as well. Sure, you can do that. Userspace code can use any ABI it wants, or none. But again, why, what do you gain? And regardless of whether it's "the" ABI or merely "a" ABI, that ABI presumably has a representation for strings and allows passing them arou…

> But again, why, what do you gain? Performance? Codegen simplicity? Why, again, must one use the syscall ABI for anything that is not a syscall? > that ABI presumably has a representation for strings and allows passing them around In this particular case, the API operates with binary buffers, not text strings. Sure, you can go the VMS way, or even IBM way, and turn files from binary blobs into arrays of fixed-length…

> Performance? Codegen simplicity? Why, again, must one use the syscall ABI for anything that is not a syscall?

If you can figure out an ABI that gives you significant advantages, sure, knock yourself out. But given that you're going to have to implement the syscall one anyway, if there's no compelling reason to use a different one then why make things more complicated?

> Again, why do you want the OS to get involved into any of this? It's not the OS's job, period

Again, why have a filesystem if you're not going to have any standardised structure for how to use it? Why have an OS at all if you're not going to give programs ways to interact with each other? The OS owns the filesystem, it should also define how it's used.

> stop trying to make the world an even worse place

Right back at you.

Re: How do I inform Windows that I'm writing a binary file?

#70

Post by zinekeller »

p_l wrote: post #59
zinekeller wrote: post #58

Earlier quoted context omitted.

As already stated multiple times here, the CRLF is actually the "correct" way (at least in the telex days, where CR and LF have actual meanings of "Return Carriage to home" and "Feed a new Line"), while the LF-only one is a Unix "hack"/abstraction (which was actually converted back into CRLF if fed to a telex or a terminal). It is not really a surprise that DOS, which was inspired by CP/M, simply copied what was supp…

It still shows up in IETF-style textual network protocols, which evolved on non-Unix systems (HTTP, SMTP, etc.)

MTA-STS, a very recent standard (RFC 8461), only allows CRLF as the line terminator (to the chagrin of *nix lovers, and to the fact that a majority of mail systems are being operated on *nix systems)
Post reply on HN