Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How do I inform Windows that I'm writing a binary file?

devblogs.microsoft.com

Post reply on HN
Print view View original

51–60 of 74 posts

Re: How do I inform Windows that I'm writing a binary file?

#51

Post by Joker_vD »

lmm wrote: post #40
Joker_vD wrote: post #36

Earlier quoted context omitted.

> At some point there's an interface where something that's part of the application asks something that's part of the OS to do something, and that interface had better be stable and well-specified. It's defined, and well-specified. > your C ABI Which is a C ABI. Borland's Turbo C and C++Builder used different ABI than Microsoft C compiler did. GCC for Windows used to use a third, entirely different ABI as well. The A…

> Which is a C ABI. Borland's Turbo C and C++Builder used different ABI than Microsoft C compiler did. GCC for Windows used to use a third, entirely different ABI as well. Sure, you can do that. Userspace code can use any ABI it wants, or none. But again, why, what do you gain? And regardless of whether it's "the" ABI or merely "a" ABI, that ABI presumably has a representation for strings and allows passing them arou…

> But again, why, what do you gain?

Performance? Codegen simplicity? Why, again, must one use the syscall ABI for anything that is not a syscall?

> that ABI presumably has a representation for strings and allows passing them around

In this particular case, the API operates with binary buffers, not text strings. Sure, you can go the VMS way, or even IBM way, and turn files from binary blobs into arrays of fixed-length records (that's why C's fwrite/fread have both num and sz arguments: some OSes literally can't write data any other way).

> At which point we're back at needing a way to write strings in an in-memory format to OS-standard files in the filesystem.

Yes? Some text editors converted LFs to NULs to work on the text in memory, and then they'd convert NULs back into LFs on writing to the disk (IIRC). Both emacs and vi don't store text in memory the way it's layed out in the file; they translate it when writing to the disk.

Again, why do you want the OS to get involved into any of this? It's not the OS's job, period, stop trying to make the world an even worse place.

Re: How do I inform Windows that I'm writing a binary file?

#52

Post by JdeBP »

foresto wrote: post #50
Isamu wrote: post #15

Earlier quoted context omitted.

>Linefeed (\n) is a single byte in DOS as well. In binary mode. In text mode if you printf(“Hello World\n”) you get CRLF because that’s how text works on DOS. Unix had the convention of only requiring the LF for text. And Unix didn’t have text/binary modes. That’s the compatibility hack on DOS. >These control codes go back to line printers. Back to teletypes even. Believe me, I go back to line printers.

I'm pretty sure that conversion was done by the C library, just as stated in the article. Not by DOS. ASCII 0x0A '\n' is always one byte*, and C library implementations for DOS would insert an ASCII 0x0D '\r' byte before it at output time if the C FILE stream had been opened in text mode. Note that printf(), which you use in your example, is a C library function that writes writes to a predefined text mode stream. So…

I wrote a C Standard library for MS/PC/DR-DOS. Your recollection is correct.

Re: How do I inform Windows that I'm writing a binary file?

#53

Post by JdeBP »

teraflop wrote: post #19
lmm wrote: post #14

Earlier quoted context omitted.

> you can of course use non-C languages to call the Win32 API. Or even directly using assembly code. Is that a supported/official API though? On Linux you "can" put your arguments in registers and trigger the system call interrupt directly, and I think Go programs even do this, but it's not the official interface and they reserve the right to break your program in future updates, at least in theory.

Sure. C has never been the only language supported on Windows. For instance, Delphi had a period of popularity for Windows application development, and AFAIK it has always used its own runtime library which is completely independent of the C runtime. Go does not trigger low-level system call interrupts on Windows. (It does that on Linux, but Windows syscall numbers are not stable even across minor Windows updates, so…

Don't forget the days when multiple C/C++ implementations from multiple vendors all came with their own runtime library DLLs, too.

Re: How do I inform Windows that I'm writing a binary file?

#54

Post by JdeBP »

wjholden wrote: post #47
jcranmer wrote: post #22

Earlier quoted context omitted.

There is a very unfortunate situation in Unix systems in that the library named 'libc' is serving several simultaneous different roles. One of those roles--what it is named for--is serving as the C standard library. The more important role is that the library also provides the implementation of a different standard API, the POSIX API, which is the main API used to access system details. There's also yet another role…

Not OP, but thank you for your sharing this. If you don't mind a follow-on question, I always hear people talk about the "runtime" in languages like Go and libraries like Tokio. What is that these runtimes are doing that you cannot get from the likes of libc and these Windows DLLs?

MSVCRT, the Microsoft Visual C/C++ Runtime library, is also 'the runtime library'. It was the runtime library for Microsoft's C/C++ compiler. In the days when there were multiple C/C++ implementations for Win32 (which still exist, if one is willing to dig up Watcom C/C++ or some such) there would be different runtime library DLLs for the different C/C++ vendors, even for different versions of their products.

Runtime libraries for C/C++ provide two general sets of stuff: the stuff mandated for the Standard C and Standard C++ libraries, and the stuff that is needed by the basic mechanics of the language.

The former is everything from abort() to wscanf(). The latter is a bunch of internal functions, calls to which the compiler inserts in order to do stuff. This is basically the split nowadays between UCRT and VCRUNTIME.

In the days of programming targetting the 80486SX without an 80487 present, for instance, every piece of floating point arithmetic was not a machine instruction but a call to a runtime library routine that did the floating point operation longhand using non-FPU instructions. Other runtime functionality over the years has included doing 32-bit or 64-bit integer arithmetic on 16-bit and 32-bit architectures where this was not a native word size, functions to do stack checking in the function perilogue, and functions to do C++ run-type type checking and exception processing.

This pattern is followed by other (compiled) programming languages. Naturally, the programming languages do not necessarily have any relation to the Standard C or Standard C++ libraries, nor do they generate code that needs the same helper functions for stuff as C/C++ code does. (But the situation is complicated by the POSIX API and the old C language bindings for the MS-DOS system call API, some of which another programming language might also allow program code to use.)

Re: How do I inform Windows that I'm writing a binary file?

#55

Post by foresto »

JdeBP wrote: post #52
foresto wrote: post #50

Earlier quoted context omitted.

I'm pretty sure that conversion was done by the C library, just as stated in the article. Not by DOS. ASCII 0x0A '\n' is always one byte*, and C library implementations for DOS would insert an ASCII 0x0D '\r' byte before it at output time if the C FILE stream had been opened in text mode. Note that printf(), which you use in your example, is a C library function that writes writes to a predefined text mode stream. So…

I wrote a C Standard library for MS/PC/DR-DOS. Your recollection is correct.

Thanks for confirming!

Re: How do I inform Windows that I'm writing a binary file?

#56

Post by Isamu »

foresto wrote: post #50
Isamu wrote: post #15

Earlier quoted context omitted.

>Linefeed (\n) is a single byte in DOS as well. In binary mode. In text mode if you printf(“Hello World\n”) you get CRLF because that’s how text works on DOS. Unix had the convention of only requiring the LF for text. And Unix didn’t have text/binary modes. That’s the compatibility hack on DOS. >These control codes go back to line printers. Back to teletypes even. Believe me, I go back to line printers.

I'm pretty sure that conversion was done by the C library, just as stated in the article. Not by DOS. ASCII 0x0A '\n' is always one byte*, and C library implementations for DOS would insert an ASCII 0x0D '\r' byte before it at output time if the C FILE stream had been opened in text mode. Note that printf(), which you use in your example, is a C library function that writes writes to a predefined text mode stream. So…

Agreed, I didn’t mean that DOS somehow converted it, this was a compatibility feature put into the C library.

Re: How do I inform Windows that I'm writing a binary file?

#57

Post by freedomben »

fragmede wrote: post #48
poizan42 wrote: post #29

Earlier quoted context omitted.

And if you switch the tty from "cooked" to "raw" mode then it doesn't do the conversion, and a CR just moves the cursor back to the start of the line and a LF just moves the cursor one line down.

Which is how you do the fun spinny icons on the command line without having to invoke ncurses!

You can also just use a \r directly without a \n. For example:

    spin='/-\|'
    while true; do 
      i=$(( (i+1) % 4 ))
      printf "\r${spin:$i:1} Working..."
      sleep 0.1
    done

Re: How do I inform Windows that I'm writing a binary file?

#58

Post by zinekeller »

lmm wrote: post #41
Joker_vD wrote: post #37

Earlier quoted context omitted.

All you need for that is the ability to read and write binary blobs to and from files, which Windows gives you, and to know what "text files" means for the other programs on that platform. Windows itself doesn't care for text much; but the other programs have a shared convention that ASCII text files have CRLF-separated variable-length lines of text, and Unicode text files store text in UTF16-LE, (including the CRLF…

The OS is the one providing the filesystem, it should define and support how it's used (including providing standard utilities for manipulating it, both from programs or by the operator) rather than leaving the programs to figure it out between themselves. (After all, if the text storage format didn't matter to the OS, why would we bother using the CRLF format on windows at all? I submit that third-party programs did…

As already stated multiple times here, the CRLF is actually the "correct" way (at least in the telex days, where CR and LF have actual meanings of "Return Carriage to home" and "Feed a new Line"), while the LF-only one is a Unix "hack"/abstraction (which was actually converted back into CRLF if fed to a telex or a terminal). It is not really a surprise that DOS, which was inspired by CP/M, simply copied what was supposed to be a physical signal. This is the reason the ASCII/ANSI code has a BEL indicator for ringing a bell. In short, CRLF is the way to handle newlines at the time that DOS was designed. You will expect that CRLF is the ending because that's how terminals work (unlike with the magicking Unix which smooshes two differing things into a character).

If you are writing a developer suite, whether you're Delphi developing for MS-DOS or Microsoft developing for Apple II, you kinda have the idea of how things should work (because you have the reference book for the platform, not the compiler/language). It is not the assumption that the OS provides abstraction for text - in thise days, everyone just implement it from scratch, really ("code page" was from literal code pages, where each character has a well-defined byte). This is manifested in command-line handling on Windows: the platform convention is that it is just a flat string, and the C runtime determines how to chop that up (MSVC and Intel C has historically disagreed heavily here) The abberation of Windows only having CRLF is because Unix-based designs took over the world: macOS is Unix, Linux was insiped by Unix, *BSD was Unix-derived.

Re: How do I inform Windows that I'm writing a binary file?

#59

Post by p_l »

zinekeller wrote: post #58
lmm wrote: post #41

Earlier quoted context omitted.

The OS is the one providing the filesystem, it should define and support how it's used (including providing standard utilities for manipulating it, both from programs or by the operator) rather than leaving the programs to figure it out between themselves. (After all, if the text storage format didn't matter to the OS, why would we bother using the CRLF format on windows at all? I submit that third-party programs did…

As already stated multiple times here, the CRLF is actually the "correct" way (at least in the telex days, where CR and LF have actual meanings of "Return Carriage to home" and "Feed a new Line"), while the LF-only one is a Unix "hack"/abstraction (which was actually converted back into CRLF if fed to a telex or a terminal). It is not really a surprise that DOS, which was inspired by CP/M, simply copied what was supp…

It still shows up in IETF-style textual network protocols, which evolved on non-Unix systems (HTTP, SMTP, etc.)

Re: How do I inform Windows that I'm writing a binary file?

#60

Post by criddell »

Isamu wrote: post #15
criddell wrote: post #12

Earlier quoted context omitted.

Linefeed (\n) is a single byte in DOS as well. I think you are talking about carriage return linefeed pair (CRLF or \r\n), These control codes go back to line printers. Linefeed advances the paper one line and carriage return moves the print head to the left.

>Linefeed (\n) is a single byte in DOS as well. In binary mode. In text mode if you printf(“Hello World\n”) you get CRLF because that’s how text works on DOS. Unix had the convention of only requiring the LF for text. And Unix didn’t have text/binary modes. That’s the compatibility hack on DOS. >These control codes go back to line printers. Back to teletypes even. Believe me, I go back to line printers.

Teletypes is what I meant to say, thank you. Line printers have no carriage return mechanism.
Post reply on HN