The distinction you are missing is what is generally meant by (and is actually valuable about) plain text is, "simultaneously human and machine readable data".
Many "plain text" formats, like markdown or INI files or json, actually have very strict formatting requirements and character set constraints, but the value-add comes from a human's ability to examine the on-file-system object, examine it with well-known and reliable tools (grep, awk, text editor, etc.), figure out what it's supposed to mean, then feed it to the machine, and compare the machine's behavior with their expectations.
With non-human-readable data, this is much harder, you pretty much need a tool to convert the binary data to readable text to distinguish between "my program is broken" and "my program works but is getting bad input."
Note that even structured ASCII can still make this hard, XML is nominally human-readable, but as a practical matter this can be difficult.