Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

161–170 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#161

Post by spoiler »

hnal943 wrote: post #118
markeganfuller wrote: post #100

Earlier quoted context omitted.

I'm looking at my English keyboard and it has Alt Gr...

Living in the US, I've never heard of an AltGr key until this discussion.

I'm from Croatia, and we have the AltGr key.

However, I discovered that Alt + Control = AltGr when I needed to use it at work[1], so it's simply a shortcut, I think.

[1]: We have (I recently switched to an UK keyboard, because it suits me better) keyboards at work, because the Croatian (all slavic languages, to be honest) is horrendously counterproductive for programming. Google the layout, and you'll realise why. An example: You need to press AltGr+B for `{` (if I remember correctly).

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#162

Post by r00fus »

jmulho wrote: post #156
dxbydt wrote: post #57

Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.

There is another: fixed width. It doesn't cause grief.

Sure it does - any change in supported field length requires a schema change.

You trade simplicity of parsing for rigidity of schema.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#163

Post by dhimes »

thwarted wrote: post #73
scelerat wrote: post #44

Earlier quoted context omitted.

ASCII 0-31 are called "control" characters so it should come as no surprise that you could type them using the Control key. Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Control-C and Control-D, End of Text and End of…

Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Vim has bindings for some control key combos, which is why you can't type them directly. I can type control-_ without issue, but to get an ASCII 30 (RS) to show up, I have…

That's a terrific hint. So the ctrl- =

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#164

Post by mantrax3 »

Pxtl wrote: post #26
mantrax3 wrote: post #13

Earlier quoted context omitted.

For CSV forbidding commas in data is not practical. For ASCII delimiters, forbidding ASCII delimiters in data is practical. Sure - you can't, say, nest ASCII tables into one another due to this limitation. But for simple structure, it doesn't hurt to have ASCII separators in the toolbox. The only big problem I see is that they're rendered as invisible characters, which will make debugging harder. If we wouldn't have…

> Sure - you can't, say, nest ASCII tables into one another due to this limitation. In hindsight it's too bad we don't have similar characters that follow a more sexpr-ish layout - say, ListStart, ListEnd, and Delimiter. Then you could tree them endlessly. If you wanted to be really fancy you could add an "assignmentSeparator" character to officially bless key-value-pairs and encompass a nice JSON-ish format, but Lis…

> In hindsight it's too bad we don't have similar characters that follow a more sexpr-ish layout - say, ListStart, ListEnd, and Delimiter. Then you could tree them endlessly.

Well, we do. Since basically no one else assign semantics to those abandoned ASCII separators anymore, you're free to use the Unit separator / Record separator as List Start / List End etc. :)

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#165

Post by flux_w42 »

markeganfuller wrote: post #100
Pxtl wrote: post #37

Earlier quoted context omitted.

> Precisely what makes them valuable is their difficulty to type. Again, if they'd caught on you'd imagine there would be some eventual convention in text-editors for what keybind would be used to enter them. Too bad the AltGr key (intended for entering rarely-used glyphs) doesn't appear on pure-English keyboards.

I'm looking at my English keyboard and it has Alt Gr...

Here in Belgium we have to AltGr-the hell out of our keyboard during development. These characters can only be entered using the AltGr key on a Belgian Azerty keyboard (be-latin1): '|' '@' '#' '^' '{' '}' '[' ']' '\' and '~' Its not realy a problem, as long as you're used to it :-)

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#166

Post by JadeNB »

chrisBob wrote: post #34
binarymax wrote: post #9

Protip - if it doesnt appear on keyboards, you can use ALT+DDD (DDD being 000 to 255) to enter a control character. For those on windows, drop into a command prompt and hold ALT while pressing 031 on the numpad. You will see it produce a ^_ character.

On what computer? Is this windows only? Now that I use a mac this might be the only thing I miss from windows computers.

As htp mentioned above (https://news.ycombinator.com/item?id=7474951), you can do this on a Mac by using Unicode Hex Input as your input source.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#167

Post by JadeNB »

rcthompson wrote: post #85

If you use unprintable characters in your file, it's no longer human-editable as text. It may as well be XML (i.e. technically text-based but not practically human-readable).

> It may as well be XML (i.e. technically text-based but not practically human-readable).

Is this really a standard complaint about XML? I thought the main complaint was that it wasn't human-writeable. I wouldn't want to read novels in XML, but I've never had a problem opening up an XML file in a text editor to get at bits of it.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#168

Post by JadeNB »

sp332 wrote: post #61
Balgair wrote: post #32

Haha! Oh wow, I just finished a project dealing with this exactly. The obvious problem is that most editors make dealing with the non-standard keyboard keys very difficult. As a consequence, most programs (Python, MatLab, etc) really don't like anything below 0x20. I was reading in binary through a serial port, and then storing data for records and processing. Any special character got obliterated in the transfer to…

C'est la vie ;)

CES la vie ;-)

http://rockhealth.com/2014/01/rock-weekly-ces-la-vie

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#169

Post by dfc »

userbinator wrote: post #160
JoshTriplett wrote: post #139

Earlier quoted context omitted.

It's not just the nonchalant mindset; it's the thought that because you pick something you don't expect to form part of your input domain, you don't have to escape. Either you have to actually restrict your input domain, or you need escaping.

This, very much this! Escaping should always be a consideration. Not thinking about it, thinking "it'll never happen", etc. is what leads to things like HTML and SQL injection vulnerabilities. If you're inputting or outputting data in any format, always keep in mind things like "what are the delimiters? What if the data in the input/output contains them?"

How is this any different than reading ASN.1 data and not worrying about the size of integers?

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#170

Post by com2kid »

Pxtl wrote: post #111
derefr wrote: post #92

Earlier quoted context omitted.

You know what's nicer than delimiting beginnings and ends of things? Length prefixing. Protocol message formats and data encoding formats both already know what they're going to say before they say it, and so know its octet length. The only reason to use delimiters, ever, is for user-modifiable data (e.g. source code) where you might want to insert or delete characters and have the containing block remain valid. ---…

The problem: I have never encountered length-prefixed data. Ever. Every data interchange file I've ever dealt with has been either delimited or fixed-width fields (and the widths are not defined anywhere in the file).

In contrast, I just got done designing an internal protocol today that has a length prefix.

My team pretty much length prefixes everything. :)

Post reply on HN