Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

151–160 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#151

Post by csixty4 »

jrochkind1 wrote: post #142
danielweber wrote: post #129

Earlier quoted context omitted.

I was disappointed that Ctrl-S and Ctrl-Q don't correspond (at least on the first ASCII chart I googled) to anything, because Ctrl-S stops text on my Unix terminals the exact same way it did on my brother's Apple ][e in 1984.

> I was disappointed that Ctrl-S and Ctrl-Q don't correspond (at least on the first ASCII chart I googled) to anything I'm pretty sure every possible combination of 7-bits is assigned SOME name in ascii, and Ctrl-any-case-insensitive-letter is a defined 7bit value. This chart says control-s is `DC3 (Device Control, X-OFF)` and control-q is `DC1 (Device Control, X-ON)`. Yup, that's what they do alright. http://www.uni…

For anyone wondering, X-ON and X-OFF are flow control characters. A device sends X-OFF to say "my buffer is full, stop sending data" and X-ON when it's ready to resume.

On most Unix terminals, these have the effect of pausing and resuming the display when a bunch of data is scrolling by.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#152

Post by louthy »

hnal943 wrote: post #118
markeganfuller wrote: post #100

Earlier quoted context omitted.

I'm looking at my English keyboard and it has Alt Gr...

Living in the US, I've never heard of an AltGr key until this discussion.

Living in the UK, I've always seen it and for some reason never questioned what the Gr meant!

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#153

Post by l-p »

radicaledward wrote: post #127
dxbydt wrote: post #57

Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.

I was surprised to see you list tsv as more common than csv. I encounter csv's on a pretty regular basis, but I don't think I've had to parse a tsv in the past 3 or 4 years. As a junior web developer, I don't have much experience though. 9 times out of 10, the csv is coming from or going to Excel, or a system that was designed to support Excel. If you don't mind my asking, what types of data do you regularly work wit…

Your comment disturbs me a little… One of my gripes with Excel was that it imported and produced TSV data by default when you asked for CSV.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#154

Post by userbinator »

dxbydt wrote: post #57

Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.

I think CSV is more common, since it allows for escaping whereas with TSV I don't believe there is any method of escaping.

I had to deal with a system using TSV once, with that "feature", and that point made it so that we had to do the escaping at some higher level, with \t and \\.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#155

Post by 3JPLW »

stingraycharles wrote: post #120
mbauman wrote: post #28

Earlier quoted context omitted.

It's a shame. The solution is in the charset, but tools never developed to use it so we don't use it. But I'd wager that if tools had historically supported them, then the situation would be no different than it is with tab. There are representational glyphs for tab, return, and others (⇥, ↵), and editors can show them in 'show whitespace' modes. There could be representational glyphs for these control characters, to…

Is this a problem that's solved with a "movement" ? I'm willing to contribute parsers and perhaps a text editor plugin if other people are willing to chip in. But indeed, this is a problem that's solvable by making the tools support it. Since most of the tools are OSS, this is mostly solvable.

You're missing my point. If this were a "solved problem," then it would be no better than the status quo with TSV. Tabs are a similar kind of control character, except they can be typed and displayed easily. And TSV files are a mess because tabs end up in the fields themselves. It's a catch-22. (I'm exaggerating; this would still be somewhat better since there will never be a big key on your keyboard dedicated to these ASCII values.)

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#156

Post by jmulho »

dxbydt wrote: post #57

Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.

There is another: fixed width. It doesn't cause grief.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#157

Post by Nilzor »

louthy wrote: post #152
hnal943 wrote: post #118

Earlier quoted context omitted.

Living in the US, I've never heard of an AltGr key until this discussion.

Living in the UK, I've always seen it and for some reason never questioned what the Gr meant!

so.. what does it mean?

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#158

Post by mjn »

mbreese wrote: post #51
Pxtl wrote: post #45

Earlier quoted context omitted.

I don't think CSV is a rigidly-defined format - I'm sure some implementations will happily accept spaces between the comma and the opening quote.

This is one reason why I prefer tab delimited files... The format is pretty simple with few edge cases. There really only one caveat to worry about - fields with tab characters. And that's extremely rare in my field. CSV on the other hand has a few different variations.

That's one reason I prefer TSV... the other one is that you can't use any of the regular Unix tools reliably on CSVs.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#160

Post by userbinator »

JoshTriplett wrote: post #139
dfc wrote: post #42

Earlier quoted context omitted.

The thing that leads to "lovely security bugs" is the nonchalant mindset; it has nothing to do with the simple text format. The same attitude paired with ASN.1 data has caused just as many vulnerabilities.

It's not just the nonchalant mindset; it's the thought that because you pick something you don't expect to form part of your input domain, you don't have to escape. Either you have to actually restrict your input domain, or you need escaping.

This, very much this!

Escaping should always be a consideration. Not thinking about it, thinking "it'll never happen", etc. is what leads to things like HTML and SQL injection vulnerabilities.

If you're inputting or outputting data in any format, always keep in mind things like "what are the delimiters? What if the data in the input/output contains them?"

Post reply on HN