Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

221–230 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#221

Post by chinpokomon »

Having read through all the comments, I think the only real benefit to using the control characters is in the original intent. That is a flat file that represents a file system, with file separators (FS), group separators (GS), like a table, record separators (RS), and unit separators (US), to identify the fields in the record, storing only printable ASCII values.

This isn't intended to be a data exchange format, it is a serial data storage format. In this way, there may be some valid usages, but modern file systems do not need this sort of representation and it has no real benefit over *SV formats for most use cases. I suppose It could still be used for limited exchange, but since it can't be used storing binary, much less Unicode (except for perhaps UTF-8), other formats are less ambiguous and more capable.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#222

Post by yardshop »

There actually are glyphs assigned to these characters, at least in the original IBM PC ASCII character set:

Ascii table for IBM PC charset (CP437) - Ascii-Codes

http://www.ascii-codes.com/

They correspond to these Unicode characters

    28  FS  ∟  221f  right angle
    29  GS  ↔  2194  left right arrow
    30  RS  ▲  25b2  black up pointing triangle
    31  US  ▼  25bc  black down pointing triangle
They may not be particularly intuitive symbols for this purpose though.

see also: IBM Globalization - Graphic character identifiers: http://www-01.ibm.com/software/globalization/gcgid/gcgid.htm... (then search for a code point, eg U00025bc)

Unicode code converter [ishida >> utilities]: http://rishida.net/tools/conversion/

http://en.wikipedia.org/wiki/Code_page_437

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#223

Post by typicalrunt »

dxbydt wrote: post #77
Istof wrote: post #66

Earlier quoted context omitted.

TLDR: it's superior, but don't do it...

That's unfortunately a very accurate summary:) Real estate data, traffic data, weather data, population demographics, stock prices, tweets - I've parsed all that and more. Every one of them was a giant Tsv (except finance ones, which were csv's because Excel). Say you purchase the database containing every single home sold/bought in California for past decade. That's 11 20gb Tsv's with 250 tab separated columns plus…

I could preface a lot of this with 'kids these days', but...

What you write is so true. So many large companies use text files to shuttle around data. I worked at one place that used pipe-delimited 10+GB files. It's not sexy, and using awk/sed/cut seems like a hack at first, then you realize that it works and it is the simplest solution to the problem.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#224

Post by scintill76 »

MichaelGG wrote: post #207
scintill76 wrote: post #99

Earlier quoted context omitted.

I think the answer is that those shouldn't occur within your data. If you're dealing with binary data, why are you using a text-based file format? If your data is textual, it shouldn't have control character delimiters within it, as they are reserved for that context. So, strip them out of your data if you have to. If you think they need to be preserved or escaped, IMO you're doing something wrong.

What if your delimited file needs to contain other delimited files?

Then you should probably use something like JSON or XML, or at least do the file combining/splitting with something like tar or zip.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#225

Post by qnaal »

hrabago wrote: post #214
gress wrote: post #196

Earlier quoted context omitted.

Tabs are not a record separator. They are a formatting code. That's why a mess is created when they are used to separate records.

You just gave further proof that tabs are a catch-22.

They are if you're saving formatted text in a tsv- thus all the discussion about this rather mediocre attempt to replace them.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#226

Post by hrjet »

I have created a (work-in-progress) Vim plugin [1], that uses Vim's conceal feature to visually map the relevant ASCII characters to printable characters.

It sort of works, but there are known issues which I have listed in the README.

[1] : https://github.com/hrj/vim-adtConceal

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#227

Post by thwarted »

qu4z-2 wrote: post #218
thwarted wrote: post #211

Earlier quoted context omitted.

Thank you for clarifying something that no one was disputing or event mentioned.

Yeah, I hate it when people point out interesting things I didn't know before too.

I'm sure raldi can repeat, again, making it 3 times, that control-d means end of transmission and that's why it was chosen for signalling end of stdin, on a thread about how to type the control characters.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#228

Post by shrikant »

l-p wrote: post #153
radicaledward wrote: post #127

Earlier quoted context omitted.

I was surprised to see you list tsv as more common than csv. I encounter csv's on a pretty regular basis, but I don't think I've had to parse a tsv in the past 3 or 4 years. As a junior web developer, I don't have much experience though. 9 times out of 10, the csv is coming from or going to Excel, or a system that was designed to support Excel. If you don't mind my asking, what types of data do you regularly work wit…

Your comment disturbs me a little… One of my gripes with Excel was that it imported and produced TSV data by default when you asked for CSV.

Excel actually doesn't 'care'. It uses the record separator defined in your Windows "Regional Settings", and the defaults there differ for each system locale.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#229

Post by Sami_Lehtinen »

mbauman wrote: post #28
Pxtl wrote: post #10

I've done this. Everybody hated it. Most text editors don't display anything useful with these characters (either hiding them altogether or showing a useless "uknown" placeholder), and spreadhseet tools don't support the record separator (although they all let you provide a custom entry separator so the "unit" separator can work). Besides the obvious problem that there's no easy way to type the darned things when som…

It's a shame. The solution is in the charset, but tools never developed to use it so we don't use it. But I'd wager that if tools had historically supported them, then the situation would be no different than it is with tab. There are representational glyphs for tab, return, and others (⇥, ↵), and editors can show them in 'show whitespace' modes. There could be representational glyphs for these control characters, to…

If there isn't a glyph, you can always use text representation for control codes. Here's an example using Notepad++. http://i60.tinypic.com/nps3d1.png

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#230

Post by Crito »

awwaiid wrote: post #220
Crito wrote: post #171

Earlier quoted context omitted.

Of those, |, @, #, ^, {, } and ~ are all keys that you need shift for on US Qwerty keyboards, so I imagine that the typing experience for those is relatively comparable.

I fixed my keyboard -- @,#,^,{,}.... no shift by default. Have to press shift to get 2,3,6,[,],9,0,... Must be annoying to have to press shift to get those symbol chars while you're coding, eh?

> Must be annoying to have to press shift to get those symbol chars while you're coding, eh?

Honestly no. It's all muscle memory for me, I don't think about it any more than I think about typing capital letters.

Post reply on HN