Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

Post reply on HN
Print view View original

191–200 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#191

Post by ithkuil »

dxbydt wrote: post #77
Istof wrote: post #66

Earlier quoted context omitted.

TLDR: it's superior, but don't do it...

That's unfortunately a very accurate summary:) Real estate data, traffic data, weather data, population demographics, stock prices, tweets - I've parsed all that and more. Every one of them was a giant Tsv (except finance ones, which were csv's because Excel). Say you purchase the database containing every single home sold/bought in California for past decade. That's 11 20gb Tsv's with 250 tab separated columns plus…

awk, cut, sed and less

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#192

Post by ithkuil »

mantrax3 wrote: post #13
mikeash wrote: post #3

It doesn't solve the problem, although it does make it far less likely to run into it. For a trivial example, try building an ASCII table using this format, with columns for numeric code, description, and actual character. You'll once again run into the whole escaping problem when you try to write out the row for character 31.

For CSV forbidding commas in data is not practical. For ASCII delimiters, forbidding ASCII delimiters in data is practical. Sure - you can't, say, nest ASCII tables into one another due to this limitation. But for simple structure, it doesn't hurt to have ASCII separators in the toolbox. The only big problem I see is that they're rendered as invisible characters, which will make debugging harder. If we wouldn't have…

If only they had included an official escape character in ascii ... oh wait

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#194

Post by tbrownaw »

mantrax3 wrote: post #177
tbrownaw wrote: post #98

Earlier quoted context omitted.

For ASCII delimiters, forbidding ASCII delimiters in data is practical. Data formats that can't handle recursion are never practical. They only work until they don't, at which point they're entrenched and impossible to replace.

You may have heard about spreadsheets or relational databases...

Yes. I've seen spreadsheets where some of the cells contain CSV data. I've heard recent news of relational databases adding support for JSON columns and seen them used to store XML, and know that a LOB field could be used to store an image of another database. I've seen hierarchical key/document databases which contain other hierarchical key/document databases (and relational databases, and anything else you can imagine).

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#195

Post by dlrush »

dxbydt wrote: post #57

Don't do this. Tsv has won this race, closely followed by Csv. Anything else will cause untold grief for you and fellow data scientists and programmers. I say this as someone who routinely parses 20gb text files, mostly Tsv's and occasionally Csv's for a living. The solution you are proposing is definitely superior but isn't going to get adopted soon.

It's hardly too late. If you just look at the use case of application logging for example, the latest fashion is logging using JSON format log files, which is INSANE for a different set of reasons. I have servers that generate TB of log files on a regular basis. ASCII delimited log file format standard could be adopted by the application logging space, could result in some uniform tools that provide better streaming support for log shipping, and gain adoption in other adjacent use cases from there.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#196

Post by gress »

3JPLW wrote: post #155
stingraycharles wrote: post #120

Earlier quoted context omitted.

Is this a problem that's solved with a "movement" ? I'm willing to contribute parsers and perhaps a text editor plugin if other people are willing to chip in. But indeed, this is a problem that's solvable by making the tools support it. Since most of the tools are OSS, this is mostly solvable.

You're missing my point. If this were a "solved problem," then it would be no better than the status quo with TSV. Tabs are a similar kind of control character, except they can be typed and displayed easily. And TSV files are a mess because tabs end up in the fields themselves. It's a catch-22. (I'm exaggerating; this would still be somewhat better since there will never be a big key on your keyboard dedicated to the…

Tabs are not a record separator. They are a formatting code. That's why a mess is created when they are used to separate records.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#197

Post by oneeyedpigeon »

Crito wrote: post #171
flux_w42 wrote: post #165

Earlier quoted context omitted.

Here in Belgium we have to AltGr-the hell out of our keyboard during development. These characters can only be entered using the AltGr key on a Belgian Azerty keyboard (be-latin1): '|' '@' '#' '^' '{' '}' '[' ']' '\' and '~' Its not realy a problem, as long as you're used to it :-)

Of those, |, @, #, ^, {, } and ~ are all keys that you need shift for on US Qwerty keyboards, so I imagine that the typing experience for those is relatively comparable.

IF those characters are actually displayed on the corresponding keys. If it's like the # on the Mac keyboard - not displayed anywhere - it's a right pain to learn in the first place without any visual cues.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#198

Post by oneeyedpigeon »

raldi wrote: post #79
thwarted wrote: post #73

Earlier quoted context omitted.

Unit Separator is Control-_ (underscore) and Record Separator is Control-^ (caret), for instance. Most modern text editors won't pass through every control character. Vim lets me type the unit separator, but not the record separator, for instance. Vim has bindings for some control key combos, which is why you can't type them directly. I can type control-_ without issue, but to get an ASCII 30 (RS) to show up, I have…

...and that's why, when running a command that's reading what you're typing to stdin, you press Ctrl-D to tell it that you're done. Because, as your "man ascii" trick shows, this generates the "end of transmission" character. Similarly, Ctrl-L clears the screen in most Unix apps because that generates the "form feed" character, and on a line printer or paper-based teletype terminal, "form feed" means to advance to th…

I am going to love not having to type out 'clear' all the time - thank you!

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#199

Post by Pitarou »

While we're on the subject, we should probably be using control code 16 (Data Link Escape) instead of the backslash character to escape strings.

The problem is, of course, that we can't see it (no glyph) and we can't "touch" it (no key for it) so people won't use it. Ultimately, we're all still stick-wielding apes.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#200

Post by oneeyedpigeon »

claudius wrote: post #140
radicaledward wrote: post #127

Earlier quoted context omitted.

I was surprised to see you list tsv as more common than csv. I encounter csv's on a pretty regular basis, but I don't think I've had to parse a tsv in the past 3 or 4 years. As a junior web developer, I don't have much experience though. 9 times out of 10, the csv is coming from or going to Excel, or a system that was designed to support Excel. If you don't mind my asking, what types of data do you regularly work wit…

TSV is nicer for output (on stderr/out or a logfile), so tends to crop op if you want to parse the output/logfile of something. I haven’t seen Excel in use at my workplace yet.

Surely only if you're dealing with fixed-width fields.
Post reply on HN