Live data from Hacker News

ASCII Delimited Text – Not CSV or TAB delimited text

ronaldduncan.wordpress.com

271–280 of 286 posts

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#271

Earlier quoted context omitted.

I'm looking at a couple Macs sitting around the house and they all have a pretty obvious # right above the 3. Perhaps you mean some other key? (I do really miss the days when Mac keyboards had the weird symbol they use for "alt" in menus on the key.)

Which keyboard do you have? The standard aluminium full size one doesn't label it, at least in the uk

All my machines have US keyboards.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#272

Earlier quoted context omitted.

TSV is nicer for output (on stderr/out or a logfile), so tends to crop op if you want to parse the output/logfile of something. I haven’t seen Excel in use at my workplace yet.

Surely only if you're dealing with fixed-width fields.

Floating point values with a given precision and some integers. We’ll have to buy a proper supercomputer before the latter take more than seven digits :\

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#273

You know where this is useful? Databases. No, please, put the gun down... let me explain. Sometimes you have a database that's so complex and HUGE that changing tables would be a nightmare, or you just don't have the time. You have a field that you want to shove some serialized data into in a compact way and not have to think about formatting. You could use JSON, you could use tabs or csv, but both of those require a…

You just described a Pick or "multivalue" database. They were a nightmare to work on, but I'll admit that's mostly because of the tools (or lack thereof). It led to people storing all sorts of different data in one table and the queries got really messy because multivalue fields had to be treated differently than regular ones.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#274
post #196

Earlier quoted context omitted.

Tabs are not a record separator. They are a formatting code. That's why a mess is created when they are used to separate records.

You just gave further proof that tabs are a catch-22.

Yes, and he also specified why they are different from record separators in this regard.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#275

Earlier quoted context omitted.

Who said anything about escaping a character? For most datasets, having actual tab characters is rare, especially if you can just replace them with spaces. Same with newlines - the just aren't needed in a lot of data. If you're dealing with user-derived text content, tab delimited files might not be the best choice. However, it's great for tabular data. With CSV, you have to escape quotes, commas, and newlines. With…

I mean for the general case. Sure, more datasets may be tab-safe than comma-safe. CSV doesn't need quotes if there's no commas. TSV needs quotes or something if the data contains tabs.

Or, you just escape the tab. \t style. (Well, then you need to escape \\, and \n, etc). Really, it just gets messy at that point. Which is why I try to avoid free text in tab delimited files. It's not much better in CSV if you allow newlines within cells.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#277
post #242
post #213

Why not use the non-printing char as the comma instead of the record separator. 1. Replace all the commas in the text with the unique non-printing char before converting to CSV. 2. Convert this char back to a comma when processing the CSV for output to be read by humans. Because commas in text are usually followed by a space, the CSV may still even be readable when using the non-printing char. I must admit I've never…

that's exactly what https://github.com/dbro/csvquote does for commas and newlines both.

Why use this instead of sed, awk, flex, lua, etc.?

sed does the job and on almost all UNIX clones it never needs to be installed.

Because it's already there.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#278

Earlier quoted context omitted.

You just gave further proof that tabs are a catch-22.

Yes, and he also specified why they are different from record separators in this regard.

It's no different, if it can be displayed on the screen and it is on the keyboard, then the user will type it into the text file for how it look, not because they want to really use it as record separator. And if user can type it in the content then you have to escape it, which is back to the same problem with using tabs.

But if it's not on the keyboard, then the user won't type it. So it doesn't get used.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#279
Devil's advocate - CSV is superior, because edge case bugs (a comma in the data) are likely to be tested.

The edge case bugs in ASCII codes could still crop up. It shouldn't, but then, valid SQL shouldn't crop up in a web form either. And when it does, we'll need escape codes just like CSV, only it won't be well tested in all the tools (because it's not going to frequently happen).

It's like all the OSS advocates laughing at Microsoft's idiotic "My Documents" folder. It's not there because they didn't realise how much trouble it would case programmers, it's there because they wanted for force people to deal with edge cases.

Re: ASCII Delimited Text – Not CSV or TAB delimited text

#280
post #278

Earlier quoted context omitted.

Yes, and he also specified why they are different from record separators in this regard.

It's no different, if it can be displayed on the screen and it is on the keyboard, then the user will type it into the text file for how it look, not because they want to really use it as record separator. And if user can type it in the content then you have to escape it, which is back to the same problem with using tabs. But if it's not on the keyboard, then the user won't type it. So it doesn't get used.

It's very different.

Tab, comma, semicolon are actual characters people DO and need to use inside records. So not only they can be typed into records, they absolutely HAVE to be typed for most textual records.

ASCII record separators are not needed inside records at all. If someone adds them "for how it looks", it's his problem.

One (not using a record separator inside a record) is a matter of choice and doing the right thing. The other (not using comma, semicolon or tab) is a non starter.

Obviously there's a difference.

Post reply on HN