61–70 of 355 posts
I never understood why the ASCII separator characters aren't used more. It seems like we're one simple text editor feature away from having easy display and modification. Is there some historical reason for not doing that?
ASCII separators (mnemonics FS, GS, RS, US) are difficult for most users to type, and have no obvious/standardized visual representation.
if you are ok with a binary format there is apache parquet or apache feather or 'jay' ( https://datatable.readthedocs.io/en/latest/api/frame/to_jay.... ).
No doubt binary formats like Parquet are the way to go for high performance with multi-GB datasets. Seems like total overkill if you have a few hundred or thousand rows of data though. Being able to create/edit/view stuff in a text editor and easily version it is very useful.
If you grab a text file from a Windows machine and bring it to a mac, you’ll see that txt is far from perfect.
This is a long way of saying that if we develop both the format and the tooling then the distinction of text vs “binary” tabular data goes away.
> No escaping. If you want to put \u001F or \u001E in your data – tough you > can’t. Use a different format. > It would be reasonably compact, efficient to parse and easy to manually edit > (Notepad++ shows the unit separator as a ‘US’ symbol). Is it me or it won't be human readable because of the lack of new lines?
> Most tabular data currently gets exchanged as: CSV, Tab separated, XML, JSON or Excel. And they are all highly sub-optimal for the job. > CSV is a mess. One quote in the wrong place and the file is invalid. That breaks the other formats too, why pick on CSV? I can imagine a format designed to be friendly to syntax errors, but contra Postel's Law I'm not sure it would be an improvement over a strict, fail-fast synta…
I literally don't get why JSON is bad: [{row1}, {row2}, {row3}] The fact that it can do more is in no way a negative. Can even make a limited JSON parser with reduced capabilities. And with JSON can do more definitions like header names vs column names vs just arrays of arrays.
Doesn’t open in Excel. And since it doesn’t require one record per line it can be a hassle to read without having to parse it. It’s really nice to be able to do “head -n 5”
Earlier quoted context omitted.
No doubt binary formats like Parquet are the way to go for high performance with multi-GB datasets. Seems like total overkill if you have a few hundred or thousand rows of data though. Being able to create/edit/view stuff in a text editor and easily version it is very useful.
Do people really edit csvs in a text editor? It's horrific, the columns don't line up at all, empty cells are represented by a bunch of commas in a row (which, are you supposed to count all the commas?) And in terms of versioning, I have seen people commit diffs of csvs before, and they're equally unreadable. CSV is a plain text format, but that basically buys you nothing. As long as you're going to be loading it int…
Earlier quoted context omitted.
I rarely want to see tabular data in a human-readable format. It is always the most tedious way to approach it. My go-to is Excel/LibreOffice Calc. This approach is at least tolerable to edit in a text editor, while something like the OpenDocument Spreadsheet format or the Excel format is impenetrable.
I rarely do it, but it’s nice to be able to Human read when I need to. Also being able to use all the command line text tools is super convenient. I think it’s a think where having the option for the .1% of times when you need it keeps me using it.
Sometimes it helps a lot to eyeball what you have before doing one off scripts to filter/massage your data. Had a recent case where the path of least resistance was database to csv to one off python to statistic tools with a gui and tabular display.
Could have probably done some enterprise looking export infrastructure but it was a one off and not worth it.
HDF5 is often used in scientific computing for this. https://en.wikipedia.org/wiki/Hierarchical_Data_Format
P.S.
sqlite> .mode csv
sqlite> .import city.csv citiesEarlier quoted context omitted.
I think CSV is crappy because commas are so common in real data. For almost all scenarios I've had to work with, I'd have been perfectly happy with TSV where literal Tab was a disallowed character. No escaping histrionics required.
Not being able to include Tabs and Carriage Returns in your data can be a problem though.
See also this page on the SQLite website (they're understandably very proud of this): https://www.sqlite.org/locrsf.html
I think it's a fantastic format for archiving and distributing data.