Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consider Using CSV

jfhr.me

Post reply on HN
Print view View original

31–40 of 112 posts

Re: Consider Using CSV

#31

Post by elcritch »

fatneckbeardz wrote: post #24
CharlesW wrote: post #10

Earlier quoted context omitted.

Yes, I feel like this would've been more helpful generalized as "Consider DSV" (delimiter-separated values) than CSV specifically, because of the interop issues that often come up. I'd have also mentioned using Parquet.

but which delimiter. if you choose pipe ok, now you have to make sure nobody typed a pipe into the input field or spreadsheet, and you cannot store unix commands if you choose tab, ok, now people will get confused when they try to edit the text file to replace tabs with spaces, and now you have trouble putting code snippets into data fields because they have tabs. this is the problem and it's why xml/json exist. in m…

Well the obvious solution would be ASCII 0x1D (Group Separator)! Accept, no one actually uses those ASCII characters. Kind of bums me out that UNIX basically skipped out on them.

Re: Consider Using CSV

#32

Post by fbdab103 »

ndsipa_pomu wrote: post #3

As much as I like and use CSV for database work, it has a problem with being poorly specified. The most common problems are when processing CSVs produced elsewhere which might not enclose text fields with quotes and thus have issues with data that includes commas and multi-line data.

In the context of an API so long as using a "real" CSV library a lot of those inconsistencies do not appear. Problems happen when you have to interface with humans and desktop software (ie Excel) which has its own rules.

Re: Consider Using CSV

#33

Post by toast0 »

fatneckbeardz wrote: post #24
CharlesW wrote: post #10

Earlier quoted context omitted.

Yes, I feel like this would've been more helpful generalized as "Consider DSV" (delimiter-separated values) than CSV specifically, because of the interop issues that often come up. I'd have also mentioned using Parquet.

but which delimiter. if you choose pipe ok, now you have to make sure nobody typed a pipe into the input field or spreadsheet, and you cannot store unix commands if you choose tab, ok, now people will get confused when they try to edit the text file to replace tabs with spaces, and now you have trouble putting code snippets into data fields because they have tabs. this is the problem and it's why xml/json exist. in m…

> but which delimiter

Control characters. Like ctrl-A and stuff. Almost nobody has them in their data.

Re: Consider Using CSV

#34

Post by fbdab103 »

scrollaway wrote: post #30
CharlesW wrote: post #10

Earlier quoted context omitted.

Yes, I feel like this would've been more helpful generalized as "Consider DSV" (delimiter-separated values) than CSV specifically, because of the interop issues that often come up. I'd have also mentioned using Parquet.

Parquet has the opposite problem of CSV though. It's so complex to work with, that unless you're specifically in data science, it's both unheard of and unusable. To read a parquet file in Python, you need Apache Arrow and Pandas. And literally the second result for "parquet python libraries" is an article titled "How To Read Parquet Files In Python Without a Distributed Cluster". I remember dealing with Parquet file…

I want to use parquet more frequently, but it creates new problems that do not exist if I dump to CSV. Last I looked, there were not any good GUIs that would let someone quickly browse the data. Now it is just a blob lacking introspection. CSV has issues, but it is universal.

Re: Consider Using CSV

#35

Post by thangalin »

CSV is also great for importing external data into documents. My text editor, KeenWrite[0], includes an R engine and a CSV-to-Markdown function[1]. This means you can write the following in a plain text R Markdown document:

    `r#csv2md('filanme.csv')`
The editor will convert Markdown to XHTML in the preview panel (in real time), then ConTeXt can typeset the XHTML into a PDF file in various styles.[2][3] This avoids spending time fighting with table formatting/consistency in certain word processors while storing the data in a machine-friendly format. (Thereby upholding the DRY principle because the data can have a single source of truth, as opposed to copying data into documents, which could go stale/diverge.)

Using JSON would be possible, but it's not as easy to convert into a Markdown table.

[0]: https://github.com/DaveJarvis/keenwrite

[1]: https://github.com/DaveJarvis/keenwrite/blob/main/R/csv.R#L3...

[2]: https://i.ibb.co/6FLXKsD/keenwrite-csv.png

[3]: https://i.ibb.co/47h6zNx/keenwrite-table.png

Re: Consider Using CSV

#36

Post by anamexis »

majkinetor wrote: post #27
tremon wrote: post #23

Earlier quoted context omitted.

Relational databases have worked fine for decades without nested structures. The simple trick is to take the nested structure out of the entity and into its own table.

That may be simple trick for the db, but not when your paradigm involves importing files - imaging telling that to users, instead of giving json, please give 75 csv files.

This scenario might be more common than you think -- spreadsheets still reign supreme, and often 75 csv files is how the users have the data to begin with.

(Incidentally, my day job is building a spreadsheet importer.)

Re: Consider Using CSV

#37

Post by ndsipa_pomu »

II2II wrote: post #25
ndsipa_pomu wrote: post #3

As much as I like and use CSV for database work, it has a problem with being poorly specified. The most common problems are when processing CSVs produced elsewhere which might not enclose text fields with quotes and thus have issues with data that includes commas and multi-line data.

> The most common problems are when processing CSVs produced elsewhere [...] The limitations of CSV are certainly worth considering and, in the instances you mentioned, it may be not be worth using CSV. (If you are going to be using a more complex parser anyway, you may as well using a format that is better defined and where you are less likely to encounter edge cases.) That being said, there remain many cases where…

It tends to be a lowest common denominator or a choice between CSV and Excel documents which are trickier to automate.

Re: Consider Using CSV

#38

Post by teddyh »

fellowniusmonk wrote: post #20

Delimited formats performance can be exceptional, they can also be phenomenally terse and avoid the string tarpits of CSV and TSV if you just use these unicode characters. U+241D, U+241E, U+241F

Those are not the unit/record/group separator characters! Those are the graphical symbols for the unit/record/group separator codes. The actual unit/record/group separator codes are in ASCII, as 'tremon' writes in a sibling comment.

Re: Consider Using CSV

#39

Post by ndsipa_pomu »

fatneckbeardz wrote: post #24
CharlesW wrote: post #10

Earlier quoted context omitted.

Yes, I feel like this would've been more helpful generalized as "Consider DSV" (delimiter-separated values) than CSV specifically, because of the interop issues that often come up. I'd have also mentioned using Parquet.

but which delimiter. if you choose pipe ok, now you have to make sure nobody typed a pipe into the input field or spreadsheet, and you cannot store unix commands if you choose tab, ok, now people will get confused when they try to edit the text file to replace tabs with spaces, and now you have trouble putting code snippets into data fields because they have tabs. this is the problem and it's why xml/json exist. in m…

Pipes are quite common, but for tricky data, I'd recommend ¬. It's on most keyboards and I can't think of any other use of it.

Re: Consider Using CSV

#40

Post by xwowsersx »

tremon wrote: post #23
xwowsersx wrote: post #12

I mean point well taken, but, as they acknowledged in the post themselves, CSV isn't suitable when you have a nested structure. And you almost always have/need a nested structure, no?

Relational databases have worked fine for decades without nested structures. The simple trick is to take the nested structure out of the entity and into its own table.

Unless I misunderstood something, I'm not sure I understand the relevance here. I assumed we were talking about sending data to clients. In such cases, you do not send database tables. Instead, you send rich, fully hydrated objects which are the result of joining those tables. The serialized representation can be backed by the relational model, but at some point you have to put those together to send something useful to the client. My only point is that CSV is unsuitable for this task in many/most cases.
Post reply on HN