Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Consider Using CSV

jfhr.me

Post reply on HN
Print view View original

71–80 of 112 posts

Re: Consider Using CSV

#71

Post by sitkack »

No one uses that format for streamed json, see ndson and jsonl

http://ndjson.org/

The size complaint is overblown, as repeated fields are compressed away.

As other folks rightfully commented, csv is a mine field. One should assume every CSV file is broken in some way. They also don't enumerate any of the downsides of CSV.

What people should consider is using formats like Avro or Parquet that carry their schema with them so the data can be loaded and analyzed without have to manually deal with column meaning.

Re: Consider Using CSV

#72

Post by sitkack »

tremon wrote: post #23
xwowsersx wrote: post #12

I mean point well taken, but, as they acknowledged in the post themselves, CSV isn't suitable when you have a nested structure. And you almost always have/need a nested structure, no?

Relational databases have worked fine for decades without nested structures. The simple trick is to take the nested structure out of the entity and into its own table.

Did they? And all the databases I use regularly support nested structures, they are extremely expressive.

Re: Consider Using CSV

#73

Post by whateveracct »

I quite like CSVs. I've used them to great effect at maybe every job I've ever had. xsv, sqlite, and Excel/LibreOffice provide useful tooling on top of them.

I see a lot of complaining about "no standard" in this thread, but the way I've used them, it's been fine. I just use Haskell's cassava. If human produce them with Excel/LibreOffice, I never have issues on the ingestion end.

Re: Consider Using CSV

#74

Post by account-5 »

ajanuary wrote: post #63
account-5 wrote: post #9

I think one of the issues it data types. JSON has them CSV doesn't, so this means your program needs to be aware of which columns are which data type and do the conversion where needed. It's similar to JSON Vs INI files for config files. On a different note I wouldn't nest JSON in a CSV column. I'd delimit with a pipe or something the split string on that. Much simpler if you're in control of the data.

How often is this a concern in practice? It’s a question I’ve been thinking about a bunch and the answer I keep coming back to is that most of the time, encoding the type in the data exchange format isn’t actually very useful. So I would be interested in use cases where it is. Are you just trusting that the types coming in are going to be the correct ones? What happens if someone sends you `{“foo”: “10”}` instead of…

> How often is this a concern in practice?

No idea really, but if you're using a JSON parsing library then that is going to automatically convert for the data types. Which, provided you trust the data, saves you a job.

Re: Consider Using CSV

#76

Post by spentu »

I cannot count how many times CSV "format" has caused problems for me..

In my country the decimal separator is comma, instead of punctuation. This causes problems when importing and exporting with this "format".

Just few weeks ago I had fun times working with API returning CSV in unknown encoding. Hopefully they will never make changes (you cannot always trust headers). Ah and i do love when CSV is missing headers and someone adds data into middle.

Of course some of these issues can be avoided by doing the things "right". Sadly you cannot trust this in real life. People write ugly structures in JSON, but at least you can validate results..

Re: Consider Using CSV

#77

Post by ndsipa_pomu »

tom_ wrote: post #62
ndsipa_pomu wrote: post #39

Earlier quoted context omitted.

Pipes are quite common, but for tricky data, I'd recommend ¬. It's on most keyboards and I can't think of any other use of it.

This symbol is not present on US keyboards.

I did not know that. It's on most UK keyboards

Re: Consider Using CSV

#78

Post by SillyUsername »

Holy cow.

If somebody asked me to support this format after you'd left the company I'd quit on the spot. This frankenformat is 100% premature optimization, non standardised, requires custom parsers (which are potentially inefficient and may negate the network performance from having to parse both json and csv) and is potentially very difficult to maintain and debug (no syntax highlighters or rest like posting tools)

Just either use GRPC or JSON with regular network level gzip encoding.

Re: Consider Using CSV

#79

Post by fellowniusmonk »

teddyh wrote: post #38
fellowniusmonk wrote: post #20

Delimited formats performance can be exceptional, they can also be phenomenally terse and avoid the string tarpits of CSV and TSV if you just use these unicode characters. U+241D, U+241E, U+241F

Those are not the unit/record/group separator characters! Those are the graphical symbols for the unit/record/group separator codes. The actual unit/record/group separator codes are in ASCII, as 'tremon' writes in a sibling comment.

I guess I shouldn't post comments when I'm terribly hungover. The fact remains, use those seperators, most developers don't even seem aware of them to any degree.

Re: Consider Using CSV

#80

Post by jointpdf »

scrollaway wrote: post #30
CharlesW wrote: post #10

Earlier quoted context omitted.

Yes, I feel like this would've been more helpful generalized as "Consider DSV" (delimiter-separated values) than CSV specifically, because of the interop issues that often come up. I'd have also mentioned using Parquet.

Parquet has the opposite problem of CSV though. It's so complex to work with, that unless you're specifically in data science, it's both unheard of and unusable. To read a parquet file in Python, you need Apache Arrow and Pandas. And literally the second result for "parquet python libraries" is an article titled "How To Read Parquet Files In Python Without a Distributed Cluster". I remember dealing with Parquet file…

We data scientists are well-known for our exclusive mastery data wrangling arcana, like…

  df = pandas.read_parquet(‘foo.parquet’)
  df.to_csv(‘foo.csv’)
  df.to_json(‘foo.json’)
(no sarcasm)—how could it be simpler than that? What problems have you encountered that make it unusable?
Post reply on HN