Consider Using CSV

#1

Post by jfhr » Sat, Dec 10, 2022, 4:44 PM UTC

Consider Using CSV
jfhr.me

Re: Consider Using CSV

#2

Post by revskill » Sat, Dec 10, 2022, 5:54 PM UTC

Sure! For example, for batch processing, CSV is always the default for me and the teams.

Re: Consider Using CSV

#3

Post by ndsipa_pomu » Sat, Dec 10, 2022, 6:00 PM UTC

As much as I like and use CSV for database work, it has a problem with being poorly specified. The most common problems are when processing CSVs produced elsewhere which might not enclose text fields with quotes and thus have issues with data that includes commas and multi-line data.

Re: Consider Using CSV

#4

Post by sheeeep86 » Sat, Dec 10, 2022, 6:01 PM UTC

You could have the advantages of both worlds by having one json object per line. You could stream process, and you could structure more complex objects and have consistent escaping.

Re: Consider Using CSV

#5

Post by majkinetor » Sat, Dec 10, 2022, 6:02 PM UTC

With gzip on web server the difference is not important at all.

CSV in general is problematic as there is no standard (RFC 4180 is not). In certain contexts this surely can be good solution but definitelly not good in general scenario.

Re: Consider Using CSV

#6

Post by ledauphin » Sat, Dec 10, 2022, 6:03 PM UTC

sheeeep86 wrote: post #4

You could have the advantages of both worlds by having one json object per line. You could stream process, and you could structure more complex objects and have consistent escaping.

and the "redundant" headers will get compressed away to nearly nothing over most transports.

Re: Consider Using CSV

#7

Post by cpeterso » Sat, Dec 10, 2022, 6:04 PM UTC

Another alternative is a streaming JSON format like JSONL (newline-delimited JSON). You can parse one record/line at a time, but still have the structure and named fields of JSON.

https://en.m.wikipedia.org/wiki/JSON_streaming

Re: Consider Using CSV

#8

Post by gugagore » Sat, Dec 10, 2022, 6:04 PM UTC

sheeeep86 wrote: post #4

You could have the advantages of both worlds by having one json object per line. You could stream process, and you could structure more complex objects and have consistent escaping.

https://jsonlines.org/

Re: Consider Using CSV

#9

Post by account-5 » Sat, Dec 10, 2022, 6:04 PM UTC

I think one of the issues it data types. JSON has them CSV doesn't, so this means your program needs to be aware of which columns are which data type and do the conversion where needed.

It's similar to JSON Vs INI files for config files.

On a different note I wouldn't nest JSON in a CSV column. I'd delimit with a pipe or something the split string on that. Much simpler if you're in control of the data.

Re: Consider Using CSV

#10

Post by CharlesW » Sat, Dec 10, 2022, 6:05 PM UTC

ndsipa_pomu wrote: post #3

As much as I like and use CSV for database work, it has a problem with being poorly specified. The most common problems are when processing CSVs produced elsewhere which might not enclose text fields with quotes and thus have issues with data that includes commas and multi-line data.

Yes, I feel like this would've been more helpful generalized as "Consider DSV" (delimiter-separated values) than CSV specifically, because of the interop issues that often come up. I'd have also mentioned using Parquet.