Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

371–380 of 459 posts

Re: Friends don't let friends export to CSV

#371

Post by puppymaster »

hn_throwaway_99 wrote: post #111
prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

Totally agree. His arguments are basically "performance!" (which is honestly not important to 99% of CSV export users) and "It's underderspecified!" And while I can agree with the second, at least partly, in the real world the spec is essentially "Can you import it to Excel?". I'm amazed at how much programmers can discount "It already works pretty much everywhere" for the sake of more esoteric improvements. All that…

ditto. if it fits in the ram, file types don't matter.

Re: Friends don't let friends export to CSV

#372

Post by orthoxerox »

eska wrote: post #314

It's strange to me that people complain about some variety in CSV files while acting as if parquet was one specific file format that's set in stone. They can't even decide which features are core, and the file format has many massive backwards-incompatible changes already. If you give me a parquet file I cannot guarantee that I can read it, and if I produce one I cannot guarantee that you can. I treat formats such as…

That's true. Parquet went through the weirdest changes between its various revisions and because it was used for Hadoop data lakes, there's a whole bunch of data that is being stored in legacy formats. Off the top of my head:

- different physical types to store timestamps: INT96 vs INT64

- different ways to interpret timestamps before tzdb (current vs earliest tzdb record)

- different ways to handle proleptic Gregorian dates and timestamps

- different ways to handle time zones (since Parquet only has the equivalents of LocalDateTime and Instant, but no OffsetDateTime or ZonedDateTime and earlier versions of Hive 3 were terribly confused which is which)

- decimal data type was written differently, as a byte array in older versions and as int/byte array/binary in the newer ones

- Hadoop ecosystem doesn't support decimals longer than 38 digits, but the file format supports them

Re: Friends don't let friends export to CSV

#374

Post by lakomen »

Ok, funny guy. Tell that to all the wholesale providers, which use software from 2005 or at least it feels that way.

No query params in their single endpoint and only csv exports possible.

Then add to that, that shopify, apparently the leader or whatever in shopping software, can't do better than require exactly the format they say, don't you dare coming with configurable fields or mapping.

The industry is stuck in the 00s, if not 90s.

Re: Friends don't let friends export to CSV

#375

Post by frob »

gerdesj wrote: post #117
GuB-42 wrote: post #97

As a French, there is another problem with CSV. In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead. It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same…

If only a totally separate data field separator character had been invented early on and been given its own key on the keyboard, coloured: "only use for field delimiting". You know as well as I do that it would have been co-opted into another rôle within weeks, probably as a currency indicator. You should probably use PSV - Point Separated Variable! Obviously we would need to adjust PSV to account for correct French…

My company has been using DSV for a bit: Dagger Separated Values. Unicode dagger (†) to separate values and double-dagger (‡) to indicate end of row. It allows us to easily handle both commas and newlines.

Re: Friends don't let friends export to CSV

#376

Post by wombatpm »

rightbyte wrote: post #275
wombatpm wrote: post #249

Earlier quoted context omitted.

No you don’t. It’s a holdover from when files were on tapes. The logic is all inverted too. Record separators were at the beginning of a record. Group and unit separators could then be nested. You really needed a spec for whatever document you were trying to makt

It doesn't matter if the sentinel byte is after or before each record. Having it before is nice for syncing byte streams.

It matters when you trying to roundtrip the data through a text editor because existing tools balk at a 300MB file with a single line.

Re: Friends don't let friends export to CSV

#377

Post by SAI_Peregrinus »

barrkel wrote: post #20
rietta wrote: post #5

Or export to CSV correctly and test with Excel and/or LibreOffice. Honestly CSV is a very simple, well defined format, that is decades old and is “obvious”. I’ve had far more trouble with various export to excel functions over the years, that have much more complex third-party dependencies to function. Parsing CSV correctly is not hard, you just can’t use split and be done with it. This has been my coding kata in eve…

CSV is not well-defined. Data in the wild doesn't even agree that it's comma separated. String encoding? Dates? Formatted numbers? Booleans (T/F/Y/N/etc)? Nested quotes? Nested CSV!? How about intermediate systems that muck things up. String encoding going through a pipeline with a misconfiguration in the middle. Data with US dates pasted into UK Excel and converted back into CSV, so that the data is a mix of m/d/yy…

Most of the problems come from treating CSV fields as something other than US-ASCII text strings. They're not dates, they're not numbers, they're just freeform text. If you want fancy-pants features like "numbers" or the ability to represent all common words in use in the US (e.g. the California town Rancho Peñasquitos) and aren't a masochist, you don't want CSV.

Re: Friends don't let friends export to CSV

#378

Post by samatman »

stephenr wrote: post #291
samatman wrote: post #155

Earlier quoted context omitted.

Yeah, hon hon hon and all, but one of my (US) bank statements exports a CSV which uses commas in numbers in the US fashion, so $1,500 and the like. Writing a custom CSV munger to intake that into ledger-csv was... fun, but then again, only had to do it once.

Surely if they're putting commas in values they were quoting the values though?

They were not.

Fortunately the values are always prefixed by a dollar sign, making parsing deterministic, though ugly.

Re: Friends don't let friends export to CSV

#379

Post by DontchaKnowit »

prepend wrote: post #85

This article seems written by someone who never had to work with diverse data pipelines. I work with large volumes of data from many different sources. I’m lucky to get them to send csv. Of course there are better formats, but all these sources aren’t able to agree on some successful format. Csv that’s zipped is producible and readable by everyone. And that makes is more efficient. I’ve been reading these “everyone i…

My entire exoerience with software development has been me bellyaching about how stupidly things are setup, why dont we do it this way instead etc... only to actually set about working on fixing these things and reqlizing its either way harder than I thought, it makes more sense than I thought, or it just plumb isnt worth the effort.

The evervescent suggestions of a brighter more logical, even obvious, solutions, is often a clear indicator of domain inexperience or ignorance.

Re: Friends don't let friends export to CSV

#380

Post by SAI_Peregrinus »

fredguth wrote: post #25
VMG wrote: post #13

Earlier quoted context omitted.

FTA: * What does missing data look like? The empty string, NaN, 0, 1/1-1970, null, nil, NULL, \0? * What date format will you need to parse? What does 5/5/12 mean? * How multiline data has been written? Does it use quotation marks, properly escape those inside multiline strings, or maybe it just expects you to count the delimiter and by the way can delimiters occur inside bare strings? And let me add my own question…

What is the encoding of the text file? UTF8, windows-1252? What is the decimal delimiter “.”, “,”? Most csv users don’t even know they have to be aware of all of these differences.

The main issue is that "CSV" isn't one format with a single schema. It's one format with thousands of schemas and no way to communicate them. Every program picks its own schema for CSVs it produces, some even change the schema depending on various factors (e.g. the presence or absence of a header row).

RFC 4180 provides a (mostly) unambiguous format for writing CSVs, but because it discards the (implied) schema it's useless for reading CSVs that come from other programs. RFC 4180 fields have only one type: text string in US-ASCII encoding. There are no dates, no decimal separators, no letters outside the US-ASCII alphabet, you get nothing! It leaves the option for the MIME type to specify a different text encoding, but that's not part of the resulting file so it's only useful when downloading from the internet.

Post reply on HN