Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

51–60 of 459 posts

Re: Friends don't let friends export to CSV

#51

Post by Macha »

rietta wrote: post #46
dtech wrote: post #11

Earlier quoted context omitted.

Unfortunately that is only the case as long as you stay within the US. for non-US users Excel has pretty annoying defaults, such as defaulting to ; instead of , as a separator for "CSV", or trouble because other languages and Excel instances use , instead of . for decimal separators. A nice alternative I've used often is to constructor an excel table and then giving is an .xls extension, which Excel happily accepts a…

Localization is a thing. None of this is a show stopper. Subclasses and configuration screen for import and export.

And then you get a business analyst at your client going "I just hit export in our internal tool, what's this delimiter that you're asking me about in the upload form? Google Sheets doesn't ask me to tell them that"

Re: Friends don't let friends export to CSV

#52

Post by Eddy_Viscosity2 »

barrkel wrote: post #20
rietta wrote: post #5

Or export to CSV correctly and test with Excel and/or LibreOffice. Honestly CSV is a very simple, well defined format, that is decades old and is “obvious”. I’ve had far more trouble with various export to excel functions over the years, that have much more complex third-party dependencies to function. Parsing CSV correctly is not hard, you just can’t use split and be done with it. This has been my coding kata in eve…

CSV is not well-defined. Data in the wild doesn't even agree that it's comma separated. String encoding? Dates? Formatted numbers? Booleans (T/F/Y/N/etc)? Nested quotes? Nested CSV!? How about intermediate systems that muck things up. String encoding going through a pipeline with a misconfiguration in the middle. Data with US dates pasted into UK Excel and converted back into CSV, so that the data is a mix of m/d/yy…

If the use-case has these complexities, then other formats may be better. I'll go out on a limb and say that MOST data export to csv are simple column data where it works just fine - at least that's been my experience.

Re: Friends don't let friends export to CSV

#53

Post by orwin »

rietta wrote: post #5

Or export to CSV correctly and test with Excel and/or LibreOffice. Honestly CSV is a very simple, well defined format, that is decades old and is “obvious”. I’ve had far more trouble with various export to excel functions over the years, that have much more complex third-party dependencies to function. Parsing CSV correctly is not hard, you just can’t use split and be done with it. This has been my coding kata in eve…

I will hard disagree here. Always have has a clrf issue or another weirdness come up.

Especially if you work with teams from different countries, csv is hell. I always generate rfc compliant csv, not once it was accepted from day one. Once, it took us two weeks to make it pass the ingestion process (we didn't have access to the ingest logs and had to request them each day, after the midnight processing) so in the end, it was only 10 different tries, but still.

I had once an issue with json (well, not one created by me) , and it was clearly a documentation mistake. And I hate json (I'm an XML proponent usually). Csv is terrible.

Re: Friends don't let friends export to CSV

#54

Post by HayBale »

Black616Angel wrote: post #18

I never liked articles about how you should replace CSV with some other format while pulling some absolutely idiotic reasons out of their rear... 1. CSV is underspecified Okay, so specify it for your use case and you're done? E.g use rfc3339 instead of the straw-man 1-1-1970 and define how no value looks like, which is mostly an empty string. 2. CSV files have terrible compression and performance Okay, who in their r…

2. You would be surprised, especially on the science/university level in stat, health or bioinfo. Unfortunately a lot of people go with the path of least resistance and use excel propertiary format or csv for everything.

Like NHS with their post covid data due to excel limitations or gene name conversion problems in sci journals.

Same happens with stupid amount of laboratory management things or bioinformatics tools.

Honestly obviously the article is biased but we should at least think about moving away from csv in non customer facing fronts.

Small files? Json Big files? SQLlite or parquet.

Re: Friends don't let friends export to CSV

#55

Post by aragilar »

Eddy_Viscosity2 wrote: post #52
barrkel wrote: post #20

Earlier quoted context omitted.

CSV is not well-defined. Data in the wild doesn't even agree that it's comma separated. String encoding? Dates? Formatted numbers? Booleans (T/F/Y/N/etc)? Nested quotes? Nested CSV!? How about intermediate systems that muck things up. String encoding going through a pipeline with a misconfiguration in the middle. Data with US dates pasted into UK Excel and converted back into CSV, so that the data is a mix of m/d/yy…

If the use-case has these complexities, then other formats may be better. I'll go out on a limb and say that MOST data export to csv are simple column data where it works just fine - at least that's been my experience.

The problem is is that it is simple until it's not, then you have problems. You need to validate immediately after export, and work out what happens when that fails.

Re: Friends don't let friends export to CSV

#57

Post by jitl »

JumpCrisscross wrote: post #4

> One of the infurating things about the format is that things often break in ways that tools can't pick up and tell you about This line is emblematic of the paradigm shift LLMs have brought. It’s now easier to build a better tool than change everyone’s behaviour. > You give up human readable files, but What are we even doing here.

We’re doing data pipelines. I would rather my data pipeline go 10x faster with Parquet than be able to human read a 30gb CSV file.

Re: Friends don't let friends export to CSV

#58

Post by Eddy_Viscosity2 »

aragilar wrote: post #55
Eddy_Viscosity2 wrote: post #52

Earlier quoted context omitted.

If the use-case has these complexities, then other formats may be better. I'll go out on a limb and say that MOST data export to csv are simple column data where it works just fine - at least that's been my experience.

The problem is is that it is simple until it's not, then you have problems. You need to validate immediately after export, and work out what happens when that fails.

I agree about validating after export, which is a good practice. But if you know your use-case to be csv-friendly, then its a nice simple long-standing almost universal format. Lots of pros with that. Using a more complex format for simple data may (or may not) save you issues with a rare edge case but could cost you in other areas. Like a non-technical manager having no idea how to look at.

Re: Friends don't let friends export to CSV

#59

Post by fragmede »

barrkel wrote: post #36
fragmede wrote: post #34

Earlier quoted context omitted.

Yeah but we're not in the dark ages of computers anymore. Export to Sqlite database instead.

Sure, you tell the finance industry that. They have systems, the systems already produce CSV. They can sign a contract worth multiples of your salary if you can consume it. Do you want the money or not?

In case it wasn't clear, I want software engineers in the finance industry to implement sqlite import and export in their various pieces of software, not too give up on lucrative, existing contracts, obviously.

Re: Friends don't let friends export to CSV

#60

Post by alserio »

rietta wrote: post #5

Or export to CSV correctly and test with Excel and/or LibreOffice. Honestly CSV is a very simple, well defined format, that is decades old and is “obvious”. I’ve had far more trouble with various export to excel functions over the years, that have much more complex third-party dependencies to function. Parsing CSV correctly is not hard, you just can’t use split and be done with it. This has been my coding kata in eve…

Excel and data precision are really at odds. But it might be interesting to see what would happen if excel shipped with parquet import and export capabilities
Post reply on HN