Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

451–459 of 459 posts

Re: Friends don't let friends export to CSV

#451

Post by outop »

eviks wrote: post #450
outop wrote: post #448

Earlier quoted context omitted.

Well, most code that loads CSVs is intended to work with certain files from certain sources, and not with all the CSVs that have ever existed. So yes, I am happy with code that works for a subset of files. There are thousands of applications which work with CSVs and they all do exactly this.

and those thousands of apps bug because they exist in a reality where few CSVs from certain sources can have different formats, not in the fantasy world where intention to work with certain files becomes a binding specification

[dead]

Re: Friends don't let friends export to CSV

#452

Post by SAI_Peregrinus »

averms wrote: post #449
SAI_Peregrinus wrote: post #380

Earlier quoted context omitted.

The main issue is that "CSV" isn't one format with a single schema. It's one format with thousands of schemas and no way to communicate them. Every program picks its own schema for CSVs it produces, some even change the schema depending on various factors (e.g. the presence or absence of a header row). RFC 4180 provides a (mostly) unambiguous format for writing CSVs, but because it discards the (implied) schema it's…

> RFC 4180 provides a (mostly) unambiguous format for writing CSVs, What are the ambiguities in RFC 4180?

It allows non-ASCII text but does not provide any way to indicate charset within the file, instead requiring it out-of-band. Once the file is saved, the text encoding becomes ambiguous. Likewise for the presence or absence of a header row.

Likewise for whether double quotes (`"`) are allowed in fields (rule 5). This one gets even worse, since the following rule (6) uses double quotes to escape line breaks and commas, but they may not be allowed at all so commas in fields may not be escapable.

It only supports text, not numbers, dates, or any other data, and provides no way to indicate any data type other than text.

Re: Friends don't let friends export to CSV

#453

Post by bruce511 »

iainmerrick wrote: post #296
bruce511 wrote: post #285

Earlier quoted context omitted.

Well you might be wrong, but EDI in general and HL7 specifically allow 3 levels of "fields in fields in field". As long as your parser copes, and as long as you have appropriate structures to import into, its no big deal.

So now when I'm exporting data I need to know what nesting level it's going to live at so I can generate the correct separators? I really think that might be the worst idea I've heard for a while!

It's not toooo bad :) But it's very much a "thing" in the real word. It's called EDI, and it's been around for a long time.

Re: Friends don't let friends export to CSV

#454

Post by deathanatos »

deathanatos wrote: post #180
lostlogin wrote: post #148

Earlier quoted context omitted.

> Are there unicode characters specifically for delimiters? We could use the HL7 pipe ‘|’ and all enjoy that hell.

God, please no. For those unfamiliar with the atrocity that is HL7v2, the format is essentially CSV, but with the record separator set to a lone CR, and the field separator usually set to |. Usually, because the format lets the consumer of the format redefine it, for whatever reasons. (The first use of the field separator it determines whatever character it will be. Thankfully, the first use is in a fixed spot, so it…

I've unfortunately had to bless my brain with much more of this standard this week, for some reason.

Did I mention that subcomponents (if you look at it like a CSV, cells are further subdivided into components & subcomponents, so subcomponents are sort of where we hit "cell text", if you want to keep going with that broken analogy) — contain escape sequences, so that you can have things like the field separator. Normal stuff, so far. The escape sequences also include highlighting, binary blobs, and a subset of roff.

Re: Friends don't let friends export to CSV

#455

Post by deathanatos »

doctor_eval wrote: post #168
lostlogin wrote: post #148

Earlier quoted context omitted.

> Are there unicode characters specifically for delimiters? We could use the HL7 pipe ‘|’ and all enjoy that hell.

It’s been years since I last worked with HL7. Isn’t there also ^ and ~ to deal with? Hell indeed.

Yes, there is. Multi-dimensional CSV?

Re: Friends don't let friends export to CSV

#456

Post by fragmede »

pixl97 wrote: post #108
fragmede wrote: post #59

Earlier quoted context omitted.

In case it wasn't clear, I want software engineers in the finance industry to implement sqlite import and export in their various pieces of software, not too give up on lucrative, existing contracts, obviously.

The person in the fintec industry rarely decies "oh, I'm going to add this format no one is asking for". They'll get a specification requiring csv and start implementing it. Get fired halfway though, then someone on the other side of the planet will implement it incorrectly. The first 3 revisions won't meet the customers needs, but the customer can't move away anyway.

Specifically, people in the FinTech industry should start asking for sqlite as an export format.

Re: Friends don't let friends export to CSV

#457

Post by culebron21 »

smcin wrote: post #446
culebron21 wrote: post #412

Earlier quoted context omitted.

I'd say compared to Shapefile, it is indeed better in every aspect (to begin with, shp has 8-character column names limit). For some kinds of data and operations GPKG is superior to other geo-formats. Like 1) store a lot of data, but retreive within an area (you can set an arbitrary polygon as a filter with GDAL driver, IIRC), 2) append/delete/modify and have the data indexed -- with CSV here you'll have to just repr…

What's the one-line on that: there is consensus that Shapefile is on its way out but no consensus on its successor, not GeoPackage or anything else? Where can we simply see as of today, what % of geospatial files in use are Shapefile, GeoPackage et al?? (I tried to estimate from references to formats on https://gis.stackexchange.com/ but it just gave me a headache.)

If you need mutability and indexing, then choose GeoPackage. If you can skip mutability, but still need indexing, probably FlatGeoBuffer. If you can skip both indexing and mutability -- then CSV or GeoJSON will suffice (especially if you need small data and human readable).

Re: Friends don't let friends export to CSV

#458

Post by nmz »

outop wrote: post #447
nmz wrote: post #396

Earlier quoted context omitted.

Google contacts does.

It seems that Google generally try and find ways of making exporting their data to use in other services as painful as possible.

You need newlines in fields, because contacts has addresses, and addresses have newlines. (And also Notes, but that's not as important)

(Oh and I forgot about headers, headers should be mandatory but people are aloof about it)

Re: Friends don't let friends export to CSV

#459

Post by chirau »

quesera wrote: post #217
chirau wrote: post #182

Earlier quoted context omitted.

you are assuming the regular end user knows the difference between 4 spaces and a tab and the nuances that come with them trying to replace one with the other or why space between two values is different at one point from the next. Commas are, by far, better delimeters than tabs in the grand scheme of things and with both expert and regular users considered.

I disagree. The end user doesn't need to know the difference between 4 spaces and a tab. Tabs are just whitespace. Tabs are uncommon but convenient whitespace. Commas are extremely common content. Tabs are a vastly better delimiter. If you are wrapping source code in a CSV, a) you're doing it wrong, and b) you'll get bitten by newlines just as quickly! If you're including content that requires specific whitespace pre…

I don't think you fully understood what i was saying.

If a regular user had rows like this

Adam Smith 27 WA

JonathanBoyd 23 NC

They are likely going to have tougher time adding a new row as compared to if it was comma delimited. You underestimate the simplicity of end users and how tabs and spaces can confuse them. This is why they prefer Excel, with boxes, because they cannot keep up with formatting and such. Tabs are spaces to many people. Commas are clearer.

Post reply on HN