Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Friends don't let friends export to CSV

kaveland.no

Post reply on HN
Print view View original

301–310 of 459 posts

Re: Friends don't let friends export to CSV

#301

Post by ric2b »

stephenr wrote: post #290
GuB-42 wrote: post #97

As a French, there is another problem with CSV. In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead. It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same…

Isn't this exactly what quoting solves? i.e.: ``` "1,20","2,3",hello "2,40","4,6",goodbye ``` If your tool reads CSV by doing `string_split(',', line);`, your tool is doing it wrong. There's a bunch of nuance and shit, which can make CSVs interesting to work with, but storing a comma in a field is a pretty solved issue if the tool in question has more than 5 minutes thought put into it.

Now all your numbers are strings

Re: Friends don't let friends export to CSV

#302

Post by wiz21c »

Pikamander2 wrote: post #277
diogocp wrote: post #129

Earlier quoted context omitted.

> in the real world the spec is essentially "Can you import it to Excel?" And the answer to that is always no. You will it think it's yes because it works for you, but when you send it to someone who has a different Excel version or simply different regional settings, it won't work. The recipient will first have to figure out what dialect you used to export.

I've been amazed by how much better LibreOffice is at importing CSVs in a sane manner than Excel. Its CSV import prompt is nothing short of the gold standard and puts Excel to shame. Also, even if the CSV format is completely valid, Excel will still find a way to misinterpret some of your cells in baffling ways, destroying the original data in the process if you don't notice it in time.

Yeah, I can complain about LO in many ways, but the way it opens CSV is much better than Excel. It was developed by a dev, that's for sure.

Re: Friends don't let friends export to CSV

#303

Post by elric »

echoangle wrote: post #245
GuB-42 wrote: post #97

As a French, there is another problem with CSV. In the French locale, the decimal point is the comma, so "121.5" is written "121,5". It means, of course, that the comma can't be used as a separator, so the semicolon is used instead. It means that depending whether or not the tool that exports the CSV is localized or not, you get commas or you get semicolons. If you are lucky, the tool that imports it speaks the same…

Does any tool seriously localize CSVs?

Any serious CSV tool has the option to pick a delimiter. Usually semicolon or comma, some offer additional options. The only impact it has is on which fields need quoting. When using comma, all decimals in many langugaes need to be quoted. When using semicolon those don't need to be quoted.

Overall, semicolon feels like the superior delimiter.

Most sensible people don't export formatted numbers (e.g. 100.000,00), but even those are pretty trivial to import.

Re: Friends don't let friends export to CSV

#304

Post by elric »

ric2b wrote: post #301
stephenr wrote: post #290

Earlier quoted context omitted.

Isn't this exactly what quoting solves? i.e.: ``` "1,20","2,3",hello "2,40","4,6",goodbye ``` If your tool reads CSV by doing `string_split(',', line);`, your tool is doing it wrong. There's a bunch of nuance and shit, which can make CSVs interesting to work with, but storing a comma in a field is a pretty solved issue if the tool in question has more than 5 minutes thought put into it.

Now all your numbers are strings

It's a text file. All your numbers were already strings. Nothing has changed.

Re: Friends don't let friends export to CSV

#306

Post by cess11 »

SQL and XML have schemas, and they're to a large extent human readable, even to people who aren't developers. If storage is cheap, compression isn't very important.

I've never come across this Parquet-format, is it grep:able? Gzip:ed CSV is. Can a regular bean counter person import Parquet into their spreadsheet software? A cursory web search indicates they can't without having a chat with IT, and SQL might be easier while XML seems pretty straightforward.

Yes, CSV is kind of brittle, because the peculiarities with a specific source is like an informal schema but someone versed in whatever programming language makes this Parquet convenient won't have much trouble figuring out a CSV.

Re: Friends don't let friends export to CSV

#307

Post by ykonstant »

josefx wrote: post #235
djbusby wrote: post #219

Earlier quoted context omitted.

ASCII has had field and record separators since like, forever. Wish we had kept using those.

Does it have nesting operators? I want to embedd ASCII within my ASCII fields. So I can have a table within my table.

The POSIX spec defines portable and non-portable ASCII characters, prudently placing the separators in the non-portable set. In order to nest tables, base64 (or whatever portable encoding) code the table into the field. This works much better, easier and more error-free than any escaping strategy.

Regarding visibility in editors, if you are nesting tables I don't think you care too much about manual editing, but if you do, it is easy to setup vim/emacs/vscode to display the ASCII separators. I am being told even notepad++ can display those, there are no excuses.

Re: Friends don't let friends export to CSV

#308

Post by jjgreen »

wizzwizz4 wrote: post #149
jjgreen wrote: post #112

Earlier quoted context omitted.

You have searched for packages that names contain parquet in suite(s) bookworm, all sections, and all architectures. Sorry, your search gave no results https://packages.debian.org/search?suite=bookworm&searchon=n...

By the same logic, there's no Photoshop Document support – but GIMP and Krita both support it.

  You have searched for photoshop in packages names and descriptions in 
  suite(s) bookworm, all sections, and all architectures (including 
  subword matching). Found 9 matching packages.

  Package abr2gbr

    bookworm (stable) (graphics): Converts PhotoShop brushes to GIMP
    1:1.0.2-5: amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x

  Package gimp

    bookworm (stable) (graphics): GNU Image Manipulation Program
    2.10.34-1+deb12u2: amd64 arm64 armel armhf i386 mips64el mipsel ppc64el s390x

  :
https://packages.debian.org/search?suite=bookworm&section=al...

Re: Friends don't let friends export to CSV

#309

Post by culebron21 »

The poor performance argument is not true even for Python ecosystem that the author discusses. Try saving geospatial data in GeoPackage, GeoJson, FlatGeobuf. They are saved slower than in plain CSV (the only inconvenience is that you must convert geometries into WKT strings). GeoPackage was "the Format of the Future" 8 years ago, but it's utterly slow when saving, because it's an SQLite database and indexes all the data.

Files in .csv.gz are more compact than anything else, unless you have some very-very specific field of work and a very compressible data. As far as I remember, Parquet files are larger than CSV with the same data.

Working with the same kind of data in Rust, I see everything saved and loaded in CSV is lightning fast. The only thing you may miss is indexing.

Whereas saving to binary is noteably slower. A data in generic binary format becomes LARGER than in CSV. (Maybe if you define your own format and write a driver for it, you'll be faster, but that means no interoperability at all.)

Re: Friends don't let friends export to CSV

#310

Post by ykonstant »

samatman wrote: post #152
crq-yml wrote: post #124

Earlier quoted context omitted.

There are characters from ASCII for delimiting records, which are underused because they cause confusion about whether they should be represented as a state change like a backspace character, or as a glyph. See also: "nobody can agree on the line ending character sequence". The USV proposal uses additional codepoints introduced in Unicode for the representation of the record delimiters, so they will always look and e…

> nobody is using them for some other purpose. There's a lot of tooling which uses them for their intended purpose , which is to represent the C0 control characters in a font, so they can be printed when they appear in a document. Your editor is probably one of those. Which is why I consider USV a terrible idea. If I see ␇ in a file, I don't want to be constantly wondering if it's "mention ␇" or "use ␇ to represent \…

I agree about USV, it creates confusion where none needs to exist. For personal use, though, it is not that bad to receive a USV: it should be postmarked ".usv" and in any case if you suspect shenanigans you can `grep` for the offending (literally!) unicode characters and `tr` them into proper ASCII separators. Now, if there is nesting in the USV, I give up.

I share the lament: the whole table issue was solved before it became a problem. POSIX divides ASCII into portable and non-portable characters; only portable characters are allowed in the fields and separators are non-portable. If you need nesting, use a portable encoding of the inner table. This scheme repeats indefinitely without escaping hell or exceptions, preventing tons of errors and headache.

Visibility is such a bizarre complaint. Text editors already handle control characters: they handle tabs, they handle newlines, it is not a tremendous, earth-shattering feature request to make them handle separators gracefully.

Post reply on HN