Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A love letter to the CSV format

github.com

Post reply on HN
Print view View original

651–660 of 711 posts

Re: A love letter to the CSV format

#651

Post by mjw_byrne »

widforss wrote: post #412
mjw_byrne wrote: post #14

CSV is ever so elegant but it has one fatal flaw - quoting has "non-local" effects, i.e. an extra or missing quote at byte 1 can change the meaning of a comma at byte 1000000. This has (at least) two annoying consequences: 1. It's tricky to parallelise processing of CSV. 2. A small amount of data corruption can have a big impact on the readability of a file (one missing or extra quote can bugger the whole thing up).…

How is this not true for every format that includes quote marks?

It is true for everything that uses quoting, I didn't mean to imply otherwise.

Re: A love letter to the CSV format

#652

Post by Gormo »

eternauta3k wrote: post #631
Gormo wrote: post #625

Earlier quoted context omitted.

Parsing Excel files in simple data interchange use cases that don't involve anyone manually using spreadsheets is an instance of unnecessary complexity. There are plenty of alternatives to CSV that remain plaintext, have much broader support, and are more rigorous than Excel in ensuring data consistency. You can use JSON, XML, ProtoBuf, among many other options.

But everyone already has a GUI installed for editing xlsx files...

Which introduces even more problems when manually editing files is out of scope.

Re: A love letter to the CSV format

#654

Post by pphysch »

hajile wrote: post #85
pphysch wrote: post #82

Earlier quoted context omitted.

The flexibility of JSON is a downside when you just want to stream large volumes of row-oriented tabular data

If you want to stream large volumes of row-oriented data, you aren't reading yourself and you should be using a binary format which is going to be significantly smaller (especially for numeric data).

Yeah that would be the next step in optimization. In the meanwhile, raw text CSV streaming (for not purely numeric data) is still extremely fast and easy to set up

Re: A love letter to the CSV format

#655

Post by jongjong »

I've found some use cases where CSV can be a good alternative to arrays for storage, search and retrieval. Storing and searching nested arrays in document databases tends to be complicated and require special queries (sometimes you don't want to create a separate collection/table when the arrays are short and 1D). Validating arrays is actually quite complicated; you have to impose limits not only on the number of elements in the array, but also on the type and size of elements within the array. Then it adds a ton of complexity if you need to pass around data because, at the end of the day, the transport protocol is either string or binary; so you need some way to indicate that something is an array if you serialize it to a string (hence why JSON exists).

Reminds me of how I built a simple query language which does not require quotation marks around strings, this means that you don't need to escape strings in user input anymore and it prevents a whole bunch of security vulnerabilities such as query injections. The only cost was to demand that each token in the query language be separated by a single space. Because if I type 2 spaces after an operator, then the second one will be treated as part of the string; meaning that the string begins with a space. If I see a quotation mark, it's just a normal quotation mark character which is part of the string; no need to escape. If you constrain user input based on its token position within a rigid query structure, you don't need special escape characters. It's amazing how much security has been sacrificed just to have programming languages which collapse space characters between tokens...

It's kind of crazy that we decided that quotation marks are OK to use as special characters within strings, but commas are totally out of bounds... That said, I think Tab Separated Values TSV are even more broadly applicable.

Re: A love letter to the CSV format

#656

Post by remram »

ahoka wrote: post #618
remram wrote: post #564

Earlier quoted context omitted.

Yeah, you can also use Parquet/JSON/protobuf/XLSX and store numbers as strings in this format. CSV is just a container.

But somehow CSV is the PHP of serialization formats, attracts the wrong kind of developers and projects.

I definitely wouldn't say that. I saw a lot of weird stuff in Excel files, and there's the whole crowd only giving you data as PDFs.

Re: A love letter to the CSV format

#657

Post by ethbr1 »

deepsun wrote: post #614
ethbr1 wrote: post #548

Earlier quoted context omitted.

> Big complex 6-figure e-discovery system? Apparently written by someone who has never heard of quoting... It's because about a certain size, system projects are captured by the large consultancy shops, who eat the majority of the price in profit and management overhead... ... and then send the coding work to a lowest-cost someone who has never heard of quoting, etc. And it's a vicious cycle, because the developers i…

Just a nitpick about consultancy shops -- I've had a chance of working in one in eastern europe and noticed that it's approach to quality was way better than client's. It also helped that client paid by hours, so consultancy company was incentivized to spend more time on refactorings, improvals and testing (with constant pushback from client). So I don't buy the consultancy company sentiment, it always boils down to…

How big was the one you worked for?

In my experience, smaller ones tend to align incentives better.

Once they grow past a certain size though, it's a labor arbitrage game. Bill client X, staff with resources costing Y (and over-represented), profit = X-Y, minimize Y to maximize profit.

PwC / IBM Global Services wasn't offering the best and brightest. (Outside of aforementioned tiger teams)

Re: A love letter to the CSV format

#658

Post by SoftTalker »

mjw_byrne wrote: post #601
SoftTalker wrote: post #596

Earlier quoted context omitted.

Not for text data. Those values are not text characters like , or " are, and have only one meaning. It would be like arguing that 0x41 isn't always the letter "A". For binary files, yeah but you don't see CSV used there anyway.

The idea that binary data doesn't go in CSVs is debatable; people do all sorts of weird stuff. Part of the robustness of a format is coping with abuse. But putting that aside, if the control chars are not text, then you sacrifice human-readability and human-writability. In which case, you may as well just use a binary format.

True, but very few people compose or edit CSV data in Notepad. You can, but it's very error-prone. Most people will use a spreadsheet and save as CSV, so field and record separator characters are not anything they would ever deal with.

Re: A love letter to the CSV format

#659

Post by fauigerzigerk »

kragen wrote: post #610
fauigerzigerk wrote: post #474

Earlier quoted context omitted.

>but in practice I see very few problems arising with using CSV That is not my experience at all. I've been processing CSV files from financial institutions for many years. The likelihood of brokenness must be around 40%. It's unbelievable. The main reason for this is not necessarily the CSV format as such. I believe the reason is that it is often the least experienced developers who are tasked with writing export co…

> And many inexperienced developers seem to think that they can generate CSV without using a library because the format is supposedly so simple. Can't they? def excel_csv_of(rows): for row in rows: for i, field in enumerate(row): if i: yield ',' yield '"' for c in field: yield '""' if c == '"' else c yield '"' yield '\n' I haven't tested this, even to see if the code parses. What did I screw up?

>Can't they?

If my experience reflects a relevant sample then the answer is that most can but a very significant minority fails at the job (under the given working conditions).

Whether or not _you_ can is a separate question. I don't see anything wrong with your code. It does of course assume that whatever is contained in rows is correct. It also assumes that the result is correctly written to a file without making any encoding mistakes or forgetting to flush the stream.

Not using name value pairs makes CSV more prone to mistakes such as incorrect ordering or number of values in some rows, a header row that doesn't correspond with the data rows, etc. Some export files are merged from multiple sources or go through many iterations over many years, which makes such mistakes far more likely.

I have also seen files that end abruptly somewhere in the middle. This isn't specific to CSV but it is specific to not using libraries and not using libraries appears to be more prevalent when people generate CSV.

You'd be surprised how many CSV files are out there where the developer tried to guess incorrectly whether or not a column would ever have to be escaped. Maybe they were right initially and it didn't have to be escaped but then years later something causes a change in number formats (internationalisation) and bang, silent data corruption.

Prioritising correctness and robustness over efficiency as you have done is the best choice in most situations. Using a well tested library is another option to get the same result.

Re: A love letter to the CSV format

#660

Post by whizzter »

arnorhs wrote: post #633
whizzter wrote: post #537

Earlier quoted context omitted.

Sadly the reviver parameter is a new invention only recently available in FF and Node, not at all in Safari. Naturally not that hard to write a custom JSON parser but the need itself is a bad thing.

No it's been there for ages. Finalized as part of ecmascript 5 What you are probably thinking of is the context parameter of the reviver callback. That is relatively recent and mostly a qol improvement

Sorry yes, i was thinking of the context object with source parameter.

The issue it solves is a big one though, since without it the JSON.parse functionality cannot parse numbers that are larger than 64bit float numbers (f.ex. bigints).

Post reply on HN