I'm not sure how "commas inside strings in CSVs can cause bugs" becomes newsworthy, but I guess even the vibecoding generation needs to learn the same old lessons.
Moldova broke our data pipeline
31–40 of 61 posts
Re: Moldova broke our data pipeline
#32I really don't understand why people think it's a good idea to use csv. In english settings, the comma can be used as 1000-delimiter in large numbers, e.g. 1,000,000 for on million, in German, the comma is used as decimal place, e.g. 1,50€ for 1 euro and 50 cents. And of course, commas can be used free text fields. Given all that, it is just logical to use tsv instead!
JSON, just use JSON. Or XML, if you don't like JSON.
Re: Moldova broke our data pipeline
#33I really don't understand why people think it's a good idea to use csv. In english settings, the comma can be used as 1000-delimiter in large numbers, e.g. 1,000,000 for on million, in German, the comma is used as decimal place, e.g. 1,50€ for 1 euro and 50 cents. And of course, commas can be used free text fields. Given all that, it is just logical to use tsv instead!
JSON, just use JSON. Or XML, if you don't like JSON.
JSON brings its own set of problems. for example, look at the python generated JSON below.
> >>> json.dumps({ "X" : 1 '{"X": 73786976294838206464}'
Re: Moldova broke our data pipeline
#34Earlier quoted context omitted.
JSON, just use JSON. Or XML, if you don't like JSON.
JSON brings its own set of problems. for example, look at the python generated JSON below. > >>> json.dumps({ "X" : 1 '{"X": 73786976294838206464}' What's the parsing result in javascript ? What's the parsing result in Java ?
What's the difference to CSV?
number,73786976294838206464Re: Moldova broke our data pipeline
#35RFC 4180 [1] Section 2.6 says: "Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes." If the DMS output isn’t quoting fields that contain commas, that’s technically invalid CSV. A small normalization step before COPY (or ensuring the writer emits RFC-compliant CSV in the first place) would make the pipeline robust without renaming countries or changing delimiters. That…
That's the real shame but also the lesson, a perfectly good and specified format, but the apparent simplicity makes everyone ignore the spec and yolo out broken stuff.
This is why SQL is "broken", it's powerful, simple and people will always do the wrong thing.
Was teaching a class on SQL, half my class was reminding them that examples with concatenating strings was bad and they should use prepared statements (JDBC).
Come practice time, half the class did string concatenations.
This is why I love Linq and the modern parametrized query-strings in JS, they make the right thing easier than the wrong thing.
Re: Moldova broke our data pipeline
#36Earlier quoted context omitted.
JSON brings its own set of problems. for example, look at the python generated JSON below. > >>> json.dumps({ "X" : 1 '{"X": 73786976294838206464}' What's the parsing result in javascript ? What's the parsing result in Java ?
What's the difference to CSV? number,73786976294838206464
For CSV, I don't know how this comes out. It depends on the library/programming language. It might be 73786976294838210000 or it might throw an exception, or whatever. I'm just saying JSON will not solve your problems neither.
Re: Moldova broke our data pipeline
#37I really don't understand why people think it's a good idea to use csv. In english settings, the comma can be used as 1000-delimiter in large numbers, e.g. 1,000,000 for on million, in German, the comma is used as decimal place, e.g. 1,50€ for 1 euro and 50 cents. And of course, commas can be used free text fields. Given all that, it is just logical to use tsv instead!
> I really don't understand why people think it's a good idea to use csv.
Because it's easy to understand. Non-technical people understand it. There is tremendous value in that, and that it's underspecified with ugly edge cases doesn't change that.
Re: Moldova broke our data pipeline
#38I'm not sure how "commas inside strings in CSVs can cause bugs" becomes newsworthy, but I guess even the vibecoding generation needs to learn the same old lessons.
When they inevitably move to JSON I sure hope nobody's shop includes a quote in the name.
Re: Moldova broke our data pipeline
#39I really don't understand why people think it's a good idea to use csv. In english settings, the comma can be used as 1000-delimiter in large numbers, e.g. 1,000,000 for on million, in German, the comma is used as decimal place, e.g. 1,50€ for 1 euro and 50 cents. And of course, commas can be used free text fields. Given all that, it is just logical to use tsv instead!
CSV can handle commas in fields just fine (quotes are required in that case). The root problem here is not the format, it's a bug in the CSV exporter used.
Re: Moldova broke our data pipeline
#40I really don't understand why people think it's a good idea to use csv. In english settings, the comma can be used as 1000-delimiter in large numbers, e.g. 1,000,000 for on million, in German, the comma is used as decimal place, e.g. 1,50€ for 1 euro and 50 cents. And of course, commas can be used free text fields. Given all that, it is just logical to use tsv instead!
CSV can handle commas in fields just fine (quotes are required in that case). The root problem here is not the format, it's a bug in the CSV exporter used. https://news.ycombinator.com/item?id=47229064
Clearly this is the issue. This article was 2000 words of trying to work around the actual problem