I really don't understand why people think it's a good idea to use csv. In english settings, the comma can be used as 1000-delimiter in large numbers, e.g. 1,000,000 for on million, in German, the comma is used as decimal place, e.g. 1,50€ for 1 euro and 50 cents. And of course, commas can be used free text fields. Given all that, it is just logical to use tsv instead!
Moldova broke our data pipeline
11–20 of 61 posts
Re: Moldova broke our data pipeline
#12"Sanitize at the boundary"
Ah, but what _is_ the boundary, asks Transnistria?
Re: Moldova broke our data pipeline
#13Considering the scope, this could be more easily resolved by just stripping ", Republic of" from that specific string (assuming "Moldova" on its own is sufficient).
Re: Moldova broke our data pipeline
#14The majority of countries official names are in this format. We just use the short forms. "Republic of ..." is the most common formal country name: https://en.wikipedia.org/wiki/List_of_sovereign_states
Re: Moldova broke our data pipeline
#15RFC 4180 [1] Section 2.6 says:
"Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes."
If the DMS output isn’t quoting fields that contain commas, that’s technically invalid CSV.
A small normalization step before COPY (or ensuring the writer emits RFC-compliant CSV in the first place) would make the pipeline robust without renaming countries or changing delimiters.
That way, if/when the DMS output is fixed upstream, nothing downstream needs to change.
Re: Moldova broke our data pipeline
#16Come on man. What are we doing here. This is not even anything interesting like Norway being interpreted as False in YAML. This is just a straightforward escaping issue.
Re: Moldova broke our data pipeline
#17Earlier quoted context omitted.
CSV quoting is dialect dependent. Honestly you should just never use CSV for anything if you can avoid it, it's inferior to TSV (or better yet JSON/JSONL) and has a tendency to appear like it's working but actually be hiding bugs like this one.
Most CSV dialects have no problem having double quoted commas. The "dialect dependent" part is usually about escaping double quotes, new lines and line continuations. Not a portable format, but it is not too bad (for this use) either considering the country list is mostly static
I'd go so far as to say any implementation that doesn't conform to RFC 4180[1] is broken and should be fixed. The vast majority of implementations get this right, it's just that some that don't are so high profile it causes people to throw up their hands and give up.
Re: Moldova broke our data pipeline
#18"Sanitize at the boundary" Ah, but what _is_ the boundary, asks Transnistria?
LoL, good one.
Re: Moldova broke our data pipeline
#19That's a cool map.
Re: Moldova broke our data pipeline
#20The majority of countries official names are in this format. We just use the short forms. "Republic of ..." is the most common formal country name: https://en.wikipedia.org/wiki/List_of_sovereign_states
Sure, but why Moldova of all places? I've seen this form usually for places where there's a dispute for the short name, like Nice/Naughty Korea, Taiwan/West Taiwan, or Macedonia/entitled Greek government.