Moldova broke our data pipeline

This very clearly seems like a bug either in their DMS script, or in the DMS job that they don't directly control, since CSV clearly allows for escaping commas (by just quoting them). Would love to see a bug report being submitted upstream as well as part of the "fix".

Re: Moldova broke our data pipeline

#5

Post by shalmanese » Tue, Mar 03, 2026, 5:24 AM UTC

Did you really name your breakaway republic Sealand'); DROP TABLE Countries;--?

Re: Moldova broke our data pipeline

#6

Post by zarzavat » Tue, Mar 03, 2026, 5:29 AM UTC

franciscop wrote: post #4

This very clearly seems like a bug either in their DMS script, or in the DMS job that they don't directly control, since CSV clearly allows for escaping commas (by just quoting them). Would love to see a bug report being submitted upstream as well as part of the "fix".

CSV quoting is dialect dependent. Honestly you should just never use CSV for anything if you can avoid it, it's inferior to TSV (or better yet JSON/JSONL) and has a tendency to appear like it's working but actually be hiding bugs like this one.

Re: Moldova broke our data pipeline

#7

Post by j16sdiz » Tue, Mar 03, 2026, 5:43 AM UTC

zarzavat wrote: post #6

franciscop wrote: post #4

This very clearly seems like a bug either in their DMS script, or in the DMS job that they don't directly control, since CSV clearly allows for escaping commas (by just quoting them). Would love to see a bug report being submitted upstream as well as part of the "fix".

CSV quoting is dialect dependent. Honestly you should just never use CSV for anything if you can avoid it, it's inferior to TSV (or better yet JSON/JSONL) and has a tendency to appear like it's working but actually be hiding bugs like this one.

Most CSV dialects have no problem having double quoted commas.

The "dialect dependent" part is usually about escaping double quotes, new lines and line continuations.

Not a portable format, but it is not too bad (for this use) either considering the country list is mostly static

Re: Moldova broke our data pipeline

#8

Post by davecahill » Tue, Mar 03, 2026, 5:55 AM UTC

I was expecting a Markdown-related .md issue. :)

Re: Moldova broke our data pipeline

#9

Post by Surac » Tue, Mar 03, 2026, 6:18 AM UTC

I personaly would shy away from binary formats whenever possible. For my column based files i use TSV or the pipe char as delimiter. even excel allowes this files if you include a "del=|" as first line

Re: Moldova broke our data pipeline

#10

Post by inevletter » Tue, Mar 03, 2026, 6:27 AM UTC

Huge skill issue. Nothing to see here.