Moldova broke our data pipeline

Post by booi » Tue, Mar 03, 2026, 8:55 AM UTC

teekert wrote: post #29

aquafox wrote: post #11

I really don't understand why people think it's a good idea to use csv. In english settings, the comma can be used as 1000-delimiter in large numbers, e.g. 1,000,000 for on million, in German, the comma is used as decimal place, e.g. 1,50€ for 1 euro and 50 cents. And of course, commas can be used free text fields. Given all that, it is just logical to use tsv instead!

I learned to program at 33 or so (in bioinformatics), my first real lesson a couple of days in: "Never ever use csv". I've never used pd.read_csv() without sep="\t". Idk where csv came from, and who thought it was a good idea. It must have been pre-spreadsheet because a tab will put you in the next cell so tabs can simply never be entered into any table by our biologist colleagues. I guess it's also why all our fancy…

Post by pasc1878 » Tue, Mar 03, 2026, 8:59 AM UTC

brazzy wrote: post #37

aquafox wrote: post #11

I really don't understand why people think it's a good idea to use csv. In english settings, the comma can be used as 1000-delimiter in large numbers, e.g. 1,000,000 for on million, in German, the comma is used as decimal place, e.g. 1,50€ for 1 euro and 50 cents. And of course, commas can be used free text fields. Given all that, it is just logical to use tsv instead!

> I really don't understand why people think it's a good idea to use csv. Because it's easy to understand. Non-technical people understand it. There is tremendous value in that, and that it's underspecified with ugly edge cases doesn't change that.

Post by kuschku » Tue, Mar 03, 2026, 9:27 AM UTC

whizzter wrote: post #35

simula67 wrote: post #15

RFC 4180 [1] Section 2.6 says: "Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes." If the DMS output isn’t quoting fields that contain commas, that’s technically invalid CSV. A small normalization step before COPY (or ensuring the writer emits RFC-compliant CSV in the first place) would make the pipeline robust without renaming countries or changing delimiters. That…

That's the real shame but also the lesson, a perfectly good and specified format, but the apparent simplicity makes everyone ignore the spec and yolo out broken stuff. This is why SQL is "broken", it's powerful, simple and people will always do the wrong thing. Was teaching a class on SQL, half my class was reminding them that examples with concatenating strings was bad and they should use prepared statements (JDBC).…

    @Dao
    public interface UserDao {
        @Query("SELECT * FROM user")
        List getAll();

        @Query("SELECT * FROM user WHERE uid IN (:userIds)")
        List loadAllByIds(int[] userIds);

        @Query("SELECT * FROM user WHERE first_name LIKE :first AND " +
            "last_name LIKE :last LIMIT 1")
        User findByName(String first, String last);

        @Insert
        void insertAll(User... users);

        @Delete
        void delete(User user);
    }

Post by andix » Tue, Mar 03, 2026, 11:07 AM UTC

toolslive wrote: post #36

andix wrote: post #34

Earlier quoted context omitted.

What's the difference to CSV? number,73786976294838206464

For CSV, I don't know how this comes out. It depends on the library/programming language. It might be 73786976294838210000 or it might throw an exception, or whatever. I'm just saying JSON will not solve your problems neither.

Post by Cthulhu_ » Tue, Mar 03, 2026, 12:04 PM UTC

derriz wrote: post #25

Unrelated to the fundamental issue (a part of your pipeline generates invalid CSV), I would never store the name of the country like this. The country's name is "The Republic of Moldova" and I would store it like this. Sure, the most common collation scheme for country names is to sort ignoring certain prefixes like "The Republic of", "The", "People's Democratic...", etc. but this is purely a presentation layer issue…