Earlier quoted context omitted.
So it always depends on the implementation. If you need something unambiguously specified, then XML with XSD is still a valid option. All number types are specified exactly, and you can use extensions for custom number types.
what's wrong with protobuf & friends ?
Moldova broke our data pipeline
51–60 of 61 posts
Re: Moldova broke our data pipeline
#52Re: Moldova broke our data pipeline
#53Earlier quoted context omitted.
That's the real shame but also the lesson, a perfectly good and specified format, but the apparent simplicity makes everyone ignore the spec and yolo out broken stuff. This is why SQL is "broken", it's powerful, simple and people will always do the wrong thing. Was teaching a class on SQL, half my class was reminding them that examples with concatenating strings was bad and they should use prepared statements (JDBC).…
I also really like the way Androidx's Room handles query parameters and the corresponding APIs. @Dao public interface UserDao { @Query("SELECT * FROM user") List getAll(); @Query("SELECT * FROM user WHERE uid IN (:userIds)") List loadAllByIds(int[] userIds); @Query("SELECT * FROM user WHERE first_name LIKE :first AND " + "last_name LIKE :last LIMIT 1") User findByName(String first, String last); @Insert void insertAl…
The Linq code is native C# that can be strongly typed for ID's,etc but you can "think" in SQL terms by writing Where,Select,OrderBy and so on (I will admit that the C# world hasn't really gotten there in terms of promoting strongly typed db ID's yet but there support is there).
Re: Moldova broke our data pipeline
#54Unrelated to the fundamental issue (a part of your pipeline generates invalid CSV), I would never store the name of the country like this. The country's name is "The Republic of Moldova" and I would store it like this. Sure, the most common collation scheme for country names is to sort ignoring certain prefixes like "The Republic of", "The", "People's Democratic...", etc. but this is purely a presentation layer issue…
While true, the default and naive sort of anything is alphabetical, and you'd need to implement this more advanced nondefault sort in every possible client. Personally I've never seen any sort where "The Republic of Moldova" would be sorted at "M".
This sometimes causes problems for the UK, which can be sorted as U, G, B, or even E (presumably for "England", making it especially annoying for people in the other countries of the UK).
Re: Moldova broke our data pipeline
#55Earlier quoted context omitted.
I also really like the way Androidx's Room handles query parameters and the corresponding APIs. @Dao public interface UserDao { @Query("SELECT * FROM user") List getAll(); @Query("SELECT * FROM user WHERE uid IN (:userIds)") List loadAllByIds(int[] userIds); @Query("SELECT * FROM user WHERE first_name LIKE :first AND " + "last_name LIKE :last LIMIT 1") User findByName(String first, String last); @Insert void insertAl…
It's one of the better abstractions given the lack of first class expressions in Java, having used EfCore/Linq a while I'd be hard pressed to like going back though. The Linq code is native C# that can be strongly typed for ID's,etc but you can "think" in SQL terms by writing Where,Select,OrderBy and so on (I will admit that the C# world hasn't really gotten there in terms of promoting strongly typed db ID's yet but…
create.select(AUTHOR.FIRST_NAME, AUTHOR.LAST_NAME, count())
.from(AUTHOR)
.join(BOOK).on(AUTHOR.ID.equal(BOOK.AUTHOR_ID))
.where(BOOK.LANGUAGE.eq("DE"))
.and(BOOK.PUBLISHED.gt(date("2008-01-01")))
.groupBy(AUTHOR.FIRST_NAME, AUTHOR.LAST_NAME)
.having(count().gt(5))
.orderBy(AUTHOR.LAST_NAME.asc().nullsFirst())
.limit(2)
.offset(1)Re: Moldova broke our data pipeline
#56I really don't understand why people think it's a good idea to use csv. In english settings, the comma can be used as 1000-delimiter in large numbers, e.g. 1,000,000 for on million, in German, the comma is used as decimal place, e.g. 1,50€ for 1 euro and 50 cents. And of course, commas can be used free text fields. Given all that, it is just logical to use tsv instead!
The JSON in CSVs does piss off the Jetbrains table viewer sometimes though, it will randomly decide to stop parsing a 50k line CSV at halfway through the JSON of line 300ish even though that JSON is no different from other JSON it parsed just fine.
But python reads and writes them fine, as does whatever SQL engine I'm touching, as does other tools.
Re: Moldova broke our data pipeline
#57RFC 4180 [1] Section 2.6 says: "Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes." If the DMS output isn’t quoting fields that contain commas, that’s technically invalid CSV. A small normalization step before COPY (or ensuring the writer emits RFC-compliant CSV in the first place) would make the pipeline robust without renaming countries or changing delimiters. That…
Re: Moldova broke our data pipeline
#58Earlier quoted context omitted.
It's one of the better abstractions given the lack of first class expressions in Java, having used EfCore/Linq a while I'd be hard pressed to like going back though. The Linq code is native C# that can be strongly typed for ID's,etc but you can "think" in SQL terms by writing Where,Select,OrderBy and so on (I will admit that the C# world hasn't really gotten there in terms of promoting strongly typed db ID's yet but…
In that case, I'd recommend jooq, which is just linq in Java :D create.select(AUTHOR.FIRST_NAME, AUTHOR.LAST_NAME, count()) .from(AUTHOR) .join(BOOK).on(AUTHOR.ID.equal(BOOK.AUTHOR_ID)) .where(BOOK.LANGUAGE.eq("DE")) .and(BOOK.PUBLISHED.gt(date("2008-01-01"))) .groupBy(AUTHOR.FIRST_NAME, AUTHOR.LAST_NAME) .having(count().gt(5)) .orderBy(AUTHOR.LAST_NAME.asc().nullsFirst()) .limit(2) .offset(1)
Re: Moldova broke our data pipeline
#59I was expecting a Markdown-related .md issue. :)
(Google did not do proper market testing: https://claude.md goes to what you might expect but https://gemini.md/ is a Moldovan condiment vendor... )
Re: Moldova broke our data pipeline
#60RFC 4180 [1] Section 2.6 says: "Fields containing line breaks (CRLF), double quotes, and commas should be enclosed in double-quotes." If the DMS output isn’t quoting fields that contain commas, that’s technically invalid CSV. A small normalization step before COPY (or ensuring the writer emits RFC-compliant CSV in the first place) would make the pipeline robust without renaming countries or changing delimiters. That…