The Norway Problem

#1

Post by dedalus » Fri, Apr 02, 2021, 1:12 PM UTC

Re: The Norway Problem

#2

Post by pintxo » Sat, Apr 03, 2021, 7:34 AM UTC

> “While the website went down and we were losing money we chased down a number of loose ends until finally finding the root cause.”

And that's why you have a staging environment. Or you debug in production, whatever you prefer.

Re: The Norway Problem

#3

Post by pietroppeter » Sat, Apr 03, 2021, 7:34 AM UTC

I like strict yaml but I have used it very little. Anyone who uses it more that can give more feedback?

Re: The Norway Problem

#4

Post by grenoire » Sat, Apr 03, 2021, 7:36 AM UTC

I was helping out a friend of mine in the risk department of a Big 4; he was parsing CSV data from a client's portfolio. Once he started parsing it, he was getting random NaNs (pandas' nan type, to be more accurate).

I couldn't get access to the original dataset but the column gave it away. Namibia's 2-letter ISO country code is NA—which happens to be in pandas' default list of NaN equivalent strings.

It was a headache and a half...

Re: The Norway Problem

#5

Post by eitland » Sat, Apr 03, 2021, 7:36 AM UTC

pintxo wrote: post #2

> “While the website went down and we were losing money we chased down a number of loose ends until finally finding the root cause.” And that's why you have a staging environment. Or you debug in production, whatever you prefer.

Everybody has a testing environment. Some people are lucky enough enough to have a totally separate environment to run production in.

https://mobile.twitter.com/stahnma/status/634849376343429120

Re: The Norway Problem

#6

Post by grenoire » Sat, Apr 03, 2021, 7:37 AM UTC

grenoire wrote: post #4

I was helping out a friend of mine in the risk department of a Big 4; he was parsing CSV data from a client's portfolio. Once he started parsing it, he was getting random NaNs (pandas' nan type, to be more accurate). I couldn't get access to the original dataset but the column gave it away. Namibia's 2-letter ISO country code is NA—which happens to be in pandas' default list of NaN equivalent strings. It was a headac…

Verbatim from the docs, on read-csv:

    na_valuesscalar, str, list-like, or dict, default None

    Additional strings to recognize as NA/NaN. If dict passed, specific per-column NA values. By default the following values are interpreted as NaN: ‘’, ‘#N/A’, ‘#N/A N/A’, ‘#NA’, ‘-1.#IND’, ‘-1.#QNAN’, ‘-NaN’, ‘-nan’, ‘1.#IND’, ‘1.#QNAN’, ‘’, ‘N/A’, ‘NA’, ‘NULL’, ‘NaN’, ‘n/a’, ‘nan’, ‘null’.

You fix it by using `keep_default_na=False`, by the way.

Re: The Norway Problem

#7

Post by mseepgood » Sat, Apr 03, 2021, 7:42 AM UTC

grenoire wrote: post #4

I was helping out a friend of mine in the risk department of a Big 4; he was parsing CSV data from a client's portfolio. Once he started parsing it, he was getting random NaNs (pandas' nan type, to be more accurate). I couldn't get access to the original dataset but the column gave it away. Namibia's 2-letter ISO country code is NA—which happens to be in pandas' default list of NaN equivalent strings. It was a headac…

A Ms True also broke Apple's iCloud: https://twitter.com/RachelTrue/status/1365461618977476610

Re: The Norway Problem

#8

Post by grenoire » Sat, Apr 03, 2021, 7:46 AM UTC

mseepgood wrote: post #7

grenoire wrote: post #4

I was helping out a friend of mine in the risk department of a Big 4; he was parsing CSV data from a client's portfolio. Once he started parsing it, he was getting random NaNs (pandas' nan type, to be more accurate). I couldn't get access to the original dataset but the column gave it away. Namibia's 2-letter ISO country code is NA—which happens to be in pandas' default list of NaN equivalent strings. It was a headac…

A Ms True also broke Apple's iCloud: https://twitter.com/RachelTrue/status/1365461618977476610

That looks like an interesting hard-coded check, I wonder what it intended to fix.

Re: The Norway Problem

#9

Post by atoav » Sat, Apr 03, 2021, 7:47 AM UTC

pintxo wrote: post #2

> “While the website went down and we were losing money we chased down a number of loose ends until finally finding the root cause.” And that's why you have a staging environment. Or you debug in production, whatever you prefer.

I'd go further and say this is why you write tests. Creating tests that cover a lot (or all) possible inputs is sometimes not that hard and really pays off if you manage to catch a very common error like the Norway thing. Even better if you catch something that would have been a nightmare to fix in production.

I say this because two days ago I wrote a test that used all country codes as input. It took 15 minutes to write that test. During the whole testing session I found at least 5 mistakes of which 3 would have been quite dramatic.

Re: The Norway Problem

#10

Post by yakshaving_jgt » Sat, Apr 03, 2021, 7:48 AM UTC

> it’s equally true that extremely strict type systems require a lot more upfront and the law of diminishing returns applies to type strictness - a cogent answer to the question “why is so little software written in haskell?“

I was with the article up until that point. I don't agree that diminishing returns with regards to type strictness applies linearly. Term-level Haskell is not massively harder than writing most equivalent code in JavaScript — in fact I'd say it's easier and you reap greater benefit. Perhaps it's a different story when you go all-in on type-level programming, but I'm not sure that's what the author was getting at. This smells of the Middle Ground logical fallacy to me. Or of course the comment was tongue-in-cheek and I'm overreacting.