The downsides aren't unique to Japanese keyboards. On Danish keyboard " requires the use of the shift key, same for = and the \ in on the 7 keys, along with /, requiring you to hit option + shift + 7 (on a mac keyboard, I believe it's Alt + shift + The Japanese keyboard in the article does have { printed on a key, it's Option + Shift + 8 and the Danish Mac keyboard, and the key isn't labels with {, } is on the 9.
As a decade long Emacs user, I can not follow: what's the problem with modifier keys?! Just kidding, I know that some people have a hard time typing them, especially when your day job consists of doing exactly that for eight hours.
Lets say your keyboard requires modifier keys for a specific symbol, but your program treats that input differently depending on whether modifier keys are held or not. Or another bad one, lets say your keyboard requires multiple strokes for a specific symbol, for example backticks, and your language uses backticks as quotes...
So many language and programs designed with strange symbols and keyboard modifiers are close to unusable in other keyboard layouts.