I don't mind languages using brackets. I used to do a lot of C and C# development and it never bothered me. The problem is that, at least to me, a Javascript code base of around 2000 lines already becomes unreadable, specially when you consider some common patterns like 'declare a function to define a namespace'. You normally end up with functions inside of functions, calling jQuery event handlers by declaring yet another inline function... By the time you use a 'for' loop, you already are 4 indentation levels/brackets in. Messy.
As you say, libraries work around a lot of these problems, but most of these workarounds are hacks. 'require.js', for example. And you shouldn't need to use underscore to get sane typing. About OO... It's debatable, I think languages like CoffeeScript and TypeScript have done a good job of abstracting the weirdness, but that's the thing: they are languages that compile to Javascript.
Anyway, I like Javascript, don't get me wrong, it's just that I don't think it's a great choice for big codebases. I know I get a lot of hate every time I say this, but to me Node.js is an excellent example of using the right tool for the wrong job. Had the energy been put behind developing the same concept as a library to a language with a decent standard library, so much work would've been saved 'catching up'.