The full, concatenated and uglified source of one-page JS web apps can easily run to >600kB. Whether my DOM manipulation library consumes 4kB of that or 35kB isn't especially important to me. I'm more concerned about features, cross browser support and familiarity other developers may, or may not have, with the API. I'll be happy to check this lib out though, the API and docs look nicely put together.
If you do it right you don't even have the client send/receive an HTTP 304. For our app we have a lot of modular dependencies but we serve all of them from a unique prefix per build (eg. 'assets-SHA(build)/js/foo.js'). The second page load and beyond has zero requests for the same static asset. Whenever we have a new build of the app the unique prefix gets auto updated to force clients to use the latest versions.
For HTTPS make sure to mark your assets as public or they won't be cached:
Cache-Control:max-age=31557600, public
[1]: http://www.semicomplete.com/blog/geekery/ssl-latency.html