I noticed some things that concern me, and while I neither have an active attack, or the time* to formalize one, I'll leave these concerns here for discussion, and take part when time permits.
First, the KDF really bugs me. Essentially, 128 bits of the data that's used to generate the AES key is derived from msg_key, which is the SHA-1 of plaintext+some_other_jazz. Also, only some of the auth_key's bits are used, along with the plaintext-dependent msg_key, as food for the KDF. What this could mean is that you're really diluting your effective keyspace. (Partially) deriving a key from the hash of public data doesn't smell like good crypto-hygiene, and intuition tells me there's the potential for leaking key bits if I query it enough, or collect a lot of ciphertext traffic.
Second, there's no integrity checking going on with the ciphertext, so I could easily ask the server to decrypt anything; don't give me that freedom, because maybe I can fool you into doing something worse. Also, if you were doing Encrypt-then-MAC, like any good boy should, you'd save yourself from wasting decryptions on bogus ciphertext. This is one reason I can't buy the performance-driven reasoning for using Methuselah's hand-me-down modes; modern modes are safer and likely faster. It's akin to paying more for used, squeaky parts, when you can pay less for new, more efficient parts.
To a cryptographer, the one thing we learn early in our careers is that it doesn't pay to be different when it comes to choosing primitives and protocols. When we find things that work, and have earned their bones, we recycle them in new designs and continue building upon the confidence we have in them by carrying on with cryptanalysis, building and breaking proofs, and so forth. There are no bonus points for those who deviate, because, really, the battle we're losing isn't that of the cryptography itself; rather, it's the way we implement it and the way we interface it. Implementation is the friend of no one; do not make it harder for yourself.
I've been a little candid, I know, but mostly because to many of us, what we're looking at is a Goldbergian gauntlet of decision-making that leaves us asking ourselves, "Okay, they took the long way home, but did they get there?" If there's a gain, I'm not seeing it, and the defense isn't leaving me optimistic that we will.
With all sincerity, and admiration for your desire to help others reap the benefits of cryptography, if enough of us tell you that there's a better, safer, and quicker way to getting this right, would you lend us an ear?
- Justin
P.S. I'm okay with being wrong, but I'm more concerned with helping you get things right.
*I have a toddler.