- This wouldn't work for larger payloads. AVs flag binary looking data that is larger than a certain size and that is later processed or assigned to a variable.
- Their veil project has some problems. Py2EXE gets marked as malware by some AVs in many cases just because it is Py2EXE. Same thing with non-commonly used obfuscators. Basically, they just pick up on the fact that something is obfuscated. If that obfuscator is not commonly used in goodware programs, it is marked as malware. This is kind of a dumb strategy on the part of AV engines, but it works okay.
You're not going to catch new malware with static (or dynamic for that matter) analysis anyway. Thing is, the problem is ill-defined.
What is malware?
Is it a program that does something a user doesn't want? If users knew what regular programs do, they wouldn't be okay with most of it either.
Is it a program that does some obfuscating tricks and exploits undocumented functionality in the system? Plenty of legit programs including a lot of AV engines do that as well.
The only usable definition in my opinion is that it is a program that makes the user unhappy with no easily accessible way of removing it completely.
This is why the only solution seems to be to only allow installation from a trusted repository. I am still not sure why Windows/Apple OSX haven't adopted such a strategy (with a developer mode override option for some advanced users).