I think there is a point about fairness as well. Taxes are progressive for a reason. Those with the broadest shoulders should bear the greater burden.
Warrent Buffet is perhaps giving away a lot of what he maybe would have given in taxes but plenty of billionaires or super rich just spend it on their spoiled sons and daughters, some who end up in rehab, spent a tremendous amount of resources to create a parallel society where meritocracy applies only to some people and/or just waste it in obscene luxuries such as a 10k bottle of champaign, etc and to boot it all, at the end of the day, they are rich only because they are in many ways not paying the working people the proper wages so that they can pay themselves obscene amounts. I mean, how much more smart are the bank CEOs who got us in this mess to start with to justify a salary 100 times more than others working there.
But even on a practical rather that political level, for most of these supper rich, the difference between 1 million or 2 million does not mater but in some silly psychological keeping up with the jones nonsense and for many even the keeping up with the jones does not apply because they are the jones. But for a lot of "working poor" lowering the taxes from 25% to 17% makes a real difference, a difference of a kind which may mean that their children will properly be looked after and have all the support to say study and do well in school.
Finally, you say you want Warrent Buffet to keep the money and you don't want people to take it away from him. Well, I don't want people to take my money away through taxes either. No one does. Its not about want, or even opinion, its about fairness. Some of the deficit was built up because the rich pay so little and to ask the non rich to bear all the burden is a double wammy. No wonder you have 250k people protesting in Israel and kids burning chain stores in London.