Income tax itself isn't the complicated part of taxes. I encourage everyone to do their own taxes by hand sometime. There are like 4 boxes for wage income and income tax, and 100s of boxes for everything else. And besides the boxes you do fill out, there is all the reading and calculating required to figure out you don't have to fill in even more boxes (e.g. in order to know if you have to fill in the AMT form, you potentially have to do all the AMT calculations).
I think the biggest single thing that would simplify the tax code is removing the separate capital gains tax and just counting it all as income. There is significant complexity in the tax return to calculate everything independently and to show that your capital gains aren't really income. You could eliminate Schedule D, Form 8949 (which you have to up to 4 times), and the Capital Gains/Schedule D worksheets to figure income.
HSAs are also super painful. E.g. CA doesn't respect the tax-exempt status of HSAs, so you have to add the HSA back into your income so they can tax it. There are also a lot of bookkeeping requirements for relatively low amounts of money. In practice, they aren't very useful since the fees are high, the returns are low, and their function is redundant with insurance (i.e. paying premiums now to cover large future expenses).
In general, there are a ton of random little exemptions and extra taxes that you might qualify for and it just takes time to figure out if they apply to you. E.g. Form 8959 (Additional Medicare Tax) is a 0.9% tax for income over a certain amount. They should have just adjusted the income tax graduations in an equivalent way. Form 8960 (Net Investment Income Tax) has a similar problem. CA gives a $60 renter's credit if you make under certain amounts, which is just too insignificant to matter (e.g. 0.3% of $1500/month annualized).