Yup, using AI for any serious tax calculation or even advice is a
REALLY BAD idea.
A close relative is a top expert in US Trust & Estate Tax law working at a well-known BigLaw firm. Of course they have substantial AI initiatives, integration with their system, mandatory training, etc.
She finds tha AI marginally useful for some things, but overall not very much and there are serious errors, particularly the types of errors only a top expert would catch.
One of the big examples is that in the world of T&E law, there are a lot of mediocre (to be kind) attorneys who claim expertise but are very bad at it (causing a lot of work for the more serious firms and a lot of costs & losses for the intended heirs). The mediocre-minus attorneys of course also write blogs and papers to market themselves, often in greater volume than the top experts. Many of these blogs/papers are seriously WRONG, as in giving the exact opposite of the right advice.
Everyone here sees where this is going. The AI has zero ability to reason or figure out which parts of its training input are from actual top experts and which are dreck. The AI can not reason, and can not even validly check their 'thinking' against existing tax code (which is massive), or the regulations and rulings (which are orders of magnitude more massive). So, the AI gives advice that is confident, cheerful, and WRONG.
Worse yet, the LLM's advice is wrong in ways only a top expert would know, and in ways that will massively screw the heirs. But the errors will likely only be discovered decades later, when it is too late to fix.
Seriously, do NOT use LLMs for tax advice, unless you are also consulting a TOP professional. And skipping the LLM part is best.
My relative is quite frustrated and annoyed by the whole thing, which should be more helpful with these massive code/regs/rulings, but finds it often more work than just using the standard WestLaw/Lexis legal database searches.