>> Animal brains can't abstract like (some of) our brains can.
You mean they can't come up with mathematical abstractions? That's right of course, but they seem perfectly able to "abstract" in the sense of drawing general rules from specific observations. For example, I noticed recently how my cat friend was happy to exit and enter the house from either of two doors and a window as the opportunity presented itself (i.e. depending on which one was open at the time he wanted to get in or out). E.g. I just happened to open the window and he hopped onto the ledge and out into the mystery of the night, entirely unaided by human hands or voices.
That's an abstraction: one door is like another door and they're both like a window. Both things lead in and out of the house and they can be open or closed at different times. If one is closed another may be open. Something like that, obviously I have no idea what concepts, exactly, he has in his head. And btw, "in", "out", "house", etc are also abstractions, so a house in the UK is like a house in France, or like one in Italy, or one in Greece: that's where the human things are; I guess.
I think it's important to understand this about animal intelligence, that it is very much capable of broad generalisation and abstract thought. Absolutely not to the same degree as humans because we got language and that probably comes with a whole other level of ability for abstraction and reasoning, but I don't believe it's possible for an animal to survive in the physical world if it can't go from the concrete to the abstract, the specific to the general, to some extent.
You mention ASI so I'm comfortable making this comment which is about AI, without feeling I'm hijacking your conversation. I think one only begins to understand how powerful the animal brain is, and how overlooked (in AI discourse) its ability to form such broad, useful abstractions that allow animals to navigate the physical world autonomously while setting and pursuing their own goals, when one tries to reproduce the same behaviour artificially, in computers.
Like you say, yes, maybe when we stop understanding what our computers produce, it will be because they moved a level up from where we are, as ASI. Maybe there is such a level; maybe there isn't. For the time being, there is no artificial system that can spontaneously move in the physical world as easily and effortlessly as my cat friend can (and he's a graceful animal; really, much closer to an African wildcat than its domestic descendant). If you want an AI system to understand the difference between "in" and "out" and that there are doors and windows that connect the two you have to somehow explicitly feed that information into it, either by hand-coding (e.g. the PDDL models used in Planning and Scheduling) or by training on carefully chosen examples of those concepts (as in machine learning), or at least some kind of objective that will lead the AI to develop them, or develop similar concepts (as in Reinforcement Learning). We're still very far from the capabilities of the animal brain.
Also, may I quibble about the fact that our brain, too, is an animal brain?