The CPU for complex stuff and GPU for simple number crushing is a really popular narrative. It's true in one sense and nonsense in another, determined solely on the software it's running.
If your program has one or two threads and spends all its time doing branchy control flow, a CPU will run it adequately and a GPU very poorly. If the program has millions of mostly independent tasks, a GPU will run it adequately and a CPU very poorly. That's the two limiting cases though and quite a lot of software sits somewhere in the middle.
The most concise distinction to draw between the hardware architectures is what they do about memory latency. We want lots of memory, that means it ends up far from the cores, so you have to do something while you wait for accesses to it. Fast CPUs use branch predictors and deep pipelines to keep the cores busy, fast GPUs keep a queue of coroutines ready to go and pick a different one to step along when waiting for memory. That's roughly why CPUs have a few threads - all the predictor and wind back logic is expensive. It's also why GPUs have many - no predictor or wind back logic, but you need to have a lot of coroutines ready to go to keep the latency hidden.
Beyond that, there's nothing either CPU or GPU can do that the other cannot. They're both finite approximations to Turing machines. There are some apparent distinctions from the hosted OS abstraction where the CPU threads get "syscall" and the GPU threads need to DIY their equivalent but the application doesn't care. Threads on either can call fprintf just fine. It makes a bit of a mess in libc but that's alright (and done in LLVM already).