Here's a trippy idea I have about consciousness, which arose from thinking about recent AI advances and also watching kids develop.
I think children's main "cost function" is the ability to predict the future. This might start out, for example, as being - how will this vertical line move as I move my head. Later, where is the ball going. And they are essentially building a "world model" in their brains, starting with the very simple like this and recursively building more complex predictors. When they predict correctly, happy feedback reinforces the connections that are firing, when they are wrong, they weaken. Just a really simple feedback algorithm that is super robust.
So the brain is building this world model, and it's essentially gradually compressing a description of the environment into a structure made of neurons. And this is the ultimate survival tactic: model the environment explicitly in your head, then adapt your behaviour to fit it. The better you are adapating to future states (dodging that tree as you run) the better survival chance you have.
At some point (complete speculation) we then begin to do something quite strange: we develop a world model _of ourselves_. We get to a level of sophistication where we begin to predict the future states of our own brains. This might emerge naturally as a way to compress existing learnt behaviour. For example, we re-learn to follow lines in a smarter way, particularly as other parts of the brain learn useful things that we can re-use in our line following. This treats our existing model as a cost function, and we learn a model of the model. But it eventually starts to model the higher-level models the brain has, higher up the abstraction stack.
And somehow, the modelling of our brain function creates a chaotic feedback loop that leads to the sensation of consciousness. It's super handwavey, I know, but somehow this recursion feeds awareness. It's like the abilty to see yourself thinking. Consider meditating and the way words appear in consciousness... you get to a point where you can observe what you're going to say before you say them, and I conjecture that is the modelling of the model that's going on.
And this is useful for survival, as you can optimise the way you think, compressing your circuits further, but also has this weird side-effect of creating awareness.
It also explains why consciousness takes time to develop - because you need to develop a model of yourself, but before that you need a model of the world.