Agreed - it seems this is to be expected. There is bound to naturally be some "reasoning" / self-reflection / self-correction data in the base model training set. For the most part text on the web is going to be the end result of reasoning rather than the process itself (with reflection, correction), but there is bound to be some "but on second thoughts ...", and "that didn't work out, so ..." etc.

I've never seen it myself, but I've heard that Sonnet 3.5 occasionally self-corrects, although nominally not a "reasoning" model (OTOH, Anthropic don't like the "reasoning" label, and prefer to refer to a continuum of abilities). Presumably this is just a reflection of some such data in the training set.

Of course, as with anything else, the LLM is just predicting based on patterns it saw during training, so any "self-reflection" it generates isn't the model itself reflecting, and there is no reason other than luck and prompt vs training set similarity to expect that this will be valid/useful reflection/reasoning.

Where RL comes in is encouraging goal-based behavior (predictions) so that any reasoning step(s) and/or process (entire CoT) are more likely to be coherent and result in a valid CoT, and it seems this will often result in emphasizing reflection and self-correction where needed to keep the CoT on track.