> what justifies our choice of a particular prior
Right that's a good question. I'll point to an answer Sean Carroll gave in an AMA episode [0] of his Mindscape podcast:
The pros and cons of Bayesian reasoning are almost all in the choice of a prior. People who are pro-Bayesian will say, look, as long as your priors aren't completely crazy, if you collect enough data, the priors cease to matter. [...] The promise of Bayesian reasoning is that data overwhelms your prior ultimately. And therefore, there is no algorithmic way of choosing what your prior should be. It's a little bit fuzzy to say when things are priors and when things are posteriors because we all have certain inclinations, intuitions, pictures of the world, and that's perfectly okay. But as a good Bayesian, you shouldn't be too worried about picking your priors. You should be mostly worried about updating those priors when data comes in, when information comes in.
> Chains of reasoning have to bottom out somewhere
Absolutely. We should do our best to keep asking the why question, but at the end, we'll be left with a brute-force fact. The question then is, where do we stop, right? In the context of this thread, religion wants to say, God is the final answer. What caused God? Nothing, God is His own cause (kalam cosmological argument). And as a naturalist I'd say, the universe can be its own cause. There's no rational inconsistency there, contrary to what the kalam argument claims. Theists and atheists give different credence to these two viewpoints. And until we have more data, the question cannot really be settled with certainty. So both will keep trying to justify why one viewpoint should have a higher prior over the other, based on secondary evidence.
[0] https://www.preposterousuniverse.com/podcast/2023/04/03/ama-...