This article is a perfect example how science is getting led into dark areas by people who didn't learn quantum mechanics right or people who pretend to understand it, but can only blindly follow the formalism without much understanding of what they actually do. Every such article continuous to mysticize the whole subject by quoting famous scientists who were either puzzled by it at the time or scientists like John von Neumann who clearly gave a dumbed down view of the now called collapse (perhaps on request to skip the math).
I really appreciate this forum - as it is one of the last places that I know of where one can have a civil discussion - and therefore I will take the effort to show that pure quantum mechanics - with no additions - essentially explains the process of measurement which is not at all sudden as the name "collapse" would suggest. The reasoning comes from von Neumann himself, but now sometimes it's attributed also to Wojciech Żurek.
TLDR of below: All processes in nature, including the measuring process are unitary, the "collapse" is just an artifact of our ignorance about the exact state of the measuring aparatus.
Here it goes:
For simplicity, let's assume that psi describing our particle is a superposition of two eigenstates:
|psi> = c1 |1> + c2 |2>, i.e., |c1|^2 + |c2|^2 = 1.
Without loss of generality we can pick:
c1 = x and c2 = sqrt(1-x^2) exp(i phi), where x is a real number smaller than 1.
The density matrix of this pure state can then be written as rho = |psi>sqrt(1-x^2) exp(i phi) and xsqrt(1-x^2) exp(-i phi).
In the most complete scenario of a measurement, the density matrix of the system can change change in many ways including the diagonal terms of the density matrix. However in this simplistic example, a measurement will by necessity, bring only the non-diagonal terms to zero (I hope most of the interested readers will have enough background to understand why).
Now, the measuring device, as a macroscopic object, will have the number of degrees of freedom far greater than the simple particle which's state we're about to measure. This number will be the order of the Avogadro number (~ 10^23) - even the smallest human visible indicator will be this big. The measurement, by necessity, includes an interaction of our small system with the enormous measuring device.
Before the interaction the whole system (the particle and the measuring device) can be written as a tensor product of the two wavefunctions:
|Omega_before> = |Psi> ⊗ |Xsi> = ( c1 |1> + c2 |2> ) ⊗ |Xsi>
where |Xsi> represents the wavefunction of the measuring device and everything it interacts with before the measurement. When the interaction occurs the state of our measuring device changes unitarily (as everything in nature) according to the full Hamiltonian of the system, and with some regrouping of the terms, we can write the state after the interaction as:
|Omega_after> = c1 |1> ⊗ |Xsi_1> + c2 |2> ⊗ |Xsi_2>
This is the true state of the system as performed by nature.
The individual subsystems are no longer in pure states, but the whole system |Omega_after> (if we're able to completely describe it) - is.
Now, comes the final part, which some call the "collapse", but in reality it is just "an average" over all possible states of the bigger (measurement) system *which we declared apriori to not be the system of interest and states of which not able to follow because we measure with it*:
Tr_{over the degrees of freedom of Xsi} |Omega_after>In result we obtain a matrix after the measurement which is just formed with the diagonal elements
x^2 and 1-x^2, i.e., the probabilities of the two measurement results and non-diagonal terms being equal to zero.
Why are they zero?
Let's inspect one of the non-diagonal elements over which the above trace is taken:
x*sqrt(1-x^2)exp(i phi)
It is effectively zero, because the trace over the degrees of freedom of Xsi is a mutliple integral, again of the multiplicity of order of Avogardo number and a similar number functions which change in various ways. It is enough that only a fraction of such integrals will have a value lesser than 1 to guarantee that the product will be equal to zero.
And this is all. Any attempt to change this fact would need to reject quantum mechanics completely, because probability calculus is at the heart of it.