I think the unlimited detail refers to rendering speed, not memory requirements. The data has to be redundant to fit in memory. He also claims it runs on a laptop without GPU. So it can't be just octree or any other regular acceleration structure. So if it isn't scam, it's something new. From the author (old post on Beyond3D): "firstly the system isn’t ray tracing at all or anything like ray tracing. Ray tracing uses…
I'm fairly certain that "sorting to find the one atom for each pixel on the screen" is exactly the definition of raytracing.
So yes. This is a kind of raytracing but I think that the implementation is radically different to the commonly used raytracing implementaions.