The fractional scaling is garbage in this setup; and not just because you put the numbers in backwards from what you'd expect(2x is 2x smaller, and 0.5 seems to be 150% larger). It seems like it just scales the rendered output; everything is blurry! Scaling the DPI up to 144 however makes almost everything nice. The term, Chrome, VSCode, Intellij and etc all seem to adjust properly based on the DPI setting. The fractional scaling has a multiplier effect on that though, and so then reducing the DPI makes it smaller but still blurry. It also causes a lot of lag.
So, fractional scaling in XUbuntu seems to be scaling the video output and not just the UI like you would expect. Adjusting this in VirtualBox itself leads to better performance.