Stupid question for people who know more about these things. Why can’t we fight the warmup time by running an already warmed up snapshot of the program? Or say dumping some data structure when it hits steady state to give hints to the JIT the next time it runs?
Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)
11–20 of 93 posts
Re: Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)
#12It would be better if VM comparisons included JSC rather than, or in addition to, V8. JSC tends to outperform V8 so if you find a pathology in V8 it’s just not so surprising. It would be more interesting if you found a pathology in JSC. I think that the use of small benchmarks obscures what’s going on. The VM is trying to win in the average. It’s like a professional gambler. Observing that the VM did something dumb f…
> - for some program, how long does it take to run that program. Start to finish. No ignoring warmup.
This methodology likely comes from Java, which has long-running server applications. "How long does it run" is often "until someone hits ^C". Here, startup cost can be slow as long as the peak performance is fine. It's accepted that the first minute or two of the server are slow, but that's small compared to the month or so that the server will be running for.
> This tells you how good a VM is.
I think papers like this approach it from the wrong angle. I don't care about the VM's theoretical peak performance. I care about being able to measure and track performance in a reliable way. Put simply, I'm fine with bad codegen as long as I can consistently measure it. Feel free to improve it, but adding to sometimes give me good codegen, unreliably, is much more frustrating than bad codegen. But this seems to be the way the VMs are going, with things like probabilistic profiling.
If I refactor my code and replace for(let i = 0; i I work on a particularly demanding website in my free time ( https://noclip.website/#smg/AstroGalaxy , unfortunately won't run in WebKit due to missing WebGL 2 ), and performance varies drastically from Chrome release to release, and I do extensive testing with node.js to make sure that I'm getting good codegen.
Re: Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)
#13Stupid question for people who know more about these things. Why can’t we fight the warmup time by running an already warmed up snapshot of the program? Or say dumping some data structure when it hits steady state to give hints to the JIT the next time it runs?
[deleted]
Re: Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)
#14Stupid question for people who know more about these things. Why can’t we fight the warmup time by running an already warmed up snapshot of the program? Or say dumping some data structure when it hits steady state to give hints to the JIT the next time it runs?
That would be great but it’s hard because the trick, at least in JS VMs, is to have the JIT specialize based on the heap.
So you’d need a heap snapshot or some way to link the generated code to a different heap.
Maybe not impossible, just hard enough that it’s not widespread.
Re: Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)
#15It would be better if VM comparisons included JSC rather than, or in addition to, V8. JSC tends to outperform V8 so if you find a pathology in V8 it’s just not so surprising. It would be more interesting if you found a pathology in JSC. I think that the use of small benchmarks obscures what’s going on. The VM is trying to win in the average. It’s like a professional gambler. Observing that the VM did something dumb f…
> - for some program, how long does it take to run that program. Start to finish. No ignoring warmup. This methodology likely comes from Java, which has long-running server applications. "How long does it run" is often "until someone hits ^C". Here, startup cost can be slow as long as the peak performance is fine. It's accepted that the first minute or two of the server are slow, but that's small compared to the mont…
I know that the warmup skipping comes from Java. It was a mistake there. Saying that it’s because Java is for servers is a lame excuse and may be getting it backwards - maybe Java only succeeded on servers because all the tuning ignored warmup.
I hear ya that having tools would be great - but the best speedups do come about from probabilistic methods so it would be weird to rely on whatever a profiler told you.
Re: Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)
#16Haven't we learned that VMs aren't worth it? The amount of engineering resources it takes to design a half-way decent one is just absolutely staggering. We all know that theoretically they can be faster than native code, but how many people have actually experienced that? The costs are too high: the warm-up costs, the engineering costs, the complexity costs. If every single language just generated LLVM IR and compiled it, we'd be in a lot better place and we'd probably have better, or at least more predictable, performance.
Like more resources have probably been put into the JVM than any other VM or compiler on earth and what has it given us, exactly? Performance is still worse than C and with homogenized operating systems, as well as the move towards the web, the portability guarantees don't feel very important. If I'm missing something, please tell me!
Re: Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)
#17Stupid question for people who know more about these things. Why can’t we fight the warmup time by running an already warmed up snapshot of the program? Or say dumping some data structure when it hits steady state to give hints to the JIT the next time it runs?
That would be great but it’s hard because the trick, at least in JS VMs, is to have the JIT specialize based on the heap. So you’d need a heap snapshot or some way to link the generated code to a different heap. Maybe not impossible, just hard enough that it’s not widespread.
It does seem like you could do JIT with a persistent cache which stores the JIT output along with a key that's a hash of all the relevant system parameters like CPU model and VM parameters like heap size. This would mean that the typical case of re-running a program in the same environment would be pre-warmed.
Re: Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)
#18Ugh at risk of being the grumpy old programmer: jit code is just never going to touch actual handwritten native code in performance. It's been like 50 years. If we care about performance we need languages for it.
I mostly agree, but compilers intentionally don't explore every optimization path to save compilation time, and a JIT is able to trace the program for real data and find places that could benefit from additional attention. I don't think JITs are the only way to address that trade-off in compilation time vs runtime performance metrics, and whether their benefits are worth their other costs is an interesting, program-d…
Any languages explore both? Compile to native and run with a jit which profiles and optimizes further based on runtime behavior?
Re: Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)
#19The title doesn't do the surprising conclusion justice.
"When we set out to look at how long VMs take to warm up, we didn’t expect to discover that they often don’t warm up. But, alas, the evidence that they frequently don’t warm up is hard to argue with."
By not warming up they refer to instances when early performance is higher than later performance or when the performance doesn't settle.
Re: Why Aren’t More Users More Happy With Our VMs? Part 1 (2018)
#20There are a couple of talks associated with this post and paper.
Virtual Machine Warmup Blows Hot and Cold
https://www.youtube.com/watch?v=LgCHAU8ZB00
Why Aren't More Users More Happy With Our VMs?