Earlier quoted context omitted.
I understand that but micro VMs don't provide better security isolation than regular VMs. So that leaves faster boot times. Faster boot times and then the agent does what ? And at how many token/s? And what's the "time to first token" anyway? How do the time to first token and then the token/s inherent limitations of LLMs not totally dominate the running time? I just don't get the use case.
imagine installing an agent in slack at a company with 1000 employees, and you want each request to have its own VM for data analysis, downloading repos and working on them, ... regular VMs just use too much memory, a typical ubuntu uses 512 MB as a baseline
then there's the disk iops used for spinning up all these VMs (loading and booting a whole distro), the security attack vectors of an entire VM vs microVM, the maintenance of the images, the hypervisor abstraction to handle all this automation, ssh for the agent to run in the VM, etc.
compared to mounting an extracted container image to a folder, starting a microVM kernel with folder mount, with specific credentials attached. minimum memory and CPU allocated, minimum possible system resource use, fastest operation, least maintenance. you get more time, more resources, more security.
(micro VMs do provide better security isolation. they have kernels with fewer built-in vulnerabilities, fewer hardware drivers to exploit, a more locked-down network, and they lack a full OS's applications and filesystem permissions to exploit)