VToonify: Controllable high-resolution portrait video style transfer
1–10 of 23 posts
Re: VToonify: Controllable high-resolution portrait video style transfer
#2Also reviewed at Two minute papers with a few examples https://youtube.com/watch?v=C9LDMzMRZv8 (hold on to your papers...)
Re: VToonify: Controllable high-resolution portrait video style transfer
#3Also reviewed at Two minute papers with a few examples https://youtube.com/watch?v=C9LDMzMRZv8 (hold on to your papers...)
What a time to be alive!
Re: VToonify: Controllable high-resolution portrait video style transfer
#4Based on the numbers in the paper this is just a little bit too slow for use as a real time video effect. At ~0.1 seconds per frame we just need about a 3x improvement in performance to get to 30fps “real time” video frame rates.
And on that thought since it appears they used nVidia hardware based on the CUDA dependency, it would be interesting to see how this performs on something like an M1/M2 where there’s dedicated ML hardware to help offload and accelerate things.
Re: VToonify: Controllable high-resolution portrait video style transfer
#5Based on the numbers in the paper this is just a little bit too slow for use as a real time video effect. At ~0.1 seconds per frame we just need about a 3x improvement in performance to get to 30fps “real time” video frame rates. And on that thought since it appears they used nVidia hardware based on the CUDA dependency, it would be interesting to see how this performs on something like an M1/M2 where there’s dedicat…
nVidia hardware has dedicated ML silicon, though?
Re: VToonify: Controllable high-resolution portrait video style transfer
#6Well it's cool and all, but the results sit right at the very deepest part of the uncanny valley.
Re: VToonify: Controllable high-resolution portrait video style transfer
#7Based on the numbers in the paper this is just a little bit too slow for use as a real time video effect. At ~0.1 seconds per frame we just need about a 3x improvement in performance to get to 30fps “real time” video frame rates. And on that thought since it appears they used nVidia hardware based on the CUDA dependency, it would be interesting to see how this performs on something like an M1/M2 where there’s dedicat…
The paper also says they used 8 Tesla V100. They are GPUs that are dedicated for ML and quite a bit more powerful than a m2.
Re: VToonify: Controllable high-resolution portrait video style transfer
#8Based on the numbers in the paper this is just a little bit too slow for use as a real time video effect. At ~0.1 seconds per frame we just need about a 3x improvement in performance to get to 30fps “real time” video frame rates. And on that thought since it appears they used nVidia hardware based on the CUDA dependency, it would be interesting to see how this performs on something like an M1/M2 where there’s dedicat…
Does M1/M2 really outperform CUDA on beefy ML GPUs in tasks like this? I'd love to see numbers if so; this seems extremely surprising.
Re: VToonify: Controllable high-resolution portrait video style transfer
#9Well it's cool and all, but the results sit right at the very deepest part of the uncanny valley.
It's like a tech demo, a preview of the future. Give it 5 years and it will be super refined and probably the future of low cost animation for kids TV shows and stuff. Then even further, like how no one animates without a computer now, no one will animate without AI assistance.
Re: VToonify: Controllable high-resolution portrait video style transfer
#10Based on the numbers in the paper this is just a little bit too slow for use as a real time video effect. At ~0.1 seconds per frame we just need about a 3x improvement in performance to get to 30fps “real time” video frame rates. And on that thought since it appears they used nVidia hardware based on the CUDA dependency, it would be interesting to see how this performs on something like an M1/M2 where there’s dedicat…
The paper also says they used 8 Tesla V100. They are GPUs that are dedicated for ML and quite a bit more powerful than a m2.
I missed the bit about using 8 of them to run it! Wow that’s a lot of GPU horsepower to do this. More efficient to just use a vtuber style pipeline using unreal engine and the metahumans or other avatars… only need one good GOU for that.