141–150 of 316 posts
Has anyone managed to run the thing? I got the streamlit demo to start after fighting with pytorch, mamba, and pip for half an hour, but the demo runs out of GPU memory after a little while. I have 24GB on GPU on the machine I used, does it need more?
Earlier quoted context omitted.
We do it for text, audio and bitmapped images. A 3D scene file format is no different, you could train a model to output a blender file format instead of a bitmap. It can learn anything you have data for. Heck, we do it with geospatial data already, generating segmentation vectors. Why not 3D?
>3D scene file format is no different Not in theory, but the level of complexity is way higher and the amount of data available is much smaller. Compare bitmaps to this: https://fossies.org/linux/blender/doc/blender_file_format/my...
Earlier quoted context omitted.
> you'll describe something and you'll get a full 3D scene, with 3D models, source of lights set up, etc. I'm always confused why I don't hear more about projects going in this direction. Controlnets are great, but there's still quite a lot of hallucination and other tiny mistakes that a skilled human would never make.
Blender files are dramatically more complex than any image format, which are basically all just 2D arrays of 3-value vectors. The blender filetype uses a weird DNA/RNA struct system that would probably require its own training run. More on the Blender file format: https://fossies.org/linux/blender/doc/blender_file_format/my...
Earlier quoted context omitted.
Blender files are dramatically more complex than any image format, which are basically all just 2D arrays of 3-value vectors. The blender filetype uses a weird DNA/RNA struct system that would probably require its own training run. More on the Blender file format: https://fossies.org/linux/blender/doc/blender_file_format/my...
But surely you wouldn't try to emit that format directly, but rather some higher level scene description? Or even just a set of instructions for how to manipulate the UI to create the imagined scene?
Earlier quoted context omitted.
Stable diffusion open source release and llama release
But what technically allowed for so much progress? There’s been open source AI/ML for 20+ years. Nothing comes close to the massive milestones over the past year.
Take AlexNet - the major "oh shit" moment in image classification.
It had an absolutely mind-blowing number of parameters at a whopping 62 million.
Holy shit, what a large network, right?
Absolutely unprecedented.
Now, for language models, anything under 1B parameters is a toy that barely works.
Stable diffusion has around 1B or so - or the early models did, I'm sure they're larger now.
A whole lot of smart people had to do a bunch of cool stuff to be able to keep networks working at all at that size.
Many, many times over the years, people have tried to make larger networks, which fail to converge (read: learn to do something useful) in all sorts of crazy ways.
At this size, it's also expensive to train these things from scratch, and takes a shit-ton of data, so research/discovery of new things is slow and difficult.
But, we kind of climbed over a cliff, and now things are absolutely taking off in all the fields around this kind of stuff.
Take a look at XTTSv2 for example, a leading open source text-to-speech model. It uses multiple models in its architecture, but one of them is GPT.
There are a few key models that are still being used in a bunch of different modalities like CLIP, U-Net, GPT, etc. or similar variants. When they were released / made available, people jumped on them and started experimenting.
Has anyone managed to run the thing? I got the streamlit demo to start after fighting with pytorch, mamba, and pip for half an hour, but the demo runs out of GPU memory after a little while. I have 24GB on GPU on the machine I used, does it need more?
Yeah, got a 24GB 4090, try to reduce the number of frames decoded to something like 4 or 8. Although, keep in mind it caps the 24Gb and goes to RAM (with the latest nvidia drivers).
Earlier quoted context omitted.
I think these are the main drivers behind the progress: - Unsupervised learning techniques, e.g. transformers and diffusion models. You need unsupervised techniques in order to utilize enough data. There have been other unsupervised techniques in the past, e.g. GANs, but they don't work as well. - Massive amounts of training data. - The belief that training these models will produce something valuable. It costs betwe…
I don't think transformers or diffusion models are inherently "unsupervised", especially not the way they're used in Stable Diffusion and related models (which are very much trained in a supervised fashion). I agree with the rest of your points though.
You're right that conditional generation start to blur the lines though.
Earlier quoted context omitted.
But what technically allowed for so much progress? There’s been open source AI/ML for 20+ years. Nothing comes close to the massive milestones over the past year.
People figuring out how to train and scale newer architectures (like transfomers) effectively, to be wildly larger than ever before. Take AlexNet - the major "oh shit" moment in image classification. It had an absolutely mind-blowing number of parameters at a whopping 62 million. Holy shit, what a large network, right? Absolutely unprecedented. Now, for language models, anything under 1B parameters is a toy that bare…
SDXL is 6.6 billion.
In the video towards the bottom of the page, there are two birds (blue jays), but in the background there are two identical buildings (which look a lot like the CN Tower). CN Tower is the main landmark of Toronto, whose baseball team happens to be the Blue Jays. It's located near the main sportsball stadium downtown. I vaguely understand how text-to-image works, and so it makes sense that the vector space for "blue j…
> Has anyone come across a solution where model can iterate (eg, with prompts like "move the bicycle to the left side of the photo")? It feels like we're close. I feel like we're close too, but for another reason. For although I love SD and these video examples are great... It's a flawed method: they never get lighting correctly and there are many incoherent things just about everywhere. Any 3D artist or photographer…
The question is whether the 99% of the audience would even care...
Earlier quoted context omitted.
I wouldn't bet either way. Back in the mid 90s to 2010 or so, graphical improvements were hailed as photorealistic only to be improved upon with each subsequent blockbuster game. I think we're in a similar phase with AI[0]: every new release in $category is better, gets hailed as super fantastic world changing, is improved upon in the subsequent Two Minute Papers video on $category, and the cycle repeats. [0] all of…
Your comment reminded me of this: https://www.reddit.com/r/gaming/comments/ktyr1/unreal_yes_th... Many more examples, of course.
Big quality improvement over Marathon 2 on a mid-90s Mac, which itself was a substantial boost over the Commodore 64 and NES I'd been playing on before that.