Live data from Hacker News

GAN Theft Auto [video]

youtube.com

11–20 of 99 posts

Re: GAN Theft Auto [video]

#11
post #9
post #7

Earlier quoted context omitted.

The GAN model is the game environment. You're playing a neural network. The novelty is no game engine, no rules, just learned how to represent the game and you can play it.

What he meant is you overfitted the network with video footage. There is no game, just seemingly clever stitching and playback of learned footage similar concept applied to animations and implemented in a state machine https://www.youtube.com/watch?v=KSTn3ePDt50 and optimized with nn https://www.youtube.com/watch?v=16CHDQK4W5k

The first link provided seems to need a very detailed human-provided cost function for specific development needs.

The second one is indeed interesting research and seems to be a combination of the prior learned motion mapping working in tandem with a generative model.

I suppose you could say that the automation of the dataset is considered as "augmentation"; but the difference here is that the dataset is just pixels and inputs rather than all that animation info and simulation data. Yes, a simulation is running; but the GAN only gets the pixels and the input.

There's a similarity there though; you're right. In either case; the explicit goal of the video you posted is to combat runtime constraints of generative models. I'm not certain it's a fair comparison.

The latter video and sentdex's result both seem to generalize to unique scenarios not present in the training set. This may mean they are creating an efficient representation of the underlying data in order to predict future samples more easily than simply overfitting.

The top level comment here is a shallow dismissal and Randomoneh could have answered these questions themselves before throwing out a smug comment like "I fail to see novelty here" when it's at the very least the first large-scale GAN successfully trained on GTA V.

Re: GAN Theft Auto [video]

#12

Great work! Hacker News still seems to have a deeply skeptical culture with regard to machine learning - not sure why. There's always someone saying it's "not novel" and it's "just doing x". Overfitting is a known issues in machine learning, people. If you still think all neural networks are doing is memorizing the dataset completely in the year 2021 - you might want to revisit the topic. It is one of the first conce…

One of the main problems with ML/NN is it often works like magic, aka the trick works as long as audience doesnt know the secret behind it. Its fascinating to gullible audience, mundane bordering on boring to practitioners.

My Tiger repelling rock^^^^^^leopard detection model works great on all animal pictures ... until you feed it a sofa https://web.archive.org/web/20150703094328/http://rocknrolln...

>able to generalize various game logic like collision/friction with vehicles and also learns aspects of rendering such as a proper reflection of the sun on the back of the car

id did none of that, what this model did is learn all the frames of video and their chronological order according to the input.

> impossible task of "splitting a car in two" to try and solve a head-on collision.

it played back both learned versions at once, like reporting confidence of round thing being 50% ball and 50% orange.

Re: GAN Theft Auto [video]

#13
post #9

Earlier quoted context omitted.

What he meant is you overfitted the network with video footage. There is no game, just seemingly clever stitching and playback of learned footage similar concept applied to animations and implemented in a state machine https://www.youtube.com/watch?v=KSTn3ePDt50 and optimized with nn https://www.youtube.com/watch?v=16CHDQK4W5k

The first link provided seems to need a very detailed human-provided cost function for specific development needs. The second one is indeed interesting research and seems to be a combination of the prior learned motion mapping working in tandem with a generative model. I suppose you could say that the automation of the dataset is considered as "augmentation"; but the difference here is that the dataset is just pixels…

The first link exposes the trick employed by your model.

>animation info and simulation data

but did your model learn any of that?

>explicit goal of the video you posted is to combat runtime constraints

The trick to motion mapping is feeding a lot of data with accompanying inputs to build an atlas you can reference during playback.

>first large-scale GAN successfully trained on GTA V

Its really cool. The problem I had is in the presentation. I immediately felt insincerity bordering on scamming the audience, because I assume someone working in this field would know how the sausage is made. From the YT clip: "the shadow and reflection works", "modeling of physics works". Do they? or did your model build an atlas of video frames it can play back according to the fed input? Im guessing weather/time of day was locked when recording training data - perfect shadow and constant sun position for a nice reflection. Searching for 1:1 matches of generated output in the training set would be interesting and pretty revealing.

Re: GAN Theft Auto [video]

#14
post #12

Great work! Hacker News still seems to have a deeply skeptical culture with regard to machine learning - not sure why. There's always someone saying it's "not novel" and it's "just doing x". Overfitting is a known issues in machine learning, people. If you still think all neural networks are doing is memorizing the dataset completely in the year 2021 - you might want to revisit the topic. It is one of the first conce…

One of the main problems with ML/NN is it often works like magic, aka the trick works as long as audience doesnt know the secret behind it. Its fascinating to gullible audience, mundane bordering on boring to practitioners. My Tiger repelling rock^^^^^^leopard detection model works great on all animal pictures ... until you feed it a sofa https://web.archive.org/web/20150703094328/http://rocknrolln... >able to genera…

In the end, everything is boiling down to matrix math, so you can always make the argument that no neural network is impressive if you want.

The model's size is ~173MB, depending on settings. That's not much space to have memorized every single possible combination of events, nor was our data enough to cover that either.

Re: GAN Theft Auto [video]

#15
post #9
post #7

Earlier quoted context omitted.

The GAN model is the game environment. You're playing a neural network. The novelty is no game engine, no rules, just learned how to represent the game and you can play it.

What he meant is you overfitted the network with video footage. There is no game, just seemingly clever stitching and playback of learned footage similar concept applied to animations and implemented in a state machine https://www.youtube.com/watch?v=KSTn3ePDt50 and optimized with nn https://www.youtube.com/watch?v=16CHDQK4W5k

We had ~100GB of data (and that was gzip compressed data). The final model is 173MB.

It's simply not large enough to have memorized every combo.

Re: GAN Theft Auto [video]

#16
post #13

Earlier quoted context omitted.

The first link provided seems to need a very detailed human-provided cost function for specific development needs. The second one is indeed interesting research and seems to be a combination of the prior learned motion mapping working in tandem with a generative model. I suppose you could say that the automation of the dataset is considered as "augmentation"; but the difference here is that the dataset is just pixels…

The first link exposes the trick employed by your model. >animation info and simulation data but did your model learn any of that? >explicit goal of the video you posted is to combat runtime constraints The trick to motion mapping is feeding a lot of data with accompanying inputs to build an atlas you can reference during playback. >first large-scale GAN successfully trained on GTA V Its really cool. The problem I ha…

> I immediately felt insincerity bordering on scamming the audience

MFW I read this. Jeez man. Model size is 173MB. It didn't just memorize every possible combo.

How the hell you went from our excitement about a fun project we shared on YT to accusing us of "scamming" the audience I really don't know. What a terribly rude and hateful attitude you have =/

Re: GAN Theft Auto [video]

#17

Great work! Hacker News still seems to have a deeply skeptical culture with regard to machine learning - not sure why. There's always someone saying it's "not novel" and it's "just doing x". Overfitting is a known issues in machine learning, people. If you still think all neural networks are doing is memorizing the dataset completely in the year 2021 - you might want to revisit the topic. It is one of the first conce…

Heh, yeah, tough crowd I guess. The full code, models, and videos are all released and people are still skeptical.

I feel like 95%+ of papers don't do anything besides tell you what happened and you're just supposed to believe them. Drives me nuts. Not sure why all the hate when you could just see for yourself. I'd welcome someone who can actually prove the model just "memorized" every combo possible and didn't do any generalization. I imagine the original GameGAN researchers from NVIDIA would be interested too.

Interesting @ guided diffusion, not aware of its existence til now. We've had our heads down for a while. Will look into it, thanks!

Re: GAN Theft Auto [video]

#18
post #14
post #12

Earlier quoted context omitted.

One of the main problems with ML/NN is it often works like magic, aka the trick works as long as audience doesnt know the secret behind it. Its fascinating to gullible audience, mundane bordering on boring to practitioners. My Tiger repelling rock^^^^^^leopard detection model works great on all animal pictures ... until you feed it a sofa https://web.archive.org/web/20150703094328/http://rocknrolln... >able to genera…

In the end, everything is boiling down to matrix math, so you can always make the argument that no neural network is impressive if you want. The model's size is ~173MB, depending on settings. That's not much space to have memorized every single possible combination of events, nor was our data enough to cover that either.

Your original self driving GTA5 videos are what helped me come to understand machine learning in the first place (along with some of Seth Bling's MarI/O, and a bit of Tom7's learn/play-fun magic). I used your tech to make an AI that played Donkey Kong Country in LSNES emulator shortly before Gym-Retro was released.

So, thanks a bunch, Sentdex. You are rad.

Re: GAN Theft Auto [video]

#20
post #16
post #13

Earlier quoted context omitted.

The first link exposes the trick employed by your model. >animation info and simulation data but did your model learn any of that? >explicit goal of the video you posted is to combat runtime constraints The trick to motion mapping is feeding a lot of data with accompanying inputs to build an atlas you can reference during playback. >first large-scale GAN successfully trained on GTA V Its really cool. The problem I ha…

> I immediately felt insincerity bordering on scamming the audience MFW I read this. Jeez man. Model size is 173MB. It didn't just memorize every possible combo. How the hell you went from our excitement about a fun project we shared on YT to accusing us of "scamming" the audience I really don't know. What a terribly rude and hateful attitude you have =/

Don't take it personal. Commenters on HN are famous for dismissing successful ideas (remember Dropbox?).

I have one question: you mentioned that the training data was 100GB. Was it the same resolution as what is output by the model (ignoring supersampling)?

Post reply on HN