Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Alias-Free GAN

nvlabs.github.io

Post reply on HN
Print view View original

51–60 of 83 posts

Re: Alias-Free GAN

#51

Post by taneq »

sillysaurusx wrote: post #49
godelski wrote: post #47

Earlier quoted context omitted.

> I do ML every day. > I go to the github. Maybe model download link is there. I see zero code Paper was released today. Chill. They said they will release the code in September (I'm guessing late September). The paper is also a pre-print. They're probably aiming for CVPR and don't want to get scooped. > Model first. Code second. Paper third. That's how you produce ML code and documentation but that is not how you re…

The fact that you had to say "chill" three times indicates that you're trying to convince yourself, not me. None of what you said is responsive to what I wrote. I think it's an opinion piece, but I'm not sure. The issue here is the scientific method. I've listed the things that are required, as I see it. And I've also listed the reasons why I haven't been able to verify it exists here, despite trying for two years. I…

The repetition here is a common rhetorical device, not necessarily an indication of self-doubt.

That said, I agree with your overall position on ML publications. So much of what we see is a tech demo protected by some kind of moat, either a private commercial dataset or insatiable processing requirements or missing code or a combination of the above. These aren’t science, they’re advertisements.

Re: Alias-Free GAN

#52

Post by godelski »

sillysaurusx wrote: post #49
godelski wrote: post #47

Earlier quoted context omitted.

> I do ML every day. > I go to the github. Maybe model download link is there. I see zero code Paper was released today. Chill. They said they will release the code in September (I'm guessing late September). The paper is also a pre-print. They're probably aiming for CVPR and don't want to get scooped. > Model first. Code second. Paper third. That's how you produce ML code and documentation but that is not how you re…

The fact that you had to say "chill" three times indicates that you're trying to convince yourself, not me. None of what you said is responsive to what I wrote. I think it's an opinion piece, but I'm not sure. The issue here is the scientific method. I've listed the things that are required, as I see it. And I've also listed the reasons why I haven't been able to verify it exists here, despite trying for two years. I…

I say chill because you're freaking out.

The scientific method is being followed here. Code is not needed for the scientific model to be followed. Even data. Literally every other field is able to advance without public code or data (in fact most areas of CS). There's absolutely no reason to believe that they won't release their code. They have a history of doing so. Models and checkpoints are not the bare-minimum for reproducibility. They describe their model enough in the paper. There's enough written in the paper (which is 30 pages) to reproduce the model. Will it be easy? No. But it can be done. And to be clear, I'm saying that the status quo of code being released is a godsend. This is not the norm in literally every other field/subfield. Code helps with reproducibility (and so should be encouraged) but is not required.

If you require someone else's code to reproduce results then you're not convincing me you're a good ML researcher nor programmer.

Re: Alias-Free GAN

#53

Post by clircle »

sillysaurusx wrote: post #21
ansk wrote: post #12

This group of researchers consistently demonstrates a degree of empirical rigor that is unmatched across any other ML lab in industry or academia - remarkable empirical results as always, reproducible experiments, open-source and well-engineered codebase, and valuable insights about low-level learning dynamics and high-level emergent artifacts. Applied ML wouldn't have such a bad rap if more researchers held themselv…

This isn't true. I do ML every day. You are mistaken. I click the website. I search "model". I see two results. Oh no, that means no download link to model. I go to the github. Maybe model download link is there. I see zero code : https://github.com/NVlabs/alias-free-gan Zero code. Zero model. You, and everyone like you, who are gushing with praise and hypnotized by pretty images and a nice-looking pdf, are doing dam…

I hardly understand this comment. Statisticians have been publishing and arguing about models for 100 years now. No one required code to verify authenticity of research. I suppose it is the sorry state of machine learning research that the methodology is so poor that a person cannot verify the research from the paper.

Re: Alias-Free GAN

#54

Post by z3ncyberpunk »

godelski wrote: post #52
sillysaurusx wrote: post #49

Earlier quoted context omitted.

The fact that you had to say "chill" three times indicates that you're trying to convince yourself, not me. None of what you said is responsive to what I wrote. I think it's an opinion piece, but I'm not sure. The issue here is the scientific method. I've listed the things that are required, as I see it. And I've also listed the reasons why I haven't been able to verify it exists here, despite trying for two years. I…

I say chill because you're freaking out. The scientific method is being followed here. Code is not needed for the scientific model to be followed. Even data. Literally every other field is able to advance without public code or data (in fact most areas of CS). There's absolutely no reason to believe that they won't release their code. They have a history of doing so. Models and checkpoints are not the bare-minimum fo…

No one is freaking out... you are projecting.

Re: Alias-Free GAN

#55

Post by [deleted] »

godelski wrote: post #52
sillysaurusx wrote: post #49

Earlier quoted context omitted.

The fact that you had to say "chill" three times indicates that you're trying to convince yourself, not me. None of what you said is responsive to what I wrote. I think it's an opinion piece, but I'm not sure. The issue here is the scientific method. I've listed the things that are required, as I see it. And I've also listed the reasons why I haven't been able to verify it exists here, despite trying for two years. I…

I say chill because you're freaking out. The scientific method is being followed here. Code is not needed for the scientific model to be followed. Even data. Literally every other field is able to advance without public code or data (in fact most areas of CS). There's absolutely no reason to believe that they won't release their code. They have a history of doing so. Models and checkpoints are not the bare-minimum fo…

[deleted]

Re: Alias-Free GAN

#56

Post by godelski »

varispeed wrote: post #41
ansk wrote: post #24

Earlier quoted context omitted.

You're clearly disillusioned with the general accessibility of ML research, but I don't think your cynicism is warranted here. Take a look at their prior works[1], and I think you'll agree they go above and beyond in making their work accessible and reproducible. There is no reason to doubt the open-source release of this work will be any different. As to why the release is delayed, I'd speculate it's because they pu…

When I was writing a paper I had to include all source code in a state to be published, otherwise it wouldn't be accepted. I guess today the bar is much lower.

What was released was a pre-print, not a publication. Every top ML conference requires releasing of code, but this is not the norm for CS research nor for research in general.

The bar isn't low, this is just a pre-print.

Re: Alias-Free GAN

#57

Post by godelski »

sillysaurusx wrote: post #25
ansk wrote: post #24

Earlier quoted context omitted.

You're clearly disillusioned with the general accessibility of ML research, but I don't think your cynicism is warranted here. Take a look at their prior works[1], and I think you'll agree they go above and beyond in making their work accessible and reproducible. There is no reason to doubt the open-source release of this work will be any different. As to why the release is delayed, I'd speculate it's because they pu…

There is no reason to doubt the open-source release of this work will be any different. Then this is not a scientific contribution yet. We must wait and see. The most important tenet of science, is to doubt. I didn’t even read the name on the paper before I wrote my comment. Yes, I know this group. They’re why I got into ML, along with the group from OpenAI who published GPT-2. Because A+ science. Their claims here a…

> When I got into ML two years ago, I thought that knowledge would come from reading scientific papers. I was wrong.

I'm in the middle of a PhD and this is always an issue. It takes awhile to learn how to read papers and to gather enough background knowledge that you can read between the lines (publications are limited, you can't put everything in a paper. This is why having code is so great, it accelerates the process). You're two years into your journey, this is often when things _start_ turning the other direction. There's a reason PhDs take so long, and that's with experts (hopefully) helping you learn how to read papers, telling you which papers to read (which is a challenge in of itself), having the ability to spend full time on learning, and learning how to build background knowledge on a subject while learning the state of the art. There's a reason ML pays the big bucks. It takes a long time to learn/gather expertise, it is fucking difficult, and it has direct applications that can lead to useful products today (a big component of why you get paid big bucks). It is also easy to lose track of your progress. I remember the first research paper I read was complete gibberish to me. I'm 3 years into my PhD and now I can understand papers in my niche. But for a long time a lot of stuff didn't click. This is normal. It takes time to learn and 2 years isn't that much (especially when you have a full time job). Making contributions in your first year of a PhD is atypical, even in your second year. It only happens at top universities where people have a lot of help and resources.

Research it hard. It takes years to become an expert and learn how to read papers. Don't give up, but calm down and recognize that given more time things will make more sense.

Re: Alias-Free GAN

#58

Post by godelski »

phreeza wrote: post #33
sillysaurusx wrote: post #27

Earlier quoted context omitted.

I wasn't sure whether to post my edit as a separate comment or not, but I significantly expanded my comment just now, that helps explain my position. I'd be very interested in your thoughts on that position, because if it's mistaken, I shouldn't be saying it. It represents whatever small contribution I can make to fellow new ML researchers, which is roughly: "watch out. " In short, for two years, I kept trying to imp…

I get your frustrations with this state of affairs, but for the reasons I mentioned above, I don't think providing the model and code is a panacea here. Maybe the last few years have also set an unrealistic expectation for the pace of progress. In my (former) field of theoretical neuroscience, if a paper was not reproducible, this knowledge kind of slowly diffused through the community, mostly through informal conver…

To be honest I don't think the StyleGAN papers are benchmark chasing. If you read StyleGAN[0], StyleGAN2[1], StyleGAN2-ADA[2], and this paper there is a clear story. They call out the mistakes in the previous papers and resolve them. The papers themselves even admit to where they fall short. But it is research. Problems don't get solved all at once. But if you pay attention to these 4 papers it is very clear Kerras has a very well defined research focus and direction. He's showing his progress over time, sharing it with the community, and learning from the community as well. This is how research should happen.

[0] https://arxiv.org/abs/1812.04948

[1] https://arxiv.org/abs/1912.04958

[2] https://arxiv.org/abs/2006.06676

Re: Alias-Free GAN

#59

Post by pabs3 »

sillysaurusx wrote: post #50
pabs3 wrote: post #48

Earlier quoted context omitted.

Would you not want the data and code used to train the model, rather than the trained model itself? Edit: the Debian Deep Learning Team's Machine Learning Policy explains why. https://salsa.debian.org/deeplearning-team/ml-policy

For various reasons, I left data out of the requirements because most interesting research uses data unavailable to the community. CLIP is such an example, and it's A+ science: Model, Code, Paper. Having the model is enough to verify the paper's claims, and also to experiment with new approaches (since you can fine-tune the model). That said, I make this concession as a "meet you halfway" compromise between hard-line…

Yeah, for the goal of verifying claims that is good enough indeed.

I'm not sure it is good enough for a variety of other possibly useful use-cases though; eliminating bias in an existing model, correcting a flaw in the training code, creating a different model and proving it is better by training on the same data etc.

It would be nice if there were more public/libre data sets for ML stuff.

Re: Alias-Free GAN

#60

Post by sillysaurusx »

godelski wrote: post #52
sillysaurusx wrote: post #49

Earlier quoted context omitted.

The fact that you had to say "chill" three times indicates that you're trying to convince yourself, not me. None of what you said is responsive to what I wrote. I think it's an opinion piece, but I'm not sure. The issue here is the scientific method. I've listed the things that are required, as I see it. And I've also listed the reasons why I haven't been able to verify it exists here, despite trying for two years. I…

I say chill because you're freaking out. The scientific method is being followed here. Code is not needed for the scientific model to be followed. Even data. Literally every other field is able to advance without public code or data (in fact most areas of CS). There's absolutely no reason to believe that they won't release their code. They have a history of doing so. Models and checkpoints are not the bare-minimum fo…

I'd be no scientist if I didn't update my priors.

I retract my claims. You're right. Thanks for calling me out.

I will say that it's... a gargantuan effort to do the things that you're proposing. But as someone who did them you're right, you can. (BigGAN-Deep took a year to track down the bug https://github.com/google/compare_gan/issues/54)

BigGAN-Deep is a decent example of the thing I was really worried about: replication. I thought it'd be really easy to "just implement the paper." But no one had. Mooch did, but not at the same scale as the DeepMind release.

Maybe you're right about me, too. You're convincing me that I'm not a very good ML programmer. It's probably best to bow out on whatever high notes I've achieved.

Karras' work is fantastic. I don't know why this preview of things to come was where I chose to do this. Thank you, nVidia group, for working so hard.

Post reply on HN