Alias-Free GAN

I click the website. I search "model". I see two results. Oh no, that means no download link to model.

I go to the github. Maybe model download link is there. I see zero code: https://github.com/NVlabs/alias-free-gan

Zero code. Zero model.

You, and everyone like you, who are gushing with praise and hypnotized by pretty images and a nice-looking pdf, are doing damage by saying that this is correct and normal.

The thing that's useful to me, first and foremost, is a model. Code alone isn't useful.

Code, however, is the recipe to create the model. It might take 400 hours on a V100, and it might not actually result in the model being created, but it slightly helps me.

There is no code here.

Do you think that the pdf is helpful? Yeah, maybe. But I'm starting to suspect that the pdf is in fact a tech demo for nVidia, not a scientific contribution whose purpose is to be helpful to people like me.

Okay? Model first. Code second. Paper third.

Every time a tech demo like this comes out, I'd like you to check that those things exist, in that order. If it doesn't, it's not reproducible science. It's a tech demo.

I need to write something about this somewhere, because a large number of people seem to be caught in this spell. You're definitely not alone, and I'm sorry for sounding like I was singling you out. I just loaded up the comment section, saw your comment, thought "Oh, awesome!" clicked through, and went "Oh no..."

Because it’s crucially important that we protect the scientific method here.

The sole goal is to help people like me reproduce the model. If I can’t reproduce the model, I can’t verify the paper.

When I saw “commercial” and then “open source” in your comment, I said “oh no…”

My duty is to the scientific method, so I don’t care if it’s the most restrictive code on the planet as long as I can use it to reproduce the model in the paper.

Because at that point, I have a baseline for evaluating the paper’s claims.

The reason I assume the paper is false until proven otherwise, is because the paper often doesn’t have enough detail to reproduce the model shown in the videos on this tech demos. Meaning, if they’re the it to help me, the ML researcher, then they’re failing to tell me how to evaluate their claims rigorously.

(That said, it’s breaking my heart that I can’t agree with you here, because I want to so badly. I’ve felt similarly for years that scientific contributions need to be “free as in beer” commercially. But I recognize signs of zealotry when I see them, and I can’t let my personal views creep in, because people like me would stop listening if I was here e.g. arguing vehemently that nVidia needed to be delivering us something commercially viable along with a high quality codebase. The price for entry to the scientific method isn’t so high.)

Then this is not a scientific contribution yet.

We must wait and see.

The most important tenet of science, is to doubt. I didn’t even read the name on the paper before I wrote my comment. Yes, I know this group. They’re why I got into ML, along with the group from OpenAI who published GPT-2. Because A+ science.

Their claims here are likely wrong unless and until proven otherwise. This isn’t a hardline position. It’s been my experience across many codebases, during my two years of trying to reproduce many ideas.

I agree that that is an example of A+ science. But why do you think they’re punishing this now, today? Either because conference deadline or because nVidia pressure. Neither of those are related to helping me achieve the scientific method: reproducing the idea in the paper, to verify their claims.

All I can do is kind of try to reverse engineer some vague claims in a pdf, without those things.

--

Let me tell you a little bit about my job, because my time with my job may soon come to an end. I think that might clear up some confusion.

My job, as an ML researcher, is to learn techniques that may or may not be true, combine them in novel ways, and present results to others.

Knowledge, Contribution, Presentation, in that order.

The first step is to obtain knowledge. Let's set aside the question of why, because why is a question for me personally, which is unrelated.

Scientific knowledge comes when Knowledge, Contribution, and Presentation are all achieved in a rigorous way. The rigor allows people like me to verify that I have knowledge. Without this, I have mistaken knowledge, which is worse than useless. It's an illusion – I'm fooling myself.

When I got into ML two years ago, I thought that knowledge would come from reading scientific papers. I was wrong.

Most papers, are wrong. That's been my experience for the past two years. My experience may be wrong. Maybe others obtain rigorous scientific knowledge through the paper alone.

But researchers happen to obtain a dangerous thing: prestige. Unfortunately, prestige doesn't come from helping others obtain knowledge. It comes from that last step -- presentation.

The presentation on this thread is excellent. It's another Karras release. I agree; there's no reason to doubt they'll be just as rigorous with this release as they are with stylegan2.

But knowledge doesn't come from presentation. Only prestige.

Prestige makes a lot of new researchers try very hard to obtain the wrong things.

If all of these were small concerns, or curious quirks, they'd be a footnote in my field guide. But I submit that these things are front and center to the current state of affairs in 2021. Every time a release like this happens, it generates a lot of fanfare and we come together in celebration because ML Is Happening, Yay!

And then I try to obtain the Knowledge in the fanfare, and discover that either it's absent or mistaken. Because there are no tools for me to verify their claims -- and when I do, I often see that they don't work!

That's right. I kept finding out that these things being claimed, just aren't true. No matter how enticing the claim is, or whether it sounds like "Foobars are Aligned in the Convolution Digit," the claim, from where I was sitting, seemed to be wrong. It contained mistaken knowledge -- worse than useless.

Unfortunately, two years with no salary takes a toll. I could spend another few years doing this if I wanted to. But I wound up so disgusted with discovering that we're all just chasing prestige, not knowledge, that I'd rather ship production-grade software for the world's most boring commercial work, as long as the work seems useful and the team seems interesting. Because at least I'd be doing something useful.

I'd be very interested in your thoughts on that position, because if it's mistaken, I shouldn't be saying it. It represents whatever small contribution I can make to fellow new ML researchers, which is roughly: "watch out."

In short, for two years, I kept trying to implement stated claims -- to reproduce them in exactly the way you say here -- and they simply didn't work as stated.

It might sound confusing that the claims were "simply wrong" or "didn't work." But every time I tried, achieving anything remotely close to "success" was the exception, not the norm.

And I don't think it was because I failed to implement what they were saying in the paper. I agree that that's the most likely thing. But I was careful. It's very easy to make mistakes, and I tried to make none, as both someone with over a decade of experience (https://shawnpresser.blogspot.com/) and someone who cares deeply about the things I'm talking about here.

It takes hard work to reproduce the technique the way you're saying. I put all my heart and soul into trying to. And I kept getting dismayed, because people kept trying to convince me of things that either I couldn't verify (because verification is extremely hard, as you well know) or were simply wrong.

So if I sound entitled, I agree. When I got into this job, as an ML researcher, I thought I was entitled to the scientific method. Or anything vaguely resembling "careful, distilled, correct knowledge that I can build on."