Since when is Carmack writing on Facebook??
A week-long programming retreat
111–120 of 158 posts
Re: A week-long programming retreat
#112Good writeup -- and one of the main reminders for me is this: People throw around words like "revolution" for the current deep-learning push. But it's worth remembering that the fundamental concepts of neural networks have been around for decades. The current explosion is due to breakthroughs in scalability and implementation through GPUs and the like, not any sort of fundamental algorithmic paradigm shift. This is s…
I don't think the revolution was about hardware improvement. I did some neural network research (and published a few papers) in the 1990s and switched to other research disciplines afterwards. So, I'm not really familiar with the recent developments. But to my knowledge, there was indeed a revolution in neural network research. It was about how to train a DEEP neural network. Traditionally, neural networks were train…
The things that have most improved training neural networks since you left were: 1. Smart (Xavier/He) initialization 2. ReLU activations 3. Batch normalization 4. Residual connections
The GPUs and dataset size were definitely very important though.
Re: A week-long programming retreat
#113Earlier quoted context omitted.
The best way to advance AI is probably to make the hardware faster, especially now that Moore's Law is in danger of going away. The people doing AI research generally seem to be fumbling around in the dark, but you can at least be certain that better hardware would make things easier.
I am not in the AI space at all, but I am under the impression that python is the most used language for it. If workload is becoming an issue, wouldn't the low hanging fruit be a more performance driven software language? With how fast things are evolving, developing something like an ASIC ($$$) for this might be outdated before it even hits release, no?
The heavy lifting done in neural networks is all offloaded to C++ code and the GPU. Very little of the computation time is spent in python.
ASICs will definitely be very helpful, and there's currently a bit of a rush to develop them. Google's TPUs might be one of the first efforts, but several other companies and startups are looking to have offerings too.
Re: A week-long programming retreat
#114>> I’m not a Unix geek. I get around ok, but I am most comfortable developing in Visual Studio on Windows. This is a lesson for me and probably many others. Don't get hung up on tools, ship!!!
Here here. Junior webdevs will often consider the use of vim and other command line tools (over GUI based tools or IDEs) a badge of honor.
Re: A week-long programming retreat
#115The recurse center (formerly known as Hackerschool) is offering one week mini retreats (their typical program is months long). I applied and didn't get in but after reading this, I may try again. https://www.recurse.com/blog/127-a-new-way-to-join-the-rc-co...
You should definitely reapply! I went and it was a blast.
Re: A week-long programming retreat
#116It's actually super refreshing learning that even programming masters like Carmack are just now learning NNs and watching Youtube Stanford classes like the rest of us. These are actual people, not gods :) Everybody poops!
Yeah .. but his first impulse was to write backprop from scratch. I saw the lectures, been dabbling with NN for years, and I never thought to do it. I always thought the Stanford people made you do it on assn 1 to pay your dues or something. I continue to think of Carmack as the Master hacker.
Re: A week-long programming retreat
#117Since when is Carmack writing on Facebook??
Since he's employed by them. (He works for Occulus, owned by FB).
That said, I chuckled at his note about being contrarian, as he writes a post on FB
Re: A week-long programming retreat
#118Good writeup -- and one of the main reminders for me is this: People throw around words like "revolution" for the current deep-learning push. But it's worth remembering that the fundamental concepts of neural networks have been around for decades. The current explosion is due to breakthroughs in scalability and implementation through GPUs and the like, not any sort of fundamental algorithmic paradigm shift. This is s…
In computer vision at least, deep learning has been a revolution. More than half of what I knew in the field became obsolete almost overnight (it took about a year or two I would say) and a lot of tasks received an immediate boost in term of performances. Yes, neural networks have been here for a while, gradually improving, but they were simply non-existent in many fields where they are now the favored solution. Ther…
It is indeed pretty amazing in CV. In a field nearly as old as CS itself, I'd say at least 75% of the existing techniques were made obsolete in a span of only a few years.
You could have started your PhD in 2006, made professor in 2012, and nearly everything you had learned would have been _completely_ different.
Re: A week-long programming retreat
#119I find it peculiar about people's continued good feelings towards Carmack. During the Zenimax/Oculus case: He claimed that he never wiped his hard drive: An independent court expert found that most of his hard drive was wiped after Carmack heard about the lawsuit. So Carmack lied in his affidavit. He claimed that no source code from Zenimax ever got transferred over to Oculus. Then later admitted that the emails he h…
I don't know anything, at all, about this case or much about him except the standard. But is there any point in bringing it up now? I mean, whether or not he did something illegal/unethical, does this now mean that literally every single time he writes something, someone feels the need to chime in with "but remember that bad thing he did once?". I'm not saying this isn't a legit conversation or that you're unethical…
I agree. The guy had a quarrel with someone over something. Maybe he was wrong. Maybe he was right. Maybe it's just not that simple. I don't see the importance of it one way or another.
But this seems to be a trend right now. Our tolerance for any sort of moral ambiguity or uncertainty has sunk to such low levels that we can no longer appreciate someone's work or insight without establishing that this person is flawless in every respect.
Re: A week-long programming retreat
#120I find it peculiar about people's continued good feelings towards Carmack. During the Zenimax/Oculus case: He claimed that he never wiped his hard drive: An independent court expert found that most of his hard drive was wiped after Carmack heard about the lawsuit. So Carmack lied in his affidavit. He claimed that no source code from Zenimax ever got transferred over to Oculus. Then later admitted that the emails he h…
He doesn't. He is in the wrong. I just don't care. I'm in the wrong every time I pack a bowl of weed. Still don't care. I understand why John doesn't own the fruit of his labor. I understand why John is willing to lie, cheat, and steal the people benefiting from the system that takes the fruit of his labor from him. If you don't like it, build a better system that prevents it.
> I'm in the wrong every time I pack a bowl of weed.
Just come to seattle.
Also, it's a false equivalency, stealing IP is far more sinister than smoking weed.