I don't trust papers out of “Top Labs” anymore

CERN is usually running experiments to validate ideas, confirm new hypotheses, in general push the boundaries. They wouldn't just go "let's do the same thing as last time - we've already got the results, but let's crank up the power to get a noise-level improvement in numbers accuracy".

This is effectively the same wealth disparity topic that exhibits itself in many areas of research and in the economy as a whole. Access to resources is conflated with ability or potential. Left unchecked, this bias naturally concentrates power and creates a moat against newcomers.

You are right that this is not specific to computation, but I think you are begging the question by saying it is "jealousy" and that "the point is to move things forward."

It does not require jealousy to ask, "is this how we want to support research?" The question can just as easily arise from empathy, or even from worry about strategic risk. A winner-takes-all approach may be myopic---by slathering attention on short-term successes, we may be neglecting to invest in the development of competitors who would bring future breakthroughs outside the currently entrenched regime.

Experimental results become credible when they can be reproduced consistently.

Theoretical results become more credible when they are independently verified.

Jeff Dean has responded on the original Reddit thread. Clarified the experiment’s purpose, results, and pointed out that researchers could conduct the experiment at a much lower cost than the OP’s estimate.

Jeff Dean responded to OP:

(The paper mentioned by OP is https://arxiv.org/abs/2205.12755, and I am one of the two authors, along with Andrea Gesmundo, who did the bulk of the work).

The goal of the work was not to get a high quality cifar10 model. Rather, it was to explore a setting where one can dynamically introduce new tasks into a running system and successfully get a high quality model for the new task that reuses representations from the existing model and introduces new parameters somewhat sparingly, while avoiding many of the issues that often plague multi-task systems, such as catastrophic forgetting or negative transfer. The experiments in the paper show that one can introduce tasks dynamically with a stream of 69 distinct tasks from several separate visual task benchmark suites and end up with a multi-task system that can jointly produce high quality solutions for all of these tasks. The resulting model that is sparsely activated for any given task, and the system introduces fewer and fewer new parameters for new tasks the more tasks that the system has already encountered (see figure 2 in the paper). The multi-task system introduces just 1.4% new parameters for incremental tasks at the end of this stream of tasks, and each task activates on average 2.3% of the total parameters of the model. There is considerable sharing of representations across tasks and the evolutionary process helps figure out when that makes sense and when new trainable parameters should be introduced for a new task.

You can see a couple of videos of the dynamic introduction of tasks and how the system responds here:

https://www.youtube.com/watch?v=THyc5lUC_-w

https://www.youtube.com/watch?v=2scExBaHweY

I would also contend that the cost calculations by OP are off and mischaracterize things, given that the experiments were to train a multi-task model that jointly solves 69 tasks, not to train a model for cifar10. From Table 7, the compute used was a mix of TPUv3 cores and TPUv4 cores, so you can't just sum up the number of core hours, since they have different prices. Unless you think there's some particular urgency to train the cifar10+68-other-tasks model right now, this sort of research can very easily be done using preemptible instances, which are $0.97/TPUv4 chip/hour and $0.60/TPUv3 chip/hour (not the "you'd have to use on-demand pricing of $3.22/hour" cited by OP). With these assumptions, the public Cloud cost of the computation described in Table 7 in the paper is more like $13,960 (using the preemptible prices for 12861 TPUv4 chip hours and 2474.5 TPUv3 chip hours), or about $202 / task.

I think that having sparsely-activated models is important, and that being able to introduce new tasks dynamically into an existing system that can share representations (when appropriate) and avoid catastrophic forgetting is at least worth exploring. The system also has the nice property that new tasks can be automatically incorporated into the system without deciding how to do so (that's what the evolutionary search process does), which seems a useful property for a continual learning system. Others are of course free to disagree that any of this is interesting.

Edit: I should also point out that the code for the paper has been open-sourced at: https://github.com/google-research/google-research/tree/mast...

We will be releasing the checkpoint from the experiments described in the paper soon (just waiting on two people to flip approval bits, and process for this was started before the reddit post by OP).

---

source: https://old.reddit.com/r/MachineLearning/comments/uyratt/d_i...