Gradients are not all you need

Interestingly, the models themselves also have compute cost associated with them, namely the cost of running them (inference). The smaller a model is, the less compute it needs for inference. Which is an interesting trade-off, because you can overtrain (rather than undertrain, as in the past) a model, i.e. use more tokens, fewer parameters, and ultimately more compute than optimal during training, to get lower inference cost.

https://www.harmdevries.com/post/model-size-vs-compute-overh...

This website has an interesting graph which visualizes this:

> For example, the compute overhead for 75% of the optimal model size is only 2.8%, whereas for half of the optimal model size, the overhead rises to 20%. As we move towards smaller models, we observe an asymptotic trend, and at 25% of the compute-optimal model size, the compute overhead increases rapidly to 188%.

So if you train your model you shouldn't just look at compute-optimal training but try to anticipate how much the model will probably be used for inference, to minimize total compute cost. Basically, the less inference you expect to do with your model, the closer the training should be to Chinchilla-optimality.

Getting cited is probably the largest individual incentive. To various degrees, authors also want to get their ideas and reputation “out there”. To that end, authors want their work to be (i) skimmed (the abstract at least); (ii) shared; (iii) read. A catchy title almost always helps (right?); there doesn’t seem to be significant downside to catchy titles.

So how do we get more desirable system behavior? It seems we have a collective action problem.

Is branding the concept in play? I don’t see it being a very good metaphorical fit in comparison to my suggestion below. Branding requires a common theme over multiple things (such as products or events).

I think we’re seeing something closer to title optimization in the service of marketing. But even marketing isn’t a great fit.

Upon reflecting on the Wikipedia definition of marketing, it strikes me that title optimization is only a tiny part of what “academic research marketing” could be. In the most generous sense, it could mean reaching out to media sources fairly early in the research/writing process to help craft a paper that will be interesting to the desired audience.

Wikipedia quotes:

> Marketing is the process of exploring, creating, and delivering value to meet the needs of a target market in terms of goods and services; potentially including selection of a target audience; selection of certain attributes or themes to emphasize in advertising; operation of advertising campaigns; attendance at trade shows and public events; design of products and packaging attractive to buyers; defining the terms of sale, such as price, discounts, warranty, and return policy; product placement in media or with people believed to influence the buying habits of others; agreements with retailers, wholesale distributors, or resellers; and attempts to create awareness of, loyalty to, and positive feelings about a brand.

> A brand is a name, term, design, symbol or any other feature that distinguishes one seller's good or service from those of other sellers. Brands are used in business, marketing, and advertising for recognition and, importantly, to create and store value as brand equity for the object identified, to the benefit of the brand's customers, its owners and shareholders.

> Personal branding is the conscious and intentional effort to create and influence public perception of an individual by positioning them as an authority in their industry, elevating their credibility, and differentiating themselves from the competition, to ultimately advance their career, widen their circle of influence, and have a larger impact.

I’ve provided the PDF URL to ChatGPT 4.0 and asked it to summarize the article with alternative titles, but for some reason it keeps getting the original title, authors, abstract, and body wrong. What prompt did you use?

There's plenty of interesting neural network designs out there but they're being overshadowed by transformers due to their recent success. I personally thing that the main reason transformers work so well is because they actually step away from the multi layer perceptron stuff and introduce some structure and in a way sparsity.