Outrageous that Science is platforming Ioannidis without a mention of his axe-grinding dismissal of COVID risks and outright attempts at intimidation of opponents. He's already burnt his reputation to slag.
AI's top startups are barely publishing their research
321–330 of 341 posts
Re: AI's top startups are barely publishing their research
#322My worry is learning and innovation. When companies sold physical devices, other companies could learn from them and improve them. We invented patents to provide economic support for innovation.
But, with services, nothing is exposed. Every innovation can be a trade secret. That makes it harder to learn and harder for cross pollination of ideas.
It probably won’t be good for employees either. Another company won’t bid for your skills as high, since you will take a longer time to train to work on their system.
Re: AI's top startups are barely publishing their research
#323My worry is learning and innovation. When companies sold physical devices, other companies could learn from them and improve them. We invented patents to provide economic support for innovation. But, with services, nothing is exposed. Every innovation can be a trade secret. That makes it harder to learn and harder for cross pollination of ideas. It probably won’t be good for employees either. Another company won’t bi…
Thats probably by design tbh. Every company is working on the next big thing, employees sign NDAs and non competes, because companies don't want other people to deliver on their "novel" idea first. And of course if employees can't transfer their skills elsewhere, they're stuck there.
Re: AI's top startups are barely publishing their research
#324Perhaps I'm imagining it but the entire industry was build on published research, this "AI wave" is at odds with that and seems to be driven by greed (although they'll claim some arms race or something to help themselves sleep at night). There should be a new ESG (Environmental, Social, and Governance) policy being pushed recognizing the important role this plays. Although ESG and all norms have been set aside in thi…
[dead]
Unfortunately. We live in a increasingly complex world and the truth gets muddled.
In the past if your local store does something unethical, it would be easy to determine this, discuss and vote with your wallet.
With increasingly large and complicated businesses this is no longer easy to determine and communicate.
Obviously this can be abused with hypocritical things like virtue signalling, but the primary purpose is to encourage businesses to do better.
There are many stakeholders.
Re: AI's top startups are barely publishing their research
#325It makes sense when there's at least two posts here on hacker news about model routers which just reimplement Sakana Fugu from its ICLR papers, as a for profit business without any revenue sharing with Sakana nor making them open source. Of course Sakana would want to keep their research private to prevent that from happening. Open router fusion is yet another reimplementation of Sakana Fugu.
Re: AI's top startups are barely publishing their research
#326Earlier quoted context omitted.
> Companies can’t be expected to publish their confidential and proprietary information about their feature development, Of course they can. Simply eliminate trade secret protections and NDAs, which have literally zero purpose for society and in fact undermine patents' incentive to publish. Give a one year grace period to apply for a patent. Patent protection should be designed to be only as long as needed to try to…
For as long as the information is confidential and proprietary , we live in a world where my advice applies. Additionally, the reason ML is rapidly developing and that companies are spending billions to do so is that they are in competition with one another. If trade secrets were impossible, they would have no economic motive. Any advantage gained, at a cost of millions or billions, would not improve their competitiv…
That's why I said you tune patents to accomplish your economic goal (we could of course also fund academic research more).
e.g. with a 3 year patent period, something like GPT-4 would lose its protection now, which would be economically meaningless since SOTA models or even open models are far more capable and/or efficient. This could be tuned to the industry to reflect investment ROI. It gives some first mover advantage while allowing technology to still proliferate within a reasonable time frame.
Re: AI's top startups are barely publishing their research
#327Earlier quoted context omitted.
Google published a lot. I still wonder this original transformer paper, and the attention, etc. Why would they allow it to let go in the open? Perhaps because was intended for translation first before someone decided to loop it over itself? Or was so obscure to fellow researchers what do they actually publish?
Kind of makes you wonder if there are other papers from Google that are potentially also profoundly groundbreaking but overlooked
Groundbreaking includes the “explain to the world why it matters” part
Re: AI's top startups are barely publishing their research
#328Academia is failing to adapt. It used to be that their journals were the only place to learn about technological breakthroughs. Now, I quickly skim the paper and go straight to the source code or data. A publication might be one of the least effective ways to gain scientific impact, second to which is word of mouth.
Also, it's too hard to publish and academics are too hostile. These people are government funded and reject work for things like mentioning Palestine. I've had multiple papers automatically rejected even for pre-publication. Even when I have jumped through all these hoops, it has no more credibility than a blog post to the average person. In fact, most people cannot even read my published work. They look at it for a few seconds, squint, and then say, "Cool."
All of this and the vast majority of publications are not even replicable. I've made do with uploading code or instructions on how to replicate work through git and then make a blog post about it.
I still create papers but they're for myself. When I complete some dense scientific project, a paper is a great format to onboard me onto it a few months later. I'm most concerned about indexing the work, but I find that online archiving tools are the best bet since journals are behind paywalls and are subject to retractions.
Re: AI's top startups are barely publishing their research
#329Earlier quoted context omitted.
Everyone says this in the abstract but to concrete examples they shug and say, of course that approach doesn't work, they did X, Y, Z wrong, they should have given it more effort, it could have worked if done properly / this can obviously never work, everyone knew already, it's nothing new etc.
Isn't that exactly the point? If in my conception that concrete example eliminates that single branch, then I can try the others, and only the others. Or confirm the negative. Which is also valid work. In general, your point stands for armchair researcher.
No, good researchers also dismiss such things, because most research out there is crap. If you reward negative results, you'll have a flood of them, and the reason that they are negative will mostly look like it's incompetence (and mostly will be, but even the small not-incompetent will be hard to distinguish).
It's asymmetric. If you make something work, beat a benchmark, invent a drug that works etc, the exact way you arrived there has some leeway. In the end, the thing worked out so it's worth knowing about.
If your project failed, there can be a million reasons for that, and it's not necessarily that the initial hypothesis or initial idea has been refuted. E.g. in ML, your model didn't learn the task. Okay, there could be a million knobs (hyperparameters) that you set up wrong, or you implemented it wrong, or you should have just added learning rate warmup, or this or that, a million things possible. People fail all the time at projects that others then manage to do later on. Science is not like simply asking the universe some clear question and getting a clear answer. It's a very messy process and even professionals are not super great at it, or at least they simply cannot afford to put so much effort in each single project to make it absolutely airtight such that the failure to make it work can be a legit refutation of the main idea.
Re: AI's top startups are barely publishing their research
#330Earlier quoted context omitted.
> The original statement was: (1) The branch to the left from A is a dead end. This is where you get things wrong at a very basic and fundamental level. Just because you failed to explore branch A, that does not mean it is a dead end. It just means you came up empty. That is why science is based on observations and theories: it is based on building up on ideas and what works and can be proven. Otherwise you will left…
You are assuming incompetence on the scientist saying that the branch from the left of A is a dead end. While nobody is perfect, there are numerous perfectly valid scientific negative results. You know, there exist things like impossibility proofs in mathematics and computer science. There are equivalents in other sciences (e.g. if X was true, that would lead to Y that is easily observable and clearly not observed).…
There are different types of research. Negative AI results are not like Michelson-Morley experiment in physics that can positively establish the lack of something. It's more of a craft, the "make it work" style of thing. Give the same task or goal to two different teams, you will see massively different outcomes. Simply failing to make it work is not conclusive.
To make such a negative result acceptable in AI, you'd have to have some clear reason why you think that your particular setup should produce the result you want, that exact configuration and architecture, dataset etc. There are countless projects in AI that fail. And it's not clear at all that it refutes any abstract hypothesis. It's a get-your-hands-dirty field. It can make or break a project whether someone has that tacit knowledge, that black magic experience to know how to properly do the project.
People can generate extremely many ideas. You'd need to convince me that your idea (among a million others that people are trying each day) is so significant that its failure is in itself interesting. If you were to review for AI conferences, you'd see the flood of papers that claim to achieve 0.5% or 1% improvement on some benchmark. Now imagine that they didn't even have that to show for it. It got worse by 2% after trying their random idea. Who cares then? Even the +1% with a random idea is quite annoying to accept. But if their random idea really made something work much better, I will at least have some reason to want to see what may be going on there, there can be some signal. With negative results, it's very uninteresting.