11–20 of 39 posts
This seems like a step in the dangerous direction.
We need to start ingesting raw scientific data through these models and see what it comes up with. What could these models identify by parsing through raw JWST or Hubble data? Or training against every published scientific paper? Is anyone doing this sort of thing already?
Meta's Galactica was an attempt to train an LLM predominantly on scientific papers, articles and so on. It failed pretty spectacularly but Galactica 2, if that's ever a things, might rectify that.
Problem with the assumption though is that transformers are good at identifying and replicating patterns given a set of rules (i.e. how proteins fold and misfold depending on the environment).
Hubble data isn’t so much “we know the rules but not their interactions” as much as “we don’t really know the full set of rules,” so that particular example probably wouldn’t be that fruitful.
In general, biology (where we understand the basic rules but not the complex ways they are combined) is the most fertile ground for transformer driven research.
This seems like a step in the dangerous direction.
It’ll be ok. The technology for “dangerous” AI doesn’t actually exist. The near term risks we face from AI are constrained to the realms of spam and privacy. World ending super-bots are science fiction.
This seems like a step in the dangerous direction.
It’ll be ok. The technology for “dangerous” AI doesn’t actually exist. The near term risks we face from AI are constrained to the realms of spam and privacy. World ending super-bots are science fiction.
Nobody's worried about the tech that exists.
> The near term risks we face from AI are constrained to the realms of spam and privacy.
Define "near term".
> World ending super-bots are science fiction.
Science fiction has become science fact before. Where's the knockdown argument that won't happen in this case?
Earlier quoted context omitted.
It’ll be ok. The technology for “dangerous” AI doesn’t actually exist. The near term risks we face from AI are constrained to the realms of spam and privacy. World ending super-bots are science fiction.
Before superintelligence scifi stuff we'll probably get some sort of superworm. Some rogue autonomous agent network that is improving itself via some framework like SKILL[1] going around 0-day'ing systems left and right and wreaking havoc. [1] https://arxiv.org/abs/2010.11944
Yo dawg, we heard you like transformers so we put transformers on your transformers so you can train while you train. The spider web graph shows metatransformers performing worse to their counterparts in almost all fields. Is there a reason I should not believe that an expert model will always outperform a general purpose one, even if it's a metatransformer?
transformer(transformer(transformer( ... x ... ))) = ?
Yo dawg, we heard you like transformers so we put transformers on your transformers so you can train while you train. The spider web graph shows metatransformers performing worse to their counterparts in almost all fields. Is there a reason I should not believe that an expert model will always outperform a general purpose one, even if it's a metatransformer?
It's an interesting question as it begs questions of conceptual "boundaries."
The sense-plan-do process requires a search and filter process for task switching, assuming an agent can do more than one thing.
So assuming you have a robotic/autonomous agent that is a collection of systems (locomotion, dexterous gripper, visual perception, etc...), if each system could be represented as an "expert module", say for example the dexterous manipulator, then so long as a discriminator can appropriately switch states using the sensor/system inputs, then it's conceptually possible that there is a canonical "expert module" that everyone uses and therefore "general purpose" would apply to the agent as a whole while expert model would apply to the dexterous manipulator.
You can walk that reasoning up the abstraction layers then to conclude that (as usual with these turtle stacks) the distinctions come as each sub system/module specializes more granularly for the environment they operate in.
I think that it's probably forever and always true that any system designed to explore/exploit a bounded environment with comprehensive observations, will always outperform a system that is required to adapt it's sense-plan-do components to the bounded environment without similar observations.
A system would either have to generate different observations than the native agent, or change the boundaries of the environment in a way that is unavailable to the native agent in order to outperform it.
Yeah, that's where I thought it would go shortly after I tried GPT-4 from openAI. We're clearly at the transformer limits imho (comparing the effectiveness between 3.5 and 4, and the number of parameter in each model is why I think we reached a soft cap). So since it'll be hard to go deeper, going broader by interlacing different model types might be a way to pierce through.
GPT-4 did not scale up substantially in depth, going from 175 b to 220 b per transformer.
Yeah, that's where I thought it would go shortly after I tried GPT-4 from openAI. We're clearly at the transformer limits imho (comparing the effectiveness between 3.5 and 4, and the number of parameter in each model is why I think we reached a soft cap). So since it'll be hard to go deeper, going broader by interlacing different model types might be a way to pierce through.
> We're clearly at the transformer limits imho GPT-4 did not scale up substantially in depth, going from 175 b to 220 b per transformer.
Or is the idea to keep the network the same size and trade off some of its nodes for image, video, etc. data?
If so has anyone shown that doing so results in better overall performance?
My lay-observation is that GPT-4 seems to be on the border of usability for most applications so if nothing is gained by simply changing the input data type as opposed to expanding the model then it feels like it won't be of much use yet.
Also apologies if I'm not making sense, I'm almost certainly not using to correct technical terms to articulate what I'm thinking.
Yo dawg, we heard you like transformers so we put transformers on your transformers so you can train while you train. The spider web graph shows metatransformers performing worse to their counterparts in almost all fields. Is there a reason I should not believe that an expert model will always outperform a general purpose one, even if it's a metatransformer?
If a general purpose model beats the specialized one, you could almost certainly distill the general purpose one into a better specialized one.