61–70 of 90 posts
There are larger foundation models for geospatial imagery available. Our pre-training method, Scale-MAE [0], has 323M parameters, makes encoders robust to changes in satellite imagery resolution, and is therefore trained on satellite imagery of all resolutions. Work out of SI Analytics [1] presents a 2.4B parameter transformers for satellite imagery.
“Model” is a dangerously overloaded word in 2023, possibly worse than Object in 1996. When I worked on ML inference I would tease the researchers with the question “what is a model?” in the hope they would say something that constrained it in any way, but no, a model can be anything at all and is whatever you want it to be. As such this press release is perfect nonsense, which is kind of appropriate for a Watson offs…
If you think of a model as a function approximator with an error that can only be characterized empirically, it's not a bad term at all to describe the class of algorithms ANNs belong to. It's a real shame that the term is being abused so badly, because it's really appropriate in a lot of these cases. Or rather, it would be if people used it mindfully.
It very literally is a model of the relationships within the training data.
Moreover, statisticians and other varieties of people doing data analysis have been using the term "model" to mean something along these lines for many decades already.
If anything, it's nice to see more people calling these things "models".
Earlier quoted context omitted.
So what's the better word?
What it actually is: https://huggingface.co/ibm-nasa-geospatial/Prithvi-100M “Prithvi is a first-of-its-kind temporal Vision transformer” That is enormously more specific and interesting, and the word model is absent.
Earlier quoted context omitted.
I question the assumption that fine-tuning should always be avoided. If a model is going to be used many times for a specific use case, it is far cheaper and uses far less energy to fine tune a small model once and run it on cheap low-power hardware than it is to continuously run a huge, do-everything model on expensive, high-power hardware. Enormous models are great for exploration and for general purpose applicatio…
We're talking about different things. You're talking about finetuning models to tasks known in advance. I'm talking about the ability to generalize to new tasks: https://arxiv.org/pdf/2206.07682 . Please don't argue against a straw-man.
As I understand it we're contrasting two opposite approaches to ML: fine tuning small models for specific applications versus training a single large model that can generalize to new tasks without preparing them ahead of time.
I'm arguing that in fine tuning is far more useful than people are currently giving it credit for, and that generalizing a single massive model to new tasks is overrated.
Can you clarify where you're seeing a straw man?
I wish the press release had a bit more detail about what this model actually does and whether it's actually useful for the suggested use cases. However, make no mistake: this is for the scientific community and will not help geospatial data to be commercialized. No one cares about your geospatial crop model or that you can identify energy infrastructure or that there's some activity around that copper mine. Well, at…
what happened with your company?
Earlier quoted context omitted.
We're talking about different things. You're talking about finetuning models to tasks known in advance. I'm talking about the ability to generalize to new tasks: https://arxiv.org/pdf/2206.07682 . Please don't argue against a straw-man.
I'm not attacking a straw man, it seems that I just don't understand the distinction you're drawing. As I understand it we're contrasting two opposite approaches to ML: fine tuning small models for specific applications versus training a single large model that can generalize to new tasks without preparing them ahead of time. I'm arguing that in fine tuning is far more useful than people are currently giving it credi…
This simply is not true! NASA and IBM need to do further literature review and rely less on press releases. There are larger foundation models for geospatial imagery available. Our pre-training method, Scale-MAE [0], has 323M parameters, makes encoders robust to changes in satellite imagery resolution, and is therefore trained on satellite imagery of all resolutions. Work out of SI Analytics [1] presents a 2.4B param…
Earlier quoted context omitted.
I'm not attacking a straw man, it seems that I just don't understand the distinction you're drawing. As I understand it we're contrasting two opposite approaches to ML: fine tuning small models for specific applications versus training a single large model that can generalize to new tasks without preparing them ahead of time. I'm arguing that in fine tuning is far more useful than people are currently giving it credi…
You're talking about tasks known in advance; I'm not.
I'm not arguing that there is no place for large, general models—they're great for exploration—just that a smaller foundation model shouldn't be dismissed offhand based solely on parameter count.
This simply is not true! NASA and IBM need to do further literature review and rely less on press releases. There are larger foundation models for geospatial imagery available. Our pre-training method, Scale-MAE [0], has 323M parameters, makes encoders robust to changes in satellite imagery resolution, and is therefore trained on satellite imagery of all resolutions. Work out of SI Analytics [1] presents a 2.4B param…
With the better resolutions that are being launched and current AI, there are many more feasible applications vs. when you started DL.
We've built this leveraging other foundation models, so all research is very much appreciated https://www.youtube.com/watch?v=2yz4DwPtdjE
Disclaimer: I'm a co-founder of Happyrobot. We're working on this as we speak.
Let's all meet at SmallSat https://smallsat.org/ conference next week and discuss in more depth!