I can't find much information about Shan Shan Wong, the new co-CEO. Not even a photo of this person on the internet.
Anyone else have information about them?
Emad Mostaque resigned as CEO of Stability AI
221–230 of 429 posts
Re: Emad Mostaque resigned as CEO of Stability AI
#222Earlier quoted context omitted.
The released image generation models run on consumer GPUs. Even the big LLMs will run on a $3500 Mac with reasonable performance, and the CPU of a dirt cheap machine if you don't care about it being slow, which is sometimes important and sometimes isn't. Also, things like this are in the works: https://news.ycombinator.com/item?id=39794864 Which will put the system RAM of the new 24-channel PC servers in range of the…
The `big' AI models are trillion parameter models. The medium sized models like GPT3 and Grok are 185b and 314b respectively. There is no way for _anyone_ to run these on a sub $50k machine in 2024, and even if you can the token generation speed on CPU is under 0.1 tokens per second.
I will make a 2 trillion parameter model just so your comment becomes outdated and wrong.
Re: Emad Mostaque resigned as CEO of Stability AI
#223Earlier quoted context omitted.
Midjourney is the most popular discord channel by far with 19.5M+ members, $200M in revenue in 2023 with 0 external investments and only 40 employees. The problem has nothing to do with commercializing image gen AI and all to do with Emad/Stability having seemingly 0 sensible business plans. Seriously this seemed to be the plan: Step 1: Release SD for free Step 2: ??? Step 3: Profit The vast majority of users couldn'…
Heres a Stable Diffusion buisness idea: sign up all the celebrities and artists who are cool with AI, and provide end users / fans with an AI image generation interface, trained on their exclusive likenesses / artwork (loras). You know, the old tried and true licensed merchandise model. Everybody gets paid.
I think the following isn't said often enough: there must be a reason why there are extremely few celebrities and artists who are cool with AI, and it cannot be something abstract and bureaucratic as copyright concerns although those are problematic.
It's just not there yet. GenAI outputs aren't something audiences wants to hang on a wall. It's something that evoke sense of distress. Otherwise everyone's tracing them at least.
Re: Emad Mostaque resigned as CEO of Stability AI
#224He's leaving to work on decentralized AI? That's exactly what Stability AI was doing before it became clear the economics no longer work out in practice, and starting a new company wouldn't change that. (Emad is an advisory board member to a decentralized GPU company, though: https://home.otoy.com/stabilityai/ ) Obviously this is the polite way to send him off given the latest news about his leadership, but this rati…
Think seti at home. Instead of wasting all the compute on bitcoin we pretrain fully open models which can run on people's hardware. A 120b ternary model is the most interesting thing in the world. No one can train one now because you need a billion dollar super computer.
Reminded me of Petals offering distributed ML inference anyone tried this?
Re: Emad Mostaque resigned as CEO of Stability AI
#225Earlier quoted context omitted.
Not twitter but here he is talking about that: https://youtu.be/BdZo4JUBSQk
You know it never ceases to amaze me how even the most respected fall prey to this money laundering scheme. If people even spent some time to read about Tether they would not touch this stuff. It's blood money.
Exactly how is cryptocurrency merely a "money laundering scheme"?
Re: Emad Mostaque resigned as CEO of Stability AI
#226Earlier quoted context omitted.
Think seti at home. Instead of wasting all the compute on bitcoin we pretrain fully open models which can run on people's hardware. A 120b ternary model is the most interesting thing in the world. No one can train one now because you need a billion dollar super computer.
SETI made sense because there is a lot of data where you download chunk and do expensive computation and return thin result. Model training is unlike that. It's large state that is constantly updated and updates require full, up to date state. This means you cannot distribute it efficiently over slow network with many smaller workers. That's why NVIDIA is providing scalable clusters with specialized connectivity so t…
That is the challenging part indeed.
But if we think of mixture of experts models outperforming "monolithic" models, why not? Maybe instead of 8 you can do 1000 and that is easy to paralellize. It sounds worth exploring to me.
Re: Emad Mostaque resigned as CEO of Stability AI
#227Earlier quoted context omitted.
Heres a Stable Diffusion buisness idea: sign up all the celebrities and artists who are cool with AI, and provide end users / fans with an AI image generation interface, trained on their exclusive likenesses / artwork (loras). You know, the old tried and true licensed merchandise model. Everybody gets paid.
Why would those celebs pay Stability any significant money for this, given they can get it for a one off payment of at most a few hundred dollars salary/opportunity cost by paying an intern to gather the images and feed it into the existing free tools for training a LoRA?
I think in this case the celebs are getting paid for using their likeness.
Re: Emad Mostaque resigned as CEO of Stability AI
#228What's with these weirdly convoluted re-spellings of Arabic/Muslim names to make them look not Arabic/Muslim?
Re: Emad Mostaque resigned as CEO of Stability AI
#229Earlier quoted context omitted.
As a counter-point, with no VCs there's more equity left for employees.
As a counter-counter-point that gets rarely discussed on HN, VCs aren't taking as much of the pie as people think. In a 2-founder, 4-engineer company, it wouldn't be unusual to have equity be roughly: 20% investors 70% founders 2-3% employees (1% emp1, 1% emp2, 0.5% emp3, 0.25% emp4) 7% for future employees before next funding round
You can't run a sustainable business if you take VC money. VCs need an exit.
Re: Emad Mostaque resigned as CEO of Stability AI
#230Earlier quoted context omitted.
SETI made sense because there is a lot of data where you download chunk and do expensive computation and return thin result. Model training is unlike that. It's large state that is constantly updated and updates require full, up to date state. This means you cannot distribute it efficiently over slow network with many smaller workers. That's why NVIDIA is providing scalable clusters with specialized connectivity so t…
That's wrong. What you described is data parallelism and it would indeed be very tricky to e.g. sync gradients across machines. But this is not the only method of training neural nets (transformers or any other kind) in parallel. If we'd like to train, say, a human brain complexity level model with 10^15 parameters, we'd need a model parallelism approach anyways. It introduces a bit of complexity since you need to ma…
Lack of data parallelism is implied by computation that is performed.
You gradient descend on your state.
Each step needs to work on up to date state otherwise you're computing gradient descend from state that doesn't exist anymore and your computed gradient descent delta is nonsensical if applied to the most recent state (it was calculated on old one, direction that your computation calculated is now wrong).
You also can't calculate it without having access to the whole state. You have to do full forward and backward pass and mutate weights.
There aren't any ways of slicing and distributing that make sense in terms of efficiency.
The reason is that too much data at too high frequency needs to be mutated and then made readable.
That's also the reason why nvidia is focusing so much on hyper efficient interconnects - because that's the bottleneck.
Computation itself is way ahead of in/out data transfer. Data transfer is the main problem and going in the direction of architecture that dramatically reduces it by several orders of magnitude is just not the way to go.
If somebody solves this problem it'll mean they solved much more interesting problem – because it'll mean you can locally uptrain model and inject this knowledge into bigger one arbitrarily.