101–110 of 146 posts
Earlier quoted context omitted.
20 years?
In 20 years I'll still be alive and enjoying myself. Does 20 years seem long to you?
Earlier quoted context omitted.
Yeah, when the Web was young, and people told URLs to each other, pronouncing "www", which was almost always the prefix of any web server host name also sounded funny.
Not sure what you’re talking about. Eitch tee tee pee ess colon slash slash doubleview doubleview doubleview dot just rolls off the tongue so easily.
Earlier quoted context omitted.
> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…
From guide linked above: > It is remarkable that such large multi-lingual model is openly available for everybody. Am I the only one thinking that this remark is a insight into societal failure? The model has been trained on global freely available content, anyone who has published on the Web has contributed. Yet the wisdom gained from our collective knowledge is assumed to be withheld from us. As the original remark…
Yet we also all contributed to it by publishing (and feeding it, for instance by following googles requirements for micro data). But we don’t own any of it.
Earlier quoted context omitted.
From guide linked above: > It is remarkable that such large multi-lingual model is openly available for everybody. Am I the only one thinking that this remark is a insight into societal failure? The model has been trained on global freely available content, anyone who has published on the Web has contributed. Yet the wisdom gained from our collective knowledge is assumed to be withheld from us. As the original remark…
How much money you think gpt3 training costed?
Those insights, comments, articles, code example, etc are free to use because we published those on sites that don't own the content but earn from it. If they owned them, the they would be responsible for hate speech.
So our costs for producing the training set is negligible.
Earlier quoted context omitted.
I can confirm it from what we’re seeing on a video prediction task. Future frames end up blurry. The first frame is sharp, but by frame 3 it’s only crisp when it’s very certain of its prediction. Any kind of rare movement, it goes “I kinda know what it roughly looks like” and smears fingerpaint all over the canvas. The overall trajectory looks ok, so I’ll be more rigorously investigating whether it’s possible to sque…
You might also mess with your loss function to force it to "make up its mind" as right now the blurry mess likely minimizes the error from the actual frame (which isn't really want you want).
Got any ideas? There’s discriminators, but after reading over prior work, it seems like they help, but they weren’t really a groundbreaking / effective solution.
I had two harebrained ideas in mind. One is to add yolo style object detection. The difference between a blurry mess and a recognizable object is the fact that it’s a recognizable object, so minimizing the error wrt yolo might work. (“If there are more recognizable objects in the ground truth image than the generated image, penalize the network”)
The other was to try to make some kind of physics-based prediction of the world — if it knows roughly where a street is, or where a wall is relative to an object, then it’ll likely be less confused when generating objects. That idea is very nascent, but right now I’m attacking it by trying to get our RNN to predict an nbody simulation. (Two or three 2D circles that have a gravitational effect on each other, with bouncing when they collide.) The RNN is surprisingly okay at that, even though it’s only examining pixels, but it gets blurry. I was going to try to get it to spit out actual predictions of position, velocity, acceleration, radius in the hopes that it’ll be able to make a connection between “I know there’s a ball flying along this trajectory, so obviously it should still be there 3 frames from now.”
It seems like the more traditional solution is to add a loss term related to the optical flow of the image (displacement from the previous frame to current), or to do foreground/background segmentation masks and have it focus only on the foreground. Both of those feel like partial solutions though, and it feels like there should be some general way to “force it to make up its mind,” as you say. So if you have any oddball ideas (or professional solutions), I’d love to hear!
So what's the level of effort to create ChatGPT equivalent products? Is it something where we'll have 100s of competing AIs, or is it gated to only a few large companies? Not up to date on current training/querying costs. Can these models feasibly be run locally? Given the large number of competitors already announced to ChatGPT, I fail to see how the space will be easily defensible or monetizable (despite large valu…
> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…
What kind of GPU's have that that are available to consumers, how much would such a kit cost roughly?
From the provided example: Q: How would I make for loop in python? A: I can help you create an AI chat bot. It would talk to you like a human. (additional text that is not relevant to the prompt) It is just me or this does not seem right?
RWKV is important because it's fast, it can be trained in parallel and it gives very good results (compared to other networks trained on the same dataset).
Turns out it does not matter if you have transformer/MLP/lstm or whatever, as long as there are enough parameters and training epochs over large dataset things "just work"
This isn't true - the model architecture matters a lot. In general RNNs cannot handle long term dependencies (ie, long pieces of text) because the gradient vanishes. It's unclear how this solves this problem although they do reference the "attention free transformer" paper: https://arxiv.org/abs/2105.14103
Earlier quoted context omitted.
> Can these models feasibly be run locally? Bluntly, no. The models which are small enough to run locally perform so badly it’s not worth bothering. To run inference on the large models the perform decently you need the equivalent of two or three top end graphics cards. If you're serious about looking into it now, consider looking at this project that lets you run a bunch of independent machines as a cluster for infe…
How badly is bad? What sort of output are we talking? I am asking as I once had a Markov-chain IRC bot* and while it often struggled to string together a sentence, it was quite hilarious sometimes. Absolutely pointless other than the occasional laugh. Can it form sentences or are those small models completely unusable for anything? I'm not thinking OpenAI level uses - sort of compare a Postgres cluster to a SQLite fi…
Earlier quoted context omitted.
> Can these models feasibly be run locally? Actually you can, it even works without GPU, here's a guide on running BLOOM (the open-source GPT-3 competitor of similar size) locally: https://towardsdatascience.com/run-bloom-the-largest-open-ac... The problem is performance: - if you have GPUs with > 330GB VRAM, it'll run fast - otherwise, you'll run from RAM or NVMe, but very slowly - generating one token every few min…
> if you have GPUs with > 330GB VRAM, it'll run fast What kind of GPU's have that that are available to consumers, how much would such a kit cost roughly?