61–68 of 68 posts
Every time I have to clean text I wonder why I haven’t just trained a byte level denoising autoencoder to handle it for me.
Earlier quoted context omitted.
Author here :), I do think it’s a good direction to look into! That said, aside from it being a bit too much to do at once, you’d also have to be careful about how you distributed your FLOP budget across the hierarchy. With two levels, you can make one level (bytes/local encoder) FLOP efficient and the other (patches/global encoder) FLOP intensive. You’d also need to find a way to group patches into larger units. But…
In a way I'm kinda sad that if tokenizers will go the way of the dinosaurs as asking someone to give me a Unicode character from the private use area was one of the last ways you could actually distinguish a co-operative human from an LLM online They simply don't have those characters tokenized, so they can't output them. (But this is technically moot if the LLM has a python interpreter handy)
Earlier quoted context omitted.
I would make the argument that these AI safety initiatives yield messaging that muddles and confuses the public on the simple fact that they should not, under any circumstances, use a video or image as proof or assume its veracity. When I tell someone this it is common for them to come back with something like "aren't they working on things to detect if a video is fake?" I think this idea, that video content can stil…
"they should not, under any circumstances, use a video or image as proof or assume its veracity" This is just silly. Courts never assume the validity of evidence. It is actually assumed to be invalid unless it can be proved to have not been tampered with. Photos have been able to be edited for over 100 years but they are still used as evidence. The person who took the photo will sign an affidavit and or testify in co…
I was fortunate to get to a talk by Ross Taylor ex-Meta recently at the AI Engineer London meetup He's recorded the full talk here now: https://www.youtube.com/watch?v=S5l5OvJ01ws I had missed how much Meta have been doing on reasoning, ToM etc.
Earlier quoted context omitted.
Except giving out the tools makes easier for competitors like TikTok to do the same, drawing revenue away from meta. So that’s not it. Naively so.
Couldn't the same argument be made for all kinds of things companies have made open? Some examples: • Tesla gave away its EV patents. • Pixar and DreamWorks have both open-sourced some of their tools, including tools used to make some of their best works. For example DreamWorks' MoonRay renderer has been used on everything they have done since "How to Train Your Dragon: The Hidden World", including "Puss in Boots: Th…
A better answer is meta releases them for some combination of they see it benefitting the business and/or a desire to provide broad benefits to everyone. They certainly expend tremendous resources to create these models. No other company has provided this much value to such a large base of users in this space.
When I wonder about the business behind Meta doing this, I see they have $70B in cash, so giving a bunch of AI experts hundreds of millions is pocket change.
everyone that has responded so far has it wrong (naively so). FB sells ad space on several apps. those apps needs people on them in order for the ad space to be worth anything. people, in turn, need content to attract them to the apps. so it's simple: enable people/companies/whomever to generate tons of content for cheap and consequently share it on the apps. that's it.
Crazy stuff. Everyone’s covering how exciting all these are (especially LCM and the non-tokenizing-tokenizer), but I have to ask in case anyone’s been paying attention: why are they using the term “advanced machine intelligence”? My initial thought is that they want to please/distract the doomers, but I’m prolly just self-centered!
There’s honestly so much interesting stuff here, esp. the llm-related things - large concept models (operating on and predicting concepts, not tokens), dynamic byte latent transformers (byte-level alternative to standard tokenization), sparse memory layers (successfully scaling key-value memory layers without an increase in computational requirements). Here they are presented as separate things, each of which apparen…
all right, yeah, it's not "open source", but hey, it is open to use and they're publishing their research openly as well.