When I wonder about the business behind Meta doing this, I see they have $70B in cash, so giving a bunch of AI experts hundreds of millions is pocket change.
Those AI experts are a played a critical role in Meta getting that $70B in the first place
Sharing new research, models, and datasets from Meta FAIR
41–50 of 68 posts
Re: Sharing new research, models, and datasets from Meta FAIR
#42I really hope Dynamic Byte Latent Transformers work out. Death to tokenizers! Interesting that it's a a hierarchical structure but only two levels of hierarchy. Stacking more levels seems like an obvious direction for further research.
Author here :), I do think it’s a good direction to look into! That said, aside from it being a bit too much to do at once, you’d also have to be careful about how you distributed your FLOP budget across the hierarchy. With two levels, you can make one level (bytes/local encoder) FLOP efficient and the other (patches/global encoder) FLOP intensive. You’d also need to find a way to group patches into larger units. But…
In a way I'm kinda sad that if tokenizers will go the way of the dinosaurs as asking someone to give me a Unicode character from the private use area was one of the last ways you could actually distinguish a co-operative human from an LLM online
They simply don't have those characters tokenized, so they can't output them. (But this is technically moot if the LLM has a python interpreter handy)
Re: Sharing new research, models, and datasets from Meta FAIR
#43Earlier quoted context omitted.
It lets those providing AI video generation services watermark all of their videos. So it isn't intended to by voluntary. You would be left with those services that don't comply with whatever the current Big Tech rules are, like people who used Grok/X.ai to generate images in support of Trump despite Grok/X.ai being inferior. https://arstechnica.com/information-technology/2024/08/musks...
Think this the wrong / older article - when I click the link, this is twitter's hosted Flux model making pictures of Kamala and Trump flying into the world trade center and Trump on a surfboard with busty cat girls. The X.ai one launched this week
X hosted a white-label Flux model for a while, and freely admitted so .
Re: Sharing new research, models, and datasets from Meta FAIR
#44Re: Sharing new research, models, and datasets from Meta FAIR
#45Meta's "Video Seal": Because nothing says "trustworthy" like a digital chastity belt. Imperceptible, they claim, yet robust enough to survive the gauntlet of internet mangling - sounds like the perfect tool to invisibly track content, not just watermark it.
Like all tools it can be used for good and evil. It could be installed directly in cameras to sign videos. And people with the power to turn it off could make AI fake videos that much more believable.
I would make the argument that these AI safety initiatives yield messaging that muddles and confuses the public on the simple fact that they should not, under any circumstances, use a video or image as proof or assume its veracity. When I tell someone this it is common for them to come back with something like "aren't they working on things to detect if a video is fake?" I think this idea, that video content can still be trusted and that {COMPANY} is being responsible is the real goal of the money pumped into these watermarking techniques. These techniques will not actually help people, images and video will continue to be used for disinformation. The only thing that can stymie that is a broad cultural shift to default to distrust of photographs and video footage, to treat it all like you might a painting or animated cartoon depicting an event; maybe an accurate portrayal, but just as easily totally fabricated. The responsible thing for companies to do would be to spread messaging indicative of this fact, but they would rather engage in safety theater and score some points while keeping users dumb and easily fooled.
Re: Sharing new research, models, and datasets from Meta FAIR
#46There’s honestly so much interesting stuff here, esp. the llm-related things - large concept models (operating on and predicting concepts, not tokens), dynamic byte latent transformers (byte-level alternative to standard tokenization), sparse memory layers (successfully scaling key-value memory layers without an increase in computational requirements). Here they are presented as separate things, each of which apparen…
Side track, but does anyone have suggestions about how to better present such content. I am struggling with similar docs/demos.
As a documentation page, each section is laid out uniformly with section heading, content, link to code and link to paper.
However the page itself is a blog post which will be difficult to find again next year.
Are there other examples of companies having well presented technical summaries which remain findable from the hime page?
Re: Sharing new research, models, and datasets from Meta FAIR
#47I was fortunate to get to a talk by Ross Taylor ex-Meta recently at the AI Engineer London meetup
He's recorded the full talk here now: https://www.youtube.com/watch?v=S5l5OvJ01ws
I had missed how much Meta have been doing on reasoning, ToM etc.
Re: Sharing new research, models, and datasets from Meta FAIR
#48When I wonder about the business behind Meta doing this, I see they have $70B in cash, so giving a bunch of AI experts hundreds of millions is pocket change.
everyone that has responded so far has it wrong (naively so). FB sells ad space on several apps. those apps needs people on them in order for the ad space to be worth anything. people, in turn, need content to attract them to the apps. so it's simple: enable people/companies/whomever to generate tons of content for cheap and consequently share it on the apps. that's it.
Except giving out the tools makes easier for competitors like TikTok to do the same, drawing revenue away from meta.
So that’s not it. Naively so.
Re: Sharing new research, models, and datasets from Meta FAIR
#49There’s honestly so much interesting stuff here, esp. the llm-related things - large concept models (operating on and predicting concepts, not tokens), dynamic byte latent transformers (byte-level alternative to standard tokenization), sparse memory layers (successfully scaling key-value memory layers without an increase in computational requirements). Here they are presented as separate things, each of which apparen…
I hope that Llama 4 or 5 will have a different architecture. All released llamas are +/- same inference with a better training pipeline. The downside is that llamacpp will probably not be able to run new models and maybe it will be too much big rewrite, so we will need new c,cpp,go,rust programs.
Re: Sharing new research, models, and datasets from Meta FAIR
#50There’s honestly so much interesting stuff here, esp. the llm-related things - large concept models (operating on and predicting concepts, not tokens), dynamic byte latent transformers (byte-level alternative to standard tokenization), sparse memory layers (successfully scaling key-value memory layers without an increase in computational requirements). Here they are presented as separate things, each of which apparen…
Side track, but does anyone have suggestions about how to better present such content. I am struggling with similar docs/demos. As a documentation page, each section is laid out uniformly with section heading, content, link to code and link to paper. However the page itself is a blog post which will be difficult to find again next year. Are there other examples of companies having well presented technical summaries w…
I'd put a table of contents-like page up front with some exciting short description of each section and use hyperlinks, allowing the user to navigate to the section and back