Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Everything we announced at our first LlamaCon

ai.meta.com

Post reply on HN
Print view View original

71–80 of 122 posts

Re: Everything we announced at our first LlamaCon

#71

Post by philipkglass »

stavros wrote: post #66
philipkglass wrote: post #29

Earlier quoted context omitted.

I didn't try original BERT at all because I didn't get good results from any LLMs on small document excerpts, so I assumed that a substantial context was necessary for good results. Traditional BERT only accepts up to 512 tokens, while ModernBERT goes up to 8192. I ended up using a 2048 token limit.

Would you happen to know of any resources for how to distill a ModernBERT model out of a larger one? I'm interested in doing exactly what you did, but I don't know how to start.

I was trying to identify "evergreen" and "time-sensitive" kinds of writing -- basically, I wanted to figure out if web pages captured in 2016 would still have content that's interesting to read today or if the passage of time would have rendered them irrelevant.

Here's the training code that I used to fine-tune ModernBERT from the ~5000 pages I had labeled with Llama 3.3. It should be a good starting point if you have your own fine-tuning task like this. If you can get away with a smaller context than I used here, it will be much faster and the batches can be larger (requires experimentation).

https://pastebin.com/Saq1EyAB

Re: Everything we announced at our first LlamaCon

#72

Post by danpalmer »

mrcwinn wrote: post #65
divan wrote: post #59

Earlier quoted context omitted.

I'm working in Quest 3 almost every day. I use Immersed, as it implements virtual displays for my MacBook better than others, but I'm impressed with the Meta ecosystem. Granted, social interaction is still awkward without proper face expressions, but it feels closer each year to the depicted vision. I recently travelled and needed to work (coding and video editing in DaVinci) a lot in hotels and random places. I can'…

In no way is Quest 3 better than Apple Vision Pro for desktop work. I’ve used both. It’s not close. (There are other critiques of AVP that might not make it the right choice, but desktop work experience isn’t one of them.)

Perhaps on macOS, but I'd be surprised if the AVP has the same experience for Linux/Windows/ChromeOS?

Re: Everything we announced at our first LlamaCon

#73

Post by behnamoh »

mvieira38 wrote: post #27
ahmedfromtunis wrote: post #23

Does anyone use llama as their primary model for any usecase? Maybe it's my fault for not spending much time with it, but I still couldn't find the applications for which llama has an advantage over the competition.

It's pretty popular in the local LLM space

Nah, most people have moved on to Gemma, Qwen, Mistral Small/Nemo variants.

Re: Everything we announced at our first LlamaCon

#74

Post by jitl »

mrcwinn wrote: post #65
divan wrote: post #59

Earlier quoted context omitted.

I'm working in Quest 3 almost every day. I use Immersed, as it implements virtual displays for my MacBook better than others, but I'm impressed with the Meta ecosystem. Granted, social interaction is still awkward without proper face expressions, but it feels closer each year to the depicted vision. I recently travelled and needed to work (coding and video editing in DaVinci) a lot in hotels and random places. I can'…

In no way is Quest 3 better than Apple Vision Pro for desktop work. I’ve used both. It’s not close. (There are other critiques of AVP that might not make it the right choice, but desktop work experience isn’t one of them.)

Wearing the AVP 3 days in a row for 30 minutes each day leaves me with neck pain that lasts another 3 days. It’s just too heavy for me and for a lot of others, even if the visual quality is very high the overall experience is very poor if it leaves me with a medical issue. I’m still impressed with some aspects and look forward to a lighter weight HMD from Apple but right now… can’t do it.

Re: Everything we announced at our first LlamaCon

#75

Post by tiahura »

divan wrote: post #59
blitzar wrote: post #48

Its impressive that Llama and the Ai teams in general survived the meta-verse push at Facebook. Congrats to the team for keeping their heads down and saving the company from itself. Its all Ai all the time now though, not seen any mention of our reimagined future of floating heads hanging out together in quite some time.

I'm working in Quest 3 almost every day. I use Immersed, as it implements virtual displays for my MacBook better than others, but I'm impressed with the Meta ecosystem. Granted, social interaction is still awkward without proper face expressions, but it feels closer each year to the depicted vision. I recently travelled and needed to work (coding and video editing in DaVinci) a lot in hotels and random places. I can'…

Have you thought about donating your brain to science for an examination as to why you are not racked migraines and nausea the way most of us are when we use these things more than an hour or so?

Re: Everything we announced at our first LlamaCon

#76

Post by littlestymaar »

KTibow wrote: post #35
wewewedxfgdf wrote: post #34

Can someone explain to me please why Meta doesn't create subject specific versions of their LLMs such as one that knows only about computer programming, computers, hardware software. I would have imagined such a thing would be smaller and thus run on smaller configurations. But since I am only a layman maybe someone can tell me why this isn't the case?

Other companies have done this (see Qwen Coder). It doesn't scale past a few disciplines like math and code though, and using mixtures of experts give you most of the same benefits.

Unlike what their names imply, MoE don't have a concept of domain expert at all.

It's just a fancy name for sparse evaluation of the total network to save compute and memory bandwidth.

Re: Everything we announced at our first LlamaCon

#77

Post by littlestymaar »

bbatha wrote: post #40
wewewedxfgdf wrote: post #34

Can someone explain to me please why Meta doesn't create subject specific versions of their LLMs such as one that knows only about computer programming, computers, hardware software. I would have imagined such a thing would be smaller and thus run on smaller configurations. But since I am only a layman maybe someone can tell me why this isn't the case?

To add on to the sibling. Specialized models, including fine tuned ones, continually have their lunch eaten by general models within 3-6 months. This time round is mixture of experts that’ll do it, next year it’ll be something else. Tuned models are expensive to produce and are benchmark kings but less do less well in the real world qualitative experience. The juice just ain’t worth the squeeze most of the time. Meta…

> Tuned models are expensive to produce

The expensive part is building the dataset, training itself isn't too expensive (you can even fine-tune small models on free colab instances!), and when you have your dataset, you can just fine tune the next generalist model as soon as it's released and you're good to go ago.

Re: Everything we announced at our first LlamaCon

#78

Post by drusepth »

mrcwinn wrote: post #65
divan wrote: post #59

Earlier quoted context omitted.

I'm working in Quest 3 almost every day. I use Immersed, as it implements virtual displays for my MacBook better than others, but I'm impressed with the Meta ecosystem. Granted, social interaction is still awkward without proper face expressions, but it feels closer each year to the depicted vision. I recently travelled and needed to work (coding and video editing in DaVinci) a lot in hotels and random places. I can'…

In no way is Quest 3 better than Apple Vision Pro for desktop work. I’ve used both. It’s not close. (There are other critiques of AVP that might not make it the right choice, but desktop work experience isn’t one of them.)

I've only briefly used an AVP (used a friend's before he returned it), but I didn't really notice that much difference in screen quality between it and my Quest 3. That's really what I was hoping it'd excel at given the price, but the real killer for me was just the bulkiness/weight of it compared to other headsets. I can't see myself working in it for longer than maybe 30-45 min compared to the 6+ hours I've put in with virtual desktops in Quest 3/Vive.

Re: Everything we announced at our first LlamaCon

#79

Post by eddythompson80 »

throwaw12 wrote: post #4

Feels like Meta is going to Cloud services business but in AI domain. They resisted entering cloud business for so long, with the success of AWS/Azure/GCP I think they are realizing they can't keep at the top only with social networks without owning a platform (hardware, cloud)

I don’t know if an API service means they are going up against AWS next. I could totally see it though, and would make sense for them as a company. It makes more sense for them than Amazon of all places.

Re: Everything we announced at our first LlamaCon

#80

Post by nailer »

This is odd - it keeps mentioning Open Source but the Llama license isn’t open source at all - see https://www.llama.com/llama3/license/. The “additional commercial terms” section violates the open source definition as does the advertising clause.

I would’ve hoped to have seen Meta, in their supposed dedication to open source, actually fix it.

Post reply on HN