SAM 2: Segment Anything in Images and Videos • Hacker News Board

Re: SAM 2: Segment Anything in Images and Videos

#41

Post by ks2048 » Tue, Jul 30, 2024, 2:24 AM UTC

I would like to train a model to classify frames in a video (and identify "best" frame for something I want to locate, according to my training data).

Is SAM-2 useful to use as a base model to finetune a classifier layer on? Or are there better options today?

Re: SAM 2: Segment Anything in Images and Videos

#42

Post by daemonologist » Tue, Jul 30, 2024, 2:26 AM UTC

Nice! Of particular interest to me is the slightly improved mIoU and 6x speedup on images [1] (though they say the speedup is mainly from the more efficient encoder, so multiple segmentations of the same image presumably would see less benefit?). It would also be nice to get a comparison to original SAM with bounding box inputs - I didn't see that in the paper though I may have missed it.

[1] - page 11 of https://ai.meta.com/research/publications/sam-2-segment-anyt...

Re: SAM 2: Segment Anything in Images and Videos

#43

Post by gpm » Tue, Jul 30, 2024, 2:31 AM UTC

Interesting how you can bully the model into accepting multiple people as one object, but it keeps trying to down-select to just one person (which you can then fix by adding another annotated frame in).

Re: SAM 2: Segment Anything in Images and Videos

#44

Post by djsavvy » Tue, Jul 30, 2024, 3:07 AM UTC

Imnimo wrote: post #19

I think the first SAM is the open source model I've gotten the most mileage out of. Very excited to play around with SAM2!

What have you found it useful for?

Re: SAM 2: Segment Anything in Images and Videos

#45

Post by vivzkestrel » Tue, Jul 30, 2024, 3:25 AM UTC

nravi20 wrote: post #2

Hi from the Segment Anything team! Today we’re releasing Segment Anything Model 2! It's the first unified model for real-time promptable object segmentation in images and videos! We're releasing the code, models, dataset, research paper and a demo! We're excited to see what everyone builds! https://ai.meta.com/blog/segment-anything-2/

stupid question from a noob: what exactly is object segmentation? what does your library actually do? Does it cut clips?

Re: SAM 2: Segment Anything in Images and Videos

#46

Post by sagz » Tue, Jul 30, 2024, 3:26 AM UTC

acacac wrote: post #9

nravi20 wrote: post #2

Hi from the Segment Anything team! Today we’re releasing Segment Anything Model 2! It's the first unified model for real-time promptable object segmentation in images and videos! We're releasing the code, models, dataset, research paper and a demo! We're excited to see what everyone builds! https://ai.meta.com/blog/segment-anything-2/

will the model ever be extended to being able to segment audio (eg. different people talking, different instruments in a soundtrack?)

Check out Facebook DeMucs, and more newer: Ultimate Vocal Remover project on GitHub

Re: SAM 2: Segment Anything in Images and Videos

#47

Post by rawrawrawrr » Tue, Jul 30, 2024, 3:35 AM UTC

ks2048 wrote: post #32

minimaxir wrote: post #14

The web demo is actually pretty neat: https://sam2.metademolab.com/demo I selected each shoe as individual objects and the model was able to segment them even as they overlapped.

It is giving me "Access Denied".

Might have issues if you're from Texas or Illinois due to their local laws.

Re: SAM 2: Segment Anything in Images and Videos

#48

Post by rawrawrawrr » Tue, Jul 30, 2024, 3:36 AM UTC

vanjajaja1 wrote: post #27

Cool! Seems this is cuda only?

Can run on CPU (slower) or AMD GPUs.

Re: SAM 2: Segment Anything in Images and Videos

#49

Post by [deleted] » Tue, Jul 30, 2024, 3:44 AM UTC

[deleted]

Re: SAM 2: Segment Anything in Images and Videos

#50

Post by ei8htyfi5e » Tue, Jul 30, 2024, 3:45 AM UTC

Will it handle tracking out of frame?

i.e. if I stand in the center of my room and take a video of the room spinning around slowly over 5 seconds. Then reverse spin around for 5 seconds.

Will it see the same couch? Or will it see two couches?