SAM 2: Segment Anything in Images and Videos • Hacker News Board

Re: SAM 2: Segment Anything in Images and Videos

#81

Post by ximilian » Tue, Jul 30, 2024, 7:13 AM UTC

Roughly how many fps could you get running this on a raspberry pi?

Re: SAM 2: Segment Anything in Images and Videos

#82

Post by sails » Tue, Jul 30, 2024, 7:14 AM UTC

Any use of this category of tools in OCR?

Re: SAM 2: Segment Anything in Images and Videos

#83

Post by cinntaile » Tue, Jul 30, 2024, 7:20 AM UTC

madduci wrote: post #65

nravi20 wrote: post #2

Hi from the Segment Anything team! Today we’re releasing Segment Anything Model 2! It's the first unified model for real-time promptable object segmentation in images and videos! We're releasing the code, models, dataset, research paper and a demo! We're excited to see what everyone builds! https://ai.meta.com/blog/segment-anything-2/

Thank you for sharing it! Is there any plans to move the codebase to a more performant programming language?

It's all C, C++ and Fortran(?) under the hood so moving languages probably won't matter as much as you expect.

Re: SAM 2: Segment Anything in Images and Videos

#84

Post by zengineer » Tue, Jul 30, 2024, 7:26 AM UTC

Would love to use it for my startup, but I believe it is to self-host on a server with GPU? Or is there an easy to use API?

Re: SAM 2: Segment Anything in Images and Videos

#85

Post by Gisbitus » Tue, Jul 30, 2024, 7:29 AM UTC

zengineer wrote: post #84

Would love to use it for my startup, but I believe it is to self-host on a server with GPU? Or is there an easy to use API?

It's OSS, so there isn't an "official" hosted version, but someone probably is gonna offer it soon.

Re: SAM 2: Segment Anything in Images and Videos

#86

Post by pzo » Tue, Jul 30, 2024, 7:31 AM UTC

zengineer wrote: post #84

Would love to use it for my startup, but I believe it is to self-host on a server with GPU? Or is there an easy to use API?

Previous SAM v1 you can use e.g. in here:

https://fal.ai/models

https://replicate.com/

You just have to wait probably few weeks for the SAM v2 to be available. Hugging Face might also have some offering

Re: SAM 2: Segment Anything in Images and Videos

#87

Post by michaelt » Tue, Jul 30, 2024, 7:33 AM UTC

alsodumb wrote: post #31

swyx wrote: post #23

i covered SAM 1 a year ago ( https://news.ycombinator.com/item?id=35558522 ). notes from quick read of the SAM 2 paper https://ai.meta.com/research/publications/sam-2-segment-anyt... 1. SAM 2 was trained on 256 A100 GPUs for 108 hours (SAM1 was 68 hrs on same cluster). Taking the upper end $2 A100 cost off gpulist means SAM2 cost ~$50k to train - surprisingly cheap for adding video understanding? 2. new dataset: the…

I might be minority, but I am not that surprised by the results or the not so significant GPU hours. I've been video segment tracking for a while now using SAM for mask generation and some of the robust academic video-object segmentation models (see CUTIE: https://hkchengrex.com/Cutie/ presented at CVPR this year.)for tracking the mask. I need to read SAM2 paper, but 4. seems a lot like what Rex has in CUTIE. CUTIE c…

Of course Facebook has had a video tracking ML model for a year or so - Co-tracker [1] - just tracking pixels rather than segments.

[1] https://co-tracker.github.io/

Re: SAM 2: Segment Anything in Images and Videos

#88

Post by leodriesch » Tue, Jul 30, 2024, 7:35 AM UTC

pzo wrote: post #55

Impressive, wondering if this is now out of the box fast enough to run on iphone. Previous SAM had some community projects such as FastSAM, MobileSAM, EfficientSAM that tried to speed up. Wish when Readme reporting FPS, provided on what hardware it was tested

I’d guess testing hardware is same as training hardware, so A100. If it was on a mobile device they would have definitely said that.

Re: SAM 2: Segment Anything in Images and Videos

#89

Post by phkahler » Tue, Jul 30, 2024, 7:42 AM UTC

8organicbits wrote: post #62

robbomacrae wrote: post #4

Earlier quoted context omitted.

Code, model, data and under Apache 2.0. Impressive. Curious how this was allowed to be more open source compared to Llama's interesting new take on "open source". Are other projects restricted in some form due to technical/legal issues and the desire is to be more like this project? Or was there an initiative to break the mold this time round?

Yeah, but there's a CLA for some reason. I'm wary they will switch to a new license down the road.

So get it today. You can't retroactively change a license on someone.

Re: SAM 2: Segment Anything in Images and Videos

#90

Post by ZephyrBlu » Tue, Jul 30, 2024, 7:44 AM UTC

glandium wrote: post #67

RobinL wrote: post #64

Earlier quoted context omitted.

Everything is a video. An image is the special case of length 1 frame

Here's a sentence I would understand: > We extend SAM to video and retrofit support for images by considering images as a video with a single frame. As it is written, I don't see the link between "We extend SAM to video" and "by considering images as a video with a single frame".

I read it like this:

- "We extend SAM to video", because is was previously only for images and it's capabilities are being extended to videos

- "by considering images as a video with a single frame", explaining how they support and build upon the previous image functionality

The main assumptions here are that images -> videos is a level up as opposed to being a different thing entirely, and the previous level is always supported.

"retrofit" implies that the ability to handle images was bolted on afterwards. "extend to video" implies this is a natural continuation of the image functionality, so the next part of the sentence is explaining why there is a natural continuation.