Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: AI climbing coach – visualize how to climb any route based on your body

climbing.ai

Post reply on HN
Print view View original

1–10 of 149 posts

Show HN: AI climbing coach – visualize how to climb any route based on your body

#1

Post by smandava »

I made SABR - an AI model that helps you visualize the beta/technique on any route, based on your body parameters. You can input a video of you climbing any route, in any orientation or lighting condition (it's truly versatile!). SABR then creates a virtual avatar of your body shape and uses it to climb the route you're climbing. Then, you can compare/contrast.

You can see the demo here: https://www.youtube.com/watch?v=cnvNPWoYZz4

Will be open sourcing the model, backend, and frontend codebase soon!

Show HN: AI climbing coach – visualize how to climb any route based on your body
climbing.ai

Re: Show HN: AI climbing coach – visualize how to climb any route based on your body

#3

Post by gus_massa »

Does the AI create the 3D model from a static camera?

Can I choose the color of the avatar? (It's not an important feature, but people will love it.)

Does it work with natural walls? (I guess the artificial walls have "handles" with an standard size that makes guesing the scale easier for the computer.)

Re: Show HN: AI climbing coach – visualize how to climb any route based on your body

#4

Post by smandava »

gus_massa wrote: post #3

Does the AI create the 3D model from a static camera? Can I choose the color of the avatar? (It's not an important feature, but people will love it.) Does it work with natural walls? (I guess the artificial walls have "handles" with an standard size that makes guesing the scale easier for the computer.)

- it creates a 3D model of you based on the first second of your body in the video. How the video is recorded does affect how the 3D avatar looks.

- don't have that functionality but its really easy to implement.

- ideally, it works for anything. I will attach some outputs of it climbing outdoor routes. The model doesn't know what a wall means. It just has seen enough data of people climbing that it can somehow correlate certain features in videos to certain human movements.

Re: Show HN: AI climbing coach – visualize how to climb any route based on your body

#5

Post by smandava »

Note:

Will attach more example outputs and make a detailed document about how the model was built and the research behind it. If this is interesting to you, feel free to sign up to the waitlist on www.climbing.ai (and make sure to sign up for our Discord!)

I originally planned on open sourcing the model, data, weights, and code. Only a few people (me and a couple friends) have access to the hosted model on the web app.

If enough people are on the waitlist, I will consider releasing access. This is very expensive to run so was only considering open sourcing it.

Note: the model works well sometimes, but most of the times it does not. This is an early research preview. Please tamper any expectations. A really good general model requires millions of videos and much more training time so it is really prohibitively expensive. As mentioned before, if someone has the compute, all they have to do is scale the existing dataset and training pipeline (which I will publish open-source in the coming weeks).

Re: Show HN: AI climbing coach – visualize how to climb any route based on your body

#7

Post by gus_massa »

smandava wrote: post #4
gus_massa wrote: post #3

Does the AI create the 3D model from a static camera? Can I choose the color of the avatar? (It's not an important feature, but people will love it.) Does it work with natural walls? (I guess the artificial walls have "handles" with an standard size that makes guesing the scale easier for the computer.)

- it creates a 3D model of you based on the first second of your body in the video. How the video is recorded does affect how the 3D avatar looks. - don't have that functionality but its really easy to implement. - ideally, it works for anything. I will attach some outputs of it climbing outdoor routes. The model doesn't know what a wall means. It just has seen enough data of people climbing that it can somehow corre…

My guess is that for the computer it's easier in an artificial wall becasue features are standarized.

(Disclaimer: I never climbed a wall, but from time to time I fall in the https://www.youtube.com/@AlbertOkay rabit hole. In particular this video looks relevant for my question https://www.youtube.com/watch?v=onXrJ5sX9BU (but I din't watch it yet.))

Re: Show HN: AI climbing coach – visualize how to climb any route based on your body

#8

Post by smandava »

jfengel wrote: post #6

I look forward to playing with this. It seems astonishing.

I will be open sourcing the codebase and model. Due to compute and training dataset size constraints, the model is not exactly "production ready". I have (and am) cherry picking rare examples of it where it is decent. You can play around with it when I open source it but truly good results require multiple millions of dollars (but it really is a linear scaling of compute and dataset).

Re: Show HN: AI climbing coach – visualize how to climb any route based on your body

#9

Post by smandava »

gus_massa wrote: post #7
smandava wrote: post #4

Earlier quoted context omitted.

- it creates a 3D model of you based on the first second of your body in the video. How the video is recorded does affect how the 3D avatar looks. - don't have that functionality but its really easy to implement. - ideally, it works for anything. I will attach some outputs of it climbing outdoor routes. The model doesn't know what a wall means. It just has seen enough data of people climbing that it can somehow corre…

My guess is that for the computer it's easier in an artificial wall becasue features are standarized. (Disclaimer: I never climbed a wall, but from time to time I fall in the https://www.youtube.com/@AlbertOkay rabit hole. In particular this video looks relevant for my question https://www.youtube.com/watch?v=onXrJ5sX9BU (but I din't watch it yet.))

Yeah it performs best with indoor bouldering videos with spread out holds and great lighting. I will post examples of it climbing natural walls in the paper. Will send soon.

Re: Show HN: AI climbing coach – visualize how to climb any route based on your body

#10

Post by goodmattg »

A few thoughts as I've done academic research & built products in this area:

- if you're using SMPL body parameters this will have to stay research / open-source - is this leveraging some sort of monocular depth estimation to estimate the wall in 3D space? Also, do you have assumed camera parameters, or is that also estimated? If there isn't any depth information, this will be highly inaccurate on any cliff routes, but still useful on flat wall climbing.

Overall, a good idea (that I've also thought about building as a climber) - the tricky part that I'm impressed you have a solution to is path planning up the wall. Even assuming a flat wall with no depth estimation, it's still looks effective.

Post reply on HN