Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Off Grid – Run AI text, image gen, vision offline on your phone

github.com

Post reply on HN
Print view View original

61–70 of 71 posts

Re: Show HN: Off Grid – Run AI text, image gen, vision offline on your phone

#61

Post by noodletheworld »

resonious wrote: post #32
noodletheworld wrote: post #28

Nice idea, but isnt this kind of daft? There are basically no useful models that run on phone hardware. > Results vary by model size and quantization. I bet they do. Look, if you cant run models on your desktop, theres no way in hell they run on your phone. The problem with all of these self hosting solutions is that the actual models you can run on them aren't any good. Not like, “chat gpt a year ago” not good. Like…

The app is basically just a wrapper that makes it super easy to set this up, which I'm very thankful for. I sometimes want to toy with this stuff but the amount of tinkering and gluing things together needed to just get a chat going is always too much for me. The fact that the quality of the AI isn't good is just the models not being quite there yet. If the models get better, this app will be killer. If there's a sim…

> If the models get better, this app will be killer.

Any random thing might happen in the future.

That doesnt have any bearing on how useful this is right now.

All we can do is judge right now how this compares to what it promises.

Re: Show HN: Off Grid – Run AI text, image gen, vision offline on your phone

#62

Post by Terretta »

wittlesus wrote: post #23

[dead]

> The fact that you're getting 15-30 tok/s for text gen on phone hardware is wild — that's basically usable for real conversations.

Wild how bad it is compared to, say, Russet for iOS/ipadOS, which runs these same models at 110 tps.

Re: Show HN: Off Grid – Run AI text, image gen, vision offline on your phone

#63

Post by dotancohen »

ali_chherawalla wrote: post #27
ali_chherawalla wrote: post #20

Earlier quoted context omitted.

sorry about that one. I'm taking a look and fixing it right away

hey, just pushed a fix for it here: https://github.com/alichherawalla/off-grid-mobile/releases/t... Thanks for spotting and reporting this.

Great work! I can confirm that the issue is resolved on the Samsung S24 Ultra.

Re: Show HN: Off Grid – Run AI text, image gen, vision offline on your phone

#64

Post by altcunn »

noodletheworld wrote: post #28

Nice idea, but isnt this kind of daft? There are basically no useful models that run on phone hardware. > Results vary by model size and quantization. I bet they do. Look, if you cant run models on your desktop, theres no way in hell they run on your phone. The problem with all of these self hosting solutions is that the actual models you can run on them aren't any good. Not like, “chat gpt a year ago” not good. Like…

[dead]

Re: Show HN: Off Grid – Run AI text, image gen, vision offline on your phone

#65

Post by ali_chherawalla »

dotancohen wrote: post #63
ali_chherawalla wrote: post #27

Earlier quoted context omitted.

hey, just pushed a fix for it here: https://github.com/alichherawalla/off-grid-mobile/releases/t... Thanks for spotting and reporting this.

Great work! I can confirm that the issue is resolved on the Samsung S24 Ultra.

awesome. thanks!

Re: Show HN: Off Grid – Run AI text, image gen, vision offline on your phone

#67

Post by ali_chherawalla »

Swauzie wrote: post #66

I gave it a try on Graphene OS: I downloaded models 1st, then blocked Network Accesses. The Apps works fine for text generation, but not for image generation: it needs network access... I did not search where it needs to go.

it doesn't need internet to generate an image. it needs it to download the model, and to be fair if you've got the zip for the model already you can just import that

Re: Show HN: Off Grid – Run AI text, image gen, vision offline on your phone

#68

Post by Swauzie »

ali_chherawalla wrote: post #67
Swauzie wrote: post #66

I gave it a try on Graphene OS: I downloaded models 1st, then blocked Network Accesses. The Apps works fine for text generation, but not for image generation: it needs network access... I did not search where it needs to go.

it doesn't need internet to generate an image. it needs it to download the model, and to be fair if you've got the zip for the model already you can just import that

I have download the model... only then I have disabled network access. It loops with an error message needing access to DSP ?

Anyway, I like what you did, as I do not want to train AI with my inputs this is perfect.

Re: Show HN: Off Grid – Run AI text, image gen, vision offline on your phone

#69

Post by ali_chherawalla »

Swauzie wrote: post #68
ali_chherawalla wrote: post #67

Earlier quoted context omitted.

it doesn't need internet to generate an image. it needs it to download the model, and to be fair if you've got the zip for the model already you can just import that

I have download the model... only then I have disabled network access. It loops with an error message needing access to DSP ? Anyway, I like what you did, as I do not want to train AI with my inputs this is perfect.

I think thats a bug, I'm guessing you're trying one of the NPU models? If you drop to CPU for now it should hold.

Only phones with qualcomm chips are able to use the NPU. I'm working towards changing that.

but yeah just to be clear there is no internet needed to run any of this. Infact I'm so averse to it, I've not even added analytics for this one. So flying pretty blind here.

Post reply on HN