Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Windows AI Studio Preview

github.com

Post reply on HN
Print view View original

51–60 of 69 posts

Re: Windows AI Studio Preview

#51

Post by nsagent »

ekianjo wrote: post #39
iotku wrote: post #17

Earlier quoted context omitted.

Counterpoint: https://forums.developer.nvidia.com/t/545-drivers-have-bad-f... https://forums.developer.nvidia.com/t/wayland-native-wayland... https://gitlab.freedesktop.org/xorg/xserver/-/issues/1317 and literally me being a Linux Nvidia 1080ti user for years and having plenty of issues https://i.imgur.com/lwHDZoq.png

I thought we were talking about CUDA here?

As someone who has had a home ML server since 2016 with two TitanX GPUs, and has worked on and maintained numerous servers since then I can definitely echo that maintaining Nvidia drivers, along with CUDA, CUDNN, etc has always been a hassle. It's certainly gotten better over time, but it's still quite fragile.

Automatic kernel update? https://forums.developer.nvidia.com/t/nvidia-smi-not-working...

What about upgrading to a new version of CUDA? https://stackoverflow.com/questions/43022843/nvidia-nvml-dri...

What about trying something like enabling forward compatibility for CUDA using an older driver? https://discuss.pytorch.org/t/torch-is-unable-to-detect-cuda... (This issue was actually just posted within the last day, so clearly people still have problems.)

If you haven't run into any issues, then I'd say you're very lucky. Just don't pretend lots of others haven't run into issues.

Re: Windows AI Studio Preview

#52

Post by lostmsu »

andruby wrote: post #5

Apple has good uniform hardware to enable this, but they are a product company and an “AI studio” would not fit their usual definition of a product. I do hope they are considering going in that direction though.

Nobody is going to train or even fine-tune large models on Apple hardware. It is too slow for that purpose.

Re: Windows AI Studio Preview

#54

Post by Const-me »

capableweb wrote: post #47
Const-me wrote: post #22

> will run only on NVIDIA GPUs for the preview I wonder why Microsoft helps nVidia, instead of using their own technology? Here’s an example: https://github.com/Const-me/Cgml

Companies of the size of Microsoft usually function more like a country with various companies within, rather than one company with a unified view and unified goals. So probably, the group wasn't even aware of that technology because it's far away by either professional connection, or by personal/relationship connections, or they knew about it but had another goal than "maximize use of own stuff" and made the call th…

I don’t work for Microsoft, and I don’t care whether people are using Microsoft’s technologies. As a consumer, I’m not happy with nVidia’s monopoly in the field. I find it disappointing that Microsoft chose to play along, instead of disrupting the landscape.

I think the only reason for that monopoly is the mental inertia of everyone involved. The complexity of these AI models is contained within the data in the models (gigabytes of numbers in these tensors), the GPU-running code is rather simple, most of that code is basic BLAS stuff. Unlike traditional GPGPU applications (FEM, numerical simulations, fluid dynamics), the compute kernels used in AI are easily portable across GPU APIs.

Maybe when I have some free time, I should port my library to Linux + Vulkan, just to prove the point.

Re: Windows AI Studio Preview

#55

Post by davikr »

capableweb wrote: post #49
davikr wrote: post #46

Anyone knows when Microsoft will release a local OCR model officially? I haven't seen anyone talking about this, but the one they're shipping with Snipping Tool (OneOcr) is top-tier and beats everything out there like Tesseract, easyOCR etc. The model is technically in everyone's Windows installs, but we don't have the C++ projected WinRT headers to use the Microsoft.Windows.Vision library.

Seems they've released a bunch already? Image-to-text models, filtered by "Microsoft": https://huggingface.co/models?pipeline_tag=image-to-text&sor...

Wow, very nice! Thank you, hadn't come across them in my search.

It looks like it doesn't recognize more than one line at once, but combined with another model or algorithm to detect text bounding boxes it'd be handy.

Re: Windows AI Studio Preview

#56

Post by Const-me »

lostmsu wrote: post #53
Const-me wrote: post #22

> will run only on NVIDIA GPUs for the preview I wonder why Microsoft helps nVidia, instead of using their own technology? Here’s an example: https://github.com/Const-me/Cgml

Have you considered ILGPU? https://ilgpu.net/

No, I have not considered ILGPU, it’s the first time I see that thing. Very interesting, but I’m not sure I would have used it for that project.

In my experience, using any non-native GPU API is asking for troubles. On Windows, the native ones are D3D 11 and 12, on Linux and Android it’s often Vulkan, and on MacOS it’s either Metal or that newer thing they have built specifically for AI. It seems ILGPU only has backends for CUDA, OpenCL, and CPU SIMD.

I have general suspicion towards custom compilers. Writing a good compiler is hard, for GPUs even harder due to the weird execution model and insufficient documentation from GPU vendors. HLSL compiler is supported by Microsoft, and compute shaders are used by many millions of gamers every day. Similarly, CUDA is supported by nVidia, usually pretty stable, my only issue with CUDA is vendor lock-in. I have an impression people are often unhappy with the quality and hardware compatibility of less popular GPU APIs like ROCm and OpenCL.

BTW, I asked a friend to test my program on their low-and laptop. The laptop has some Intel core i3 with integrated GPU. The performance wasn’t great at about 1 token/second (single-channel memory), but at least my code worked. I’m not sure it would have worked on that computer if the backend was based on OpenCL.

Re: Windows AI Studio Preview

#57

Post by jeroenhd »

hu3 wrote: post #50
jeroenhd wrote: post #32

Earlier quoted context omitted.

I think it's clear that GPU manufacturers are scamming their customers more than Apple does with their ridiculous prices. There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM. Nvidia intentionally nerfs the amount of VRAM in their consumer cards so you need to buy their ridiculously overpriced enterprise cards. I think it's fair to say Apple's lineup is the ch…

> There's an 80GB GPU out there that costs about six times as much as an equivalent Mac with 96 GB of shared RAM. Those are not equivalent in speed. macs RAM are much slower than these GPUs.

There is no Apple-to-apples comparison between a Macbook and a GPU. However, the problem with running most models is the lack of simultaneous RAM.

You can swap memory back and forth between RAM and VRAM (with a huge performance penalty) of course, but that's not exactly usable or comparable to what Apple's VRAM sharing setup allows.

Nvidia doesn't sell a nice-but-not-amazing GPU equivalent to Apple's processing power and memory bandwidth that's also capable of operating on >80GB of VRAM at once. Apple's SoC is kind of an oddball in that regard.

I suppose you could take a regular old iGPU (for AMD, Intel, probably also Qualcom/Mediatek) and use its shared memory capabilities as a comparison. However, iGPUs are terrible at machine learning tasks, they don't come close to what Apple can do with their dedicated accelerators.

The best middle ground may be the laptop GPUs with both dedicated RAM and shared RAM, but those will start swapping memory back and forth like crazy running large ML workloads so they're not really comparable.

If you want to run a model that operates on a huge amount of memory at once, I don't think there is a desktop option that can do what Apple does without going for the massive overkill GPUs that will crush the Macbook in terms of performance (at great cost).

Perhaps you know a GPU or iGPU that's capable of running 80GB VRAM workloads at comparable speeds? Because I don't.

Re: Windows AI Studio Preview

#58

Post by CharlieDigital »

davikr wrote: post #55
capableweb wrote: post #49

Earlier quoted context omitted.

Seems they've released a bunch already? Image-to-text models, filtered by "Microsoft": https://huggingface.co/models?pipeline_tag=image-to-text&sor...

Wow, very nice! Thank you, hadn't come across them in my search. It looks like it doesn't recognize more than one line at once, but combined with another model or algorithm to detect text bounding boxes it'd be handy.

Not sure if it's what's used in the Azure Document Intelligence service, but my experience with it is pretty good.

The one somewhat unique offering in Azure is the Document Layout model which gives you back the OCR with titles, headers, paragraphs, and tables all labeled.

This is really, really good for RAG since it's often useful to stuff the nearest header into the chunk of text when generating an embedding (much, much better results this way).

Re: Windows AI Studio Preview

#59

Post by angm128 »

freeone3000 wrote: post #10
theturtle32 wrote: post #9

Earlier quoted context omitted.

Having had the displeasure of trying to get CUDA running under WSL2, I can tell you it is most definitely not a better user experience :-P

Is it a better user experience than trying to get CUDA natively? :P

My personal experience: Fedora 38- just needed to compile gcc-12 (took very long), Windows- Installer failed

This clearly shows that GNU/Linux must be superior

Re: Windows AI Studio Preview

#60

Post by fragmede »

TheRoque wrote: post #15
andruby wrote: post #5

Apple has good uniform hardware to enable this, but they are a product company and an “AI studio” would not fit their usual definition of a product. I do hope they are considering going in that direction though.

That's the weird thing. They seem to have the best hardware for the price, for individuals to develop and use local LLMs, but so far they have been pretty quiet on all this.

https://github.com/ml-explore/mlx-examples/ is Apple engineers working to get ML models going on Apple Silicon's Metal hardware.
Post reply on HN