341–350 of 385 posts
Earlier quoted context omitted.
How is it bypassed?
There's a whitelist identifier that you can add bundle IDs to, to get access to the more sensitive APIs. Then you can download the extension file and install it manually. I don't have the exact process right now but just Google it :)
Earlier quoted context omitted.
True but try to find a 96GB GPU.
This drives me insane. PC Part Picker, DDR5-8400 48 GB (2x24GB) is... $340 right now. For $680 you can get 96 GB of very fast RAM. How about someone make an NVidia GPU with 96 GB of RAM at a reasonable price? Please?
Earlier quoted context omitted.
>the telemetry of VSCode is non-personal metrics But I don't want it. I want my software to work for me, not against me. >and some of the most popular extensions are only available with VSCode, not with Codium. I'll manage without them. What's especially annoying is that this restriction is completely artificial. Having said that, MS did a great job with VsCode and I applaud them for that. I guess nothing is perfect,…
> But I don't want it. I want my software to work for me, not against me. How is said software working "against" you by collecint non-personal telemetry while purpose of that telemetry usually is making the software better for most users?
For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…
Many setup rely on Nvidia GPUs, Intel stuff, Windows or other stuff, that I would rather not use, or are not very clear about how to set things up. What are some recommendations for running models locally, on decent CPUs and getting good valuable output from them? Is that llama stuff portable across CPUs and hardware vendors? And what do people use it for?
Ollama works on anything: Windows, Linux, Mac and Nvidia or AMD. I don't know if other cards like Arc are supported by anything yet, bit of it supports the open Vulkan API (like AMD) then it should work.
Every inference server out there supports running from CPU, but realize that it's much slower than running on a GPU - that's why this revolution didn't begin until GPUs became powerful and affordable.
As far as being clear to setup, Ollama is trivial: it's a single command line that only asks what model you want and they provide you with a list on their website. They even have a Docker container if you don't want to worry about installing any dependencies. I don't know what could be easier than that.
Most other tools like LM Studio or Jan are just a fancy UI running llama.cpp as their server and using HuggingFace to download the models. They don't even offer anything beyond simple inference, such as RAG or agents.
I've yet to see anything more than a simple RAG that's available to use out of the box for local use. The only full service tools are online services like Microsoft Copilot or ChatGPT. Anyone else who wants to do that more advanced kind of system ends up writing their own code. It's not hard if you know Python - there are lots of libraries available like HuggingFace, LangChain, and Llama-Index, as well as millions of tutorials (every blog has one).
Maybe that's a sign that there's room for an open source platform for this kind of thing, but given that it's a young field and everyone is rushing to become the next big online service or toolkit, there might not be as much interest from developers to build an open source version of a high quality online service.
For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…
For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…
But how good are these models compared to gpt4o? My last experience with llama2-8b was not great at all. Are there really that good models that would fit on an average consumer hardware (mine has already 32GB ram and 16GB vram)?
Earlier quoted context omitted.
Not only are they the only future worth living in, incentives are aligned with client-side AI. For governments and government contractors, plumbing confidential information through a network isn't an option, let alone spewing it across the internet. It's a non-starter, regardless of the productivity bumps stuff like Copilot can provide. The only solution is to put AI compute on a cleared individual's work computer.
> plumbing confidential information through a network isn't an option So do you think government doesn't use networks?
In comments sections for repost, I did mentioned that big data is not dead but just having its AI winter moment, and big data is not only about volume (of storage) but also on the memory requirements (of RAM) [3].
Fast forward a few months, it seems the AI LLM is reviving the big data scenario and is seems that big data is far from dead but it's very much healthy in the era of LLMs, cloud or local. Wait until IoT and machine-to-machine (M2M) based systems are in full effects and big data will not just surviving but will be thriving.
[1] Big data is dead (2023) - original
https://news.ycombinator.com/item?id=34694926
[2] Big data is dead (2023) - repost
https://news.ycombinator.com/item?id=40488844
[3] Comments on: Big data is dead (2023):
For anyone who hasn't tried local models because they think it's too complicated or their computer can't handle it, download a single llamafile and try it out in just moments. https://future.mozilla.org/builders/news_insights/introducin... https://github.com/Mozilla-Ocho/llamafile They even have whisperfiles now, which is the same thing but for whisper.cpp, aka real-time voice transcription. You can also take this a…
I have now learned that my laptop is capable of a whopping 0.37 tokens per second.
11th Gen Intel® Core™ i7-1185G7 @ 3.00GHz × 8
Earlier quoted context omitted.
I just tried now. Super easy indeed but slow to the point it's not usable on my PC
You need an RTX 4090 if you want enough speed