211–220 of 286 posts
Earlier quoted context omitted.
>I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from confidential sources. Think it is NOT just you. Most company with decent management also would not want their data going to anything outside the physical server they have in control of. But yeah for most people just use an app and hosted server. But this is HN,there are ppl he…
"Most company with decent management also would not want their data going to anything outside the physical server they have in control of." I don't think that's been true for over a decade: AWS wouldn't be trillion dollar business if most companies still wanted to stay on-premise.
Earlier quoted context omitted.
It is due to the risk of a leak. Laundering of data through training makes it a more complicated case than a simple data theft or copyright infringement. Leaks could be accidental, e.g. due to an employee logging in to their free-as-in-labor personal account instead of a no-training Enterprise account. It's safer to have a complete ban on providers that may collect data for training.
How can you be sure that AWS will not use your data to train their models? They got enormous data, probably most data in the world.
There is no world in which training on customer data without permission would be worth it for AWS.
Your data really isn't that useful anyway.
Earlier quoted context omitted.
Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…
My tooling doesn't measure TPS yet. It feels snappy to me on MLX. I agree that hosted models are usually a better option for most people - much faster, higher quality, handle longer inputs, really cheap. I enjoy local models for research and for the occasional offline scenario. I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from…
Can you explain this further? It seems in contrast to your previous comment about trusting Anthropic with your data
I have a few private “vibe check” questions and the 4 bit QAT 27B model got them all correctly. I’m kind of shocked at the information density locked in just 13 GB of weights. If anyone at Deepmind is reading this — Gemma 3 27B is the single most impressive open source model I have ever used. Well done!
I think this means I either have to train the -pt model with my own instruction tuning or use another provider :(
I think gemma-3-27b-it-qat-4bit is my new favorite local model - or at least it's right up there with Mistral Small 3.1 24B. I've been trying it on an M2 64GB via both Ollama and MLX. It's very, very good, and it only uses ~22Gb (via Ollama) or ~15GB (MLX) leaving plenty of memory for running other apps. Some notes here: https://simonwillison.net/2025/Apr/19/gemma-3-qat-models/ Last night I had it write me a complete…
Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…
(-t 32 -ngl 100 -c 8192 -fa -ctk q8_0 -ctv q8_0 -m models/gemma-3-27b-it-qat-q4_0.gguf)
Earlier quoted context omitted.
This is a really cool idea. Do you pretrain the model so it can tag people? I have so many photo's that it seems impossible to ever categorize them,using a workflow like yours might help a lot
No, tagging of people is already handled by another model. Gemma just describes what's in the image, and produces a comma separated list of keywords. No additional training is required besides a few tweaks to the prompt so that it outputs just the description, without any "fluff". E.g. it normally prepends such outputs with "Here's a description of the image:" unless you really insist that it should output only the d…
As an aside, what model/tools do you prefer for tagging people?
Earlier quoted context omitted.
My tooling doesn't measure TPS yet. It feels snappy to me on MLX. I agree that hosted models are usually a better option for most people - much faster, higher quality, handle longer inputs, really cheap. I enjoy local models for research and for the occasional offline scenario. I'm also interested in their applications for journalism, specifically for dealing with extremely sensitive data like leaked information from…
> specifically for dealing with extremely sensitive data like leaked information from confidential sources. Can you explain this further? It seems in contrast to your previous comment about trusting Anthropic with your data
If they get hit by a government subpoena because a journalist has been using them to analyze leaked corporate or government secret files I also trust them to honor that subpoena.
Sometimes journalists deal with material that they cannot risk leaving their own machine.
"News is what somebody somewhere wants to suppress"
Earlier quoted context omitted.
Can you quote tps? More and more I start to realize that cost saving is a small problem for local LLMs. If it is too slow, it becomes unusable, so much that you might as well use public LLM endpoints. Unless you really care about getting things done locally without sending information to another server. With OpenAI API/ChatGPT, I get response much faster than I can read, and for simple question, it means I just need…
> Whereas on local LLM, I watch it painstakingly prints preambles that I don't care about, and get what I actually need after 20 seconds. You may need to "right-size" the models you use to match your hardware, model, and TPS expectations, which may involve using a smaller version of the model with faster TPS, upgrading your jardware, or paying for hosted models. Alternatively, if you can use agentic workflows or tool…
Earlier quoted context omitted.
This is a really cool idea. Do you pretrain the model so it can tag people? I have so many photo's that it seems impossible to ever categorize them,using a workflow like yours might help a lot
No, tagging of people is already handled by another model. Gemma just describes what's in the image, and produces a comma separated list of keywords. No additional training is required besides a few tweaks to the prompt so that it outputs just the description, without any "fluff". E.g. it normally prepends such outputs with "Here's a description of the image:" unless you really insist that it should output only the d…