Wondering why use Rust other than C++
Why use C++?
So C++ users don't need to learn something new.
Show HN: We made our own inference engine for Apple Silicon
41–48 of 48 posts
Re: Show HN: We made our own inference engine for Apple Silicon
#42I just spun up a AWS EC2 g6.xlarge instance to do some llm work. The GPU is NVIDIA L4 24GB and costs $0.8048/per hour. Starting to think about switching to an Apple mac2-m2.metal instance for $0.878/ per hour. Big question is the Mac instance only has 24GB of unified memory.
Unified memory doesn't compare to a Nvidia GPU, the latter is much better.
Just depends on what performance level you need.
Re: Show HN: We made our own inference engine for Apple Silicon
#43Have you added it to HomeBrew and other package managers yet? Also any app deployed to PROD but developed on Mac need to be consistent i.e. work on Linux/in container.
Needs an "API key".
https://github.com/trymirai/uzu-swift?tab=readme-ov-file#qui...
Re: Show HN: We made our own inference engine for Apple Silicon
#44>faster than llama cpp in all of the use cases What's your deliberate, well-thought roadmap for achieving adoption similar to llama cpp?
Ollama is the leader isn't it? Brew stats (downloads last 30 days) Ollama - 28,232 Lama.cpp - 7,826
Ollama isn't an inference engine, its a GUI slapped onto a perpetually out-of-date vendored copy of Llama.cpp underneath.
So, if you're trying to actually count LLama.cpp downloads, you'd combine those two. Also, I imagine most users on OSX aren't using Homebrew, they're getting it directly from the GH releases, so you'd also have to count those.
Re: Show HN: We made our own inference engine for Apple Silicon
#45Earlier quoted context omitted.
Ollama is the leader isn't it? Brew stats (downloads last 30 days) Ollama - 28,232 Lama.cpp - 7,826
Ollama isn't an inference engine, its a GUI slapped onto a perpetually out-of-date vendored copy of Llama.cpp underneath. So, if you're trying to actually count LLama.cpp downloads, you'd combine those two. Also, I imagine most users on OSX aren't using Homebrew, they're getting it directly from the GH releases, so you'd also have to count those.
Actually, ollama has stopped using llama.cpp and is using ggml directly nowadays.
Re: Show HN: We made our own inference engine for Apple Silicon
#46cooollll
Re: Show HN: We made our own inference engine for Apple Silicon
#47Amazing! How was your experience using Rust on this project? I'm considering a project in an adjacent space and I'm trying to decide between Rust, C, and Zig. Rust seems a bit burdensome with its complexity compared to C and Zig. Reminds me of C++ in its complexity (although not as bad). I find it difficult to walk through and understand a complicated Rust repository. I don't have that problem with C and Zig for the…
You are confusing familiarity with intrinsic complexity. I have 20 years experience with C/C++ before switching to rust a few years ago. After the initial hurdle, it is way easier and very simple to follow.
Are you generally able to quickly understand what is going on in somebody else's codebase written in Rust? I find it quite difficult to understand other people's Rust code. Is this just a familiarity thing? I have not written anything particularly huge or complex in Rust, but I have written a few CLI utilities. With an equivalent level of Go exposure, I find it much easier to understand code written in Go, compared to code written in Rust.
I'm quite proficient in C/C++ (started coding in C/C++ in 1997) but I still have a much harder time understanding a new C++ project compared to a C project.
Re: Show HN: We made our own inference engine for Apple Silicon
#48[dead]