Deep Neural Networks from Scratch in Zig • Hacker News Board

Re: Deep Neural Networks from Scratch in Zig

#31

Post by wyldfire » Tue, Apr 25, 2023, 5:45 PM UTC

hansvm wrote: post #29

wyldfire wrote: post #27

Earlier quoted context omitted.

> 2a. Inline assembly or inspecting the architecture before choosing the @Vector width are both options, so you can write your high-level code with that information in mine if necessary (e.g., to make bolt vector quantization work well in Zig I'm pretty sure you need to inline-assembly one of the swizzling operations). Inline assembly is great but support for intrinsics would be really valuable for Zig IMO.

You can link in intrinsics too. Again, C compatibility is high.

I could write a `pub fn main()` that consisted only of a call to a C library that implemented world_peace(). But it wouldn't change the fact that "support for intrinsics would be really valuable for Zig IMO."

Re: Deep Neural Networks from Scratch in Zig

#32

Post by andrepd » Tue, Apr 25, 2023, 9:44 PM UTC

brrrrrm wrote: post #16

I've been working on mix of ML performance and abstraction for a while and this is a breath of fresh air. Zig is truly a fascinating language, and the semantics revealed in this post are surprisingly powerful. "By hard coding the forward and backward methods at comptime we have some more comfort with the general correctness and expected errors we would receive if we passed in incorrectly shaped data at runtime." This…

What does this offer that basic dependent typing doesn't?

Re: Deep Neural Networks from Scratch in Zig

#33

Post by brrrrrm » Tue, Apr 25, 2023, 10:43 PM UTC

andrepd wrote: post #32

brrrrrm wrote: post #16

I've been working on mix of ML performance and abstraction for a while and this is a breath of fresh air. Zig is truly a fascinating language, and the semantics revealed in this post are surprisingly powerful. "By hard coding the forward and backward methods at comptime we have some more comfort with the general correctness and expected errors we would receive if we passed in incorrectly shaped data at runtime." This…

What does this offer that basic dependent typing doesn't?

nothing really, that's effectively what's achieved by comptime.

it's funny to see it written as "basic" dependent typing, considering it's an extremely complex type system to implement.

Re: Deep Neural Networks from Scratch in Zig

#34

Post by kookamamie » Wed, Apr 26, 2023, 6:56 AM UTC

remorses wrote: post #12

kookamamie wrote: post #9

> I would guess the constant memory allocation and frees in the training loop are the bottleneck No, the bottleneck would be not utilizing the idling GPU.

Using the CPU with quantized weights on GPT models makes sense, an example is llama.cpp, that’s because these models are constrained by memory bandwidth and not compute (low arithmetic density) https://github.com/ggerganov/llama.cpp

LLAMA is a LLM, very little to do with a model trying to learn MNIST. CNNs in particular gain from using a GPU (or ten), as they're optimizing weights for convolutional/spatial kernels.