Deep Neural Networks from Scratch in Zig • Hacker News Board

Re: Deep Neural Networks from Scratch in Zig

#21

Post by hansvm » Tue, Apr 25, 2023, 3:39 PM UTC

hansvm wrote: post #20

waynecochran wrote: post #19

Lacking a Linear Algebra / Tensor library is what kills performance for DNN. For example, this kind of element by element manipulation must be avoided: while (i What are the options for vector / matrix / tensor operations in zig?

1. The compiler will vectorize simple operations like that pretty well. 2. Zig has built-in @Vector types that are fixed-size data types designed to compile down to things like SIMD as efficiently as possible given that you might be asking it to do 16x operations on a CPU only supporting 8x width SIMD. You'd often write your high-level code as a runtime-known iteration count over those comptime-known vector widths. 2…

And their async/await is being redesigned IIRC, but at least the last version would make for a low overhead (in both developer time and runtime) way to parallelize any of the above. In the old version it was totally trivial to write a high performance parallel variant (knight's move and whatnot) sudoku solver, and I doubt the new version will be any worse. Not that the interior of a linear algebra operation is often the best place to apply parallelization, but if you had a good reason to do so it wouldn't be hard.

I'm not aware of anything in particular that would make multi-machine computations even slightly less painful than other languages, but maybe someone can chime in here with ideas.

Re: Deep Neural Networks from Scratch in Zig

#22

Post by brabel » Tue, Apr 25, 2023, 4:12 PM UTC

kuroguro wrote: post #8

brabel wrote: post #4

I didn't investigate this more deeply , but I think if you used an arena allocator on each iteration of the loop, and simply freed the whole allocator after each iteration, you might get much better performance. EDIT: the MNIST link requires authentication: http://yann.lecun.com/exdb/mnist/ How can we get access to the test run at the end?

> the MNIST link requires authentication doesn't ask for auth for me?

It's working now! I guess someone opened access after I checked.

Re: Deep Neural Networks from Scratch in Zig

#23

Post by krastanov » Tue, Apr 25, 2023, 4:13 PM UTC

waynecochran wrote: post #19

Lacking a Linear Algebra / Tensor library is what kills performance for DNN. For example, this kind of element by element manipulation must be avoided: while (i What are the options for vector / matrix / tensor operations in zig?

To echo the sibling, while this should be avoided in Python, languages like Zig, C++, Julia, Rust can expect the compiler to SIMD-ify these expressions.

Re: Deep Neural Networks from Scratch in Zig

#24

Post by brundolf » Tue, Apr 25, 2023, 4:18 PM UTC

waynecochran wrote: post #19

Lacking a Linear Algebra / Tensor library is what kills performance for DNN. For example, this kind of element by element manipulation must be avoided: while (i What are the options for vector / matrix / tensor operations in zig?

The goal of the post is to de-mystify, not to make something production-ready. Doing everything in plain code without any magic libraries helps people understand what's happening

Re: Deep Neural Networks from Scratch in Zig

#25

Post by Freire_Herval » Tue, Apr 25, 2023, 4:18 PM UTC

waynecochran wrote: post #19

Lacking a Linear Algebra / Tensor library is what kills performance for DNN. For example, this kind of element by element manipulation must be avoided: while (i What are the options for vector / matrix / tensor operations in zig?

You need to move these sections into the GPU. Neither zig, rust, C++ nor C can do that alone. You'll need to use cuda or futhark. That's the real speedup for these types of things.

Optimization in this case largely concerns memory management in the GPU and keeping data transfer between cpu to gpu at a minimum.

Essentially a massively parallel API combined with a massively parallel processor. I'm thinking of doing an end to end tutorial about this.

Re: Deep Neural Networks from Scratch in Zig

#26

Post by dundarious » Tue, Apr 25, 2023, 4:29 PM UTC

krastanov wrote: post #23

waynecochran wrote: post #19

Lacking a Linear Algebra / Tensor library is what kills performance for DNN. For example, this kind of element by element manipulation must be avoided: while (i What are the options for vector / matrix / tensor operations in zig?

To echo the sibling, while this should be avoided in Python, languages like Zig, C++, Julia, Rust can expect the compiler to SIMD-ify these expressions.

Somewhat, but I think people vastly over-estimate their ability.

A common example is if there's any accumulation/reduction, compilers will almost entirely fail to generate SIMD unless you use -funsafe-math-optimizations type flags, because of non-associativity of floating point. Sum of squares is the classic example (not saying that specific operation is used in NN).

Explicit vectorization (e.g., using intrinsics) is almost always a relatively simple way to get orders of magnitude speedup compared to auto-vectorization, because of the above. Also because data layouts usually need to change as well (AoS vs SoA, etc.), though NN people seem to write decent data layouts.

I don't have any experience with `#pragma omp` type approaches which may be a middle ground.

Re: Deep Neural Networks from Scratch in Zig

#27

Post by wyldfire » Tue, Apr 25, 2023, 4:35 PM UTC

hansvm wrote: post #20

waynecochran wrote: post #19

Lacking a Linear Algebra / Tensor library is what kills performance for DNN. For example, this kind of element by element manipulation must be avoided: while (i What are the options for vector / matrix / tensor operations in zig?

1. The compiler will vectorize simple operations like that pretty well. 2. Zig has built-in @Vector types that are fixed-size data types designed to compile down to things like SIMD as efficiently as possible given that you might be asking it to do 16x operations on a CPU only supporting 8x width SIMD. You'd often write your high-level code as a runtime-known iteration count over those comptime-known vector widths. 2…

> 2a. Inline assembly or inspecting the architecture before choosing the @Vector width are both options, so you can write your high-level code with that information in mine if necessary (e.g., to make bolt vector quantization work well in Zig I'm pretty sure you need to inline-assembly one of the swizzling operations).

Inline assembly is great but support for intrinsics would be really valuable for Zig IMO.

Re: Deep Neural Networks from Scratch in Zig

#28

Post by waynecochran » Tue, Apr 25, 2023, 4:39 PM UTC

hansvm wrote: post #21

hansvm wrote: post #20

Earlier quoted context omitted.

1. The compiler will vectorize simple operations like that pretty well. 2. Zig has built-in @Vector types that are fixed-size data types designed to compile down to things like SIMD as efficiently as possible given that you might be asking it to do 16x operations on a CPU only supporting 8x width SIMD. You'd often write your high-level code as a runtime-known iteration count over those comptime-known vector widths. 2…

And their async/await is being redesigned IIRC, but at least the last version would make for a low overhead (in both developer time and runtime) way to parallelize any of the above. In the old version it was totally trivial to write a high performance parallel variant (knight's move and whatnot) sudoku solver, and I doubt the new version will be any worse. Not that the interior of a linear algebra operation is often…

C++ / Eigen performs a lot of optimizations that you might expect from a Fortran compiler. For example, it ability to broadcast and perform reductions is pretty slick, e.g.:

        X.noalias() = (A - B).colwise().squaredNorm().mean();

Short of having tensor cores, this does a good job of static optimization for underlying vectorization support.

Re: Deep Neural Networks from Scratch in Zig

#29

Post by hansvm » Tue, Apr 25, 2023, 4:46 PM UTC

wyldfire wrote: post #27

hansvm wrote: post #20

Earlier quoted context omitted.

1. The compiler will vectorize simple operations like that pretty well. 2. Zig has built-in @Vector types that are fixed-size data types designed to compile down to things like SIMD as efficiently as possible given that you might be asking it to do 16x operations on a CPU only supporting 8x width SIMD. You'd often write your high-level code as a runtime-known iteration count over those comptime-known vector widths. 2…

> 2a. Inline assembly or inspecting the architecture before choosing the @Vector width are both options, so you can write your high-level code with that information in mine if necessary (e.g., to make bolt vector quantization work well in Zig I'm pretty sure you need to inline-assembly one of the swizzling operations). Inline assembly is great but support for intrinsics would be really valuable for Zig IMO.

You can link in intrinsics too. Again, C compatibility is high.

Re: Deep Neural Networks from Scratch in Zig

#30

Post by dundarious » Tue, Apr 25, 2023, 4:47 PM UTC

brabel wrote: post #6

camel-cdr wrote: post #5

Earlier quoted context omitted.

Why would any DNN ever need to deallocate other than at program termination? (And allocate other than at startup)

That's the point. With an arena allocator, you don't "really" de-allocate the OS memory, you just free the bytes to be re-used again in the next iteration (by using a FixedBufferAllocator as the backing allocator, if you know how much memory will be needed in advance). This is much better than trying to re-use the structs manually as the OP mentioned at the end. EDIT: FixedBufferAllocator has a reset method , so mayb…

FixedBufferAllocator is just a "bump allocator", i.e., just a size and a pointer that moves forward. You allocate the memory it uses from somewhere else (stack, a big alloc from a general purpose allocator (think malloc), some pages from the OS via mmap or VirtualAlloc, etc. -- zig has simple cross platform functions for all of these).

Individual "allocations" just bump the pointer forward, and it succeeds unless the result is past the end of the fixed size given to it initially. You don't free any individual allocations (except maybe the most recent one by just moving the pointer back again), you just reset the pointer to the very start when you're done with all the allocations.