Llama2.java: Karpathy's llama2.c ported to Java

#1

Post by mukel » Tue, Aug 08, 2023, 10:51 AM UTC

Llama2.java: Karpathy's llama2.c ported to Java
github.com

Re: Llama2.java: Karpathy's llama2.c ported to Java

#2

Post by mukel » Tue, Aug 08, 2023, 10:51 AM UTC

A Java port of llama2.c that performs very close to C on large models. Llama 2 7B runs at a whooping 1.6 tokens/s.

Re: Llama2.java: Karpathy's llama2.c ported to Java

#3

Post by shortrounddev2 » Tue, Aug 08, 2023, 2:09 PM UTC

How you all used these things for anything useful? I can't get them to give useful results on my 3060 8gb. If I wanted to get decent results I think I'd need to rent a GPU node somewhere, but chatGPT is still free

Re: Llama2.java: Karpathy's llama2.c ported to Java

#4

Post by SushiHippie » Tue, Aug 08, 2023, 2:11 PM UTC

shortrounddev2 wrote: post #3

How you all used these things for anything useful? I can't get them to give useful results on my 3060 8gb. If I wanted to get decent results I think I'd need to rent a GPU node somewhere, but chatGPT is still free

The 4bit quantized 13B models, give really decent answers (not as good as gpt4, but often as good as gpt 3)

Re: Llama2.java: Karpathy's llama2.c ported to Java

#5

Post by gavinray » Tue, Aug 08, 2023, 3:02 PM UTC

The Java code is impressively written, using newer features like MemorySegment.

Looked at the author and realized it's Alfonso from the Graal team -- makes sense.

I wonder whether the "matmul" code could be further optimized with the Vector API and SIMD.

Re: Llama2.java: Karpathy's llama2.c ported to Java

#6

Post by nmfisher » Tue, Aug 08, 2023, 3:08 PM UTC

shortrounddev2 wrote: post #3

How you all used these things for anything useful? I can't get them to give useful results on my 3060 8gb. If I wanted to get decent results I think I'd need to rent a GPU node somewhere, but chatGPT is still free

I know it might be asking a lot, but it would be great if someone could put up a HF space so I could try all the various flavours/sizes.

Re: Llama2.java: Karpathy's llama2.c ported to Java

#7

Post by kurhan » Tue, Aug 08, 2023, 4:42 PM UTC

gavinray wrote: post #5

The Java code is impressively written, using newer features like MemorySegment. Looked at the author and realized it's Alfonso from the Graal team -- makes sense. I wonder whether the "matmul" code could be further optimized with the Vector API and SIMD.

Also new virtual threads might be beneficial. I was experimenting using Vector api for matrix multiplication once and effect was pretty good.

Re: Llama2.java: Karpathy's llama2.c ported to Java

#8

Post by lazylion2 » Tue, Aug 08, 2023, 6:28 PM UTC

nmfisher wrote: post #6

shortrounddev2 wrote: post #3

How you all used these things for anything useful? I can't get them to give useful results on my 3060 8gb. If I wanted to get decent results I think I'd need to rent a GPU node somewhere, but chatGPT is still free

I know it might be asking a lot, but it would be great if someone could put up a HF space so I could try all the various flavours/sizes.

/r/LocalLLaMA/

Re: Llama2.java: Karpathy's llama2.c ported to Java

#9

Post by jiehong » Tue, Aug 08, 2023, 6:38 PM UTC

This makes me wonder: what’s the status of GPU programming on the JVM?

Any abstraction for GPGPU or shaders programming?

Re: Llama2.java: Karpathy's llama2.c ported to Java

#10

Post by mike_hearn » Tue, Aug 08, 2023, 7:08 PM UTC

kurhan wrote: post #7

gavinray wrote: post #5

The Java code is impressively written, using newer features like MemorySegment. Looked at the author and realized it's Alfonso from the Graal team -- makes sense. I wonder whether the "matmul" code could be further optimized with the Vector API and SIMD.

Also new virtual threads might be beneficial. I was experimenting using Vector api for matrix multiplication once and effect was pretty good.

Virtual threads shouldn't help as the program isn't I/O or wait bottlenecked. It's a pure computation, so it's all about vectorization here.