[Enhancement] Add the Ability to Use NVIDIA for Docker

[ZaxLofful]

Allow large language models with graphics card with large RAM.

https://github.com/NVIDIA/nvidia-docker

[alph4b3th]

Yes please! the model is very slow on amd epyc cpu..

[fishscene]

I too would like to see this capability.
Another person here also looking for this: https://github.com/nsarrazin/serge/discussions/97

Looks like it is possible with this project: https://github.com/ggerganov/llama.cpp
Appears to be some kind of CPU/GPU sharing: https://www.reddit.com/r/LocalLLaMA/comments/13gok03/llamacpp_now_officially_supports_gpu_acceleration/

[SurvivaLlama]

I am willing to test GPU / Nvidia / Docker

[lucienn3]

Hey guys,

Please reach out to me if you want to test it.
I have an i9-13900KF, 64 GB DDR5 RAM, and an RTX4090. I would love to help with testing :)

[gaby]

This is in the TODO list, we have to split the Dockerfiles first though. Having GPU support will make the image huge. The plan is to publish separate images for GPU/Non-GPU support

[realies]

@gaby, any idea when could these be expected? Happy to test too.

[Jonpro03]

I got it running. Feel free to borrow: https://gist.github.com/Jonpro03/604430a3e64735a0a9df6b7e385d15be

[gaby]

I got it running. Feel free to borrow: https://gist.github.com/Jonpro03/604430a3e64735a0a9df6b7e385d15be

Ideally during runtime the image should be "-runtime" not "-devel". My plan is to make a compose that use the dockerfile for the final stage

[gaby]

@gaby, any idea when could these be expected? Happy to test too.

This week

[Jonpro03]

I got it running. Feel free to borrow: https://gist.github.com/Jonpro03/604430a3e64735a0a9df6b7e385d15be

Ideally during runtime the image should be "-runtime" not "-devel". My plan is to make a compose that use the dockerfile for the final stage

Groovy. Can confirm it works with the runtime image too.

[gaby]

@Jonpro03 llama-cpp-python compiles correctly even using the "-runtime" tag?

[gaby]

I see that you are adding CMAKE flags, not sure what those do, haha

[Jonpro03]

I see that you are adding CMAKE flags, not sure what those do, haha

Possibly a me problem, since I'm running older CPUs (Xeon E5-2640's), but these flags were necessary to get it to compile with the pip.
-DLLAMA_CUBLAS=ON -DLLAMA_NATIVE=ON should be all that's necessary, but wasn't in my case (I suspect a lama-cpp bug) ggerganov/llama.cpp#1982

[gaby]

I see that you are adding CMAKE flags, not sure what those do, haha
Possibly a me problem, since I'm running older CPUs (Xeon E5-2640's), but these flags were necessary to get it to compile with the pip.
-DLLAMA_CUBLAS=ON -DLLAMA_NATIVE=ON should be all that's necessary, but wasn't in my case (I suspect a lama-cpp bug) ggerganov/llama.cpp#1982

Thanks for the info!

[Jonpro03]

@Jonpro03 llama-cpp-python compiles correctly even using the "-runtime" tag?

Oops, I spoke too soon. It only works with the devel tag.

[gaby]

@Jonpro03 llama-cpp-python compiles correctly even using the "-runtime" tag?

Oops, I spoke too soon. It only works with the devel tag.

Yeah, that's what i figured. The devel tag is needed to compile the llama-cpp-python wheel. After that the runtime tag can be used.

My main problem has been that the devel tag is like 6.5GB

[visorcraft]

We're also willing to test Nvidia support, have 2x RTX 3090 w/nvlink set up on Pop!_OS and eager see Nvidia support in Serge!

[anethema]

@Jonpro03 llama-cpp-python compiles correctly even using the "-runtime" tag?

Oops, I spoke too soon. It only works with the devel tag.

Yeah, that's what i figured. The devel tag is needed to compile the llama-cpp-python wheel. After that the runtime tag can be used.

My main problem has been that the devel tag is like 6.5GB

Any progress? I would love to be able to run a local LLM on unraid with GPU acceleration

[gaby]

@Jonpro03 llama-cpp-python compiles correctly even using the "-runtime" tag?

Oops, I spoke too soon. It only works with the devel tag.

Yeah, that's what i figured. The devel tag is needed to compile the llama-cpp-python wheel. After that the runtime tag can be used.
My main problem has been that the devel tag is like 6.5GB

Any progress? I would love to be able to run a local LLM on unraid with GPU acceleration

I'm on vacation, will get this done in like 2 weeks. :-)

[creed2415]

I just found no use of gpu when I started a chat ( 30 GPU Layers) ,Should I install CUDA inside Docker?

Below are my proposed modifications:
step1: set the LLAMA_CUBLAS=1 environment variable

# Set ENV
ENV NODE_ENV='production'
ENV TZ=Etc/UTC
ENV LLAMA_CUBLAS=1
WORKDIR /usr/src/app

step2: replace the installation method for "llama-cpp-python" in the "/scripts/deploy.sh" file

# Install python bindings
UNAME_M=$(dpkg --print-architecture) CMAKE_ARGS="-DLLAMA_CUBLAS=on" pip install llama-cpp-python || {
	echo 'Failed to install llama-cpp-python'
	exit 1
}

step3:docker build

step4:install nvidia-container-toolkit

step5:start docker

[SurvivaLlama]

@creed2415 Did you get GPU to work?

[syonfox]

Cool stuff does anyone have numbers on the infrance speed on gpu vs cpu using this

[creed2415]

@creed2415 Did you get GPU to work?

yes! After I installed CUDA inside Docker,it worked with gpu.

[TechnoKittyKatyusha]

@syonfox

well, the GPU Creed showed has 3840 Cuda-cores compared to the 24 cores/32Threads a current consumer Intel CPU provides.
sooooooo, i would assume, depending on creeds CPU model/ number of CPUs, that its a moderate to huge boost to the performance.
i didnt manage to get it to run using my GPU, so im jealous XD

[anethema]

@creed2415 did you do this in unraid by chance? Not sure how to actually make those edits to the package since it comes from the unraid community applications.

@gaby any tips on how to make that happen? Or is that something you still planned on ?

[gaby]

@anethema I have it ready, but I need to merge and test #866 first :-)

[anethema]

@anethema I have it ready, but I need to merge and test #866 first :-)

Awesome! Exciting to see.

[creed2415]

@gaby If possible, I would like to submit my solution (linux for nvidia) to the repository.

[gaby]

@creed2415 Serge has to work for both gpu/non-gpu, so images have to be separate, etc. So, it's a bit more complicated than just adding nvidia-toolkit. If you got a solution that works for everyone, sure.

[creed2415]

@gaby well,here are the adjustments I made. just take a look and let me know if they are useful.
https://github.com/creed2415/serge/tree/gpu-support

[mkeshav]

any news on this pls?

[gaby]

@mkeshav This week :-)

[syonfox]

Woh knows the best nvidia gpu provider iaas XD looking forward to it!

ps maybe harddware really is worth it ! but need med scale ;

[anethema]

@mkeshav This week :-)

Any luck with this?

[gaby]

@mkeshav This week :-)

Any luck with this?

Still in progress

[iamzoltan]

Can I help?

[TheQuickestFox]

@gaby looks like #944 passed checks, can I get excited? thanks so much for the amazing work on the rest of this by the way - by far the most user friendly AI interface for local LLMs on Unraid.

[gaby]

@TheQuickestFox Not yet, i'm helping the llama-cpp-python folks tests their new wheels to make this more straight forward.

[JuniperChris929]

any ETA on this? and also on the latest IPv6 fixes? The last release is from Feb :(

[gaby]

@JuniperChris929 Within the next 2 weeks. The main library used by Serge now has built-in support for CUDA Python wheels which will streamline our approach to support multiple platforms.

Even though last release was in February, there's over 100 commits between then and now.

[CodeMazeSolver]

Are there any updates regarding Serge using GPU acceleration?

[VendorAttestation]

@gaby is this gunna happen?

[Imperial929]

Any news here? We keep hearing over and over again "within a week or two" but the last release is from February and neither CUDA SUpport is easily working nor support for IPv6-only :(