AMD GPU Linux

[linuxreitt]

I'm having difficulty understanding if it is possible to install lollms-webui on linux with amd gpu and rocm etc. I am aware that certain bindings work with amd gpu,however I am unclear on how to make it all work together as on the settings screen AMD GPU is not detected and when I install the bindings they all install specific to nvidia gpu. Can anyone help me? Thank you.

[ParisNeo]

try gpt4all binding

[linuxreitt]

Hi, thanks for your reply. I have managed to get it working now, using exllamaV2 installing a rocm whl manually. My issue now is that when I send a message the response is generated quickly, however the response is not printed unless I click stop. It seems that the tts extension (warming up) is delaying the reply until is has completed, which for me is extremely slow.

My hardware is AMD RX6800XT with alderlake 12600KF, AsRock Z690 DDR5. I can't seem to find a way of disabling the bound_extensions.

I have deselected the auto input and output in audio settings but it still says "warming up" and won't print the output until complete.

It would be great to know if 1) possible to speed up the audio processing for my hardware and 2) how I can disable any or all of the bound extensions. Thank you

[ParisNeo]

Hi, and thanks for sharing this. Well, the audio starts talking as soon as a sentence is received by the Webui. It is not yet generated by the backend. I wonder if the wheel you installed has some modifications that makes streaming not working!

In LoLLMs exllamaV2 I activate stream mode which allows me to read the outputs one by one and then show it.
I only have an nvidia GPU so I didn't test the recm solution yet. I'll try to see of I can ask someone to test for me and report the output to identify what's wrong on Rocm enabled devices.

Can you share the Rocm wheel address. I can add that to the installer so that other people can use it from the start.

[linuxreitt]

I did also try to use exllama v1 as I have used in the past but couldn't quite figure out how to re-implement it back into the latest lollms-webui successfully.

[linuxreitt]

Videos.zip

Hi, I have taken 2 screen recordings to show what I mean, I'm not the best at explaining things! You will see from lollms_1 video it takes some time to run until outputting, in lollms_2 you will see what happens when I stop the generation and it prints the output. I have included my terminal windows so that you can see the token generation etc.

For the exllamav2 I downloaded from the following link: https://github.com/turboderp/exllamav2/releases/download/v0.0.7/exllamav2-0.0.7+rocm5.6-cp310-cp310-linux_x86_64.whl

Thank you