Hacker Read

abraham_lincoln · 2018-09-20 18:40:10

I did manage to setup the notebook and cuda on my local machine. I have an older GTX card.

rain1 | karma 1002 | avg karma 6.15 · | 2023-05-14 12:31:28

I don't think the integrated GPU on that supports CUDA. So you will need to use CPU mode only.

m45t3r | karma 1652 | avg karma 3.23 · | 2017-11-15 17:36:55+00:00

Yeah, you can actually do it. Just run desktop in your integrated Intel GPU and CUDA in NVIDIA discrete GPU.

vince_liu | karma 6 | avg karma 0.29 · | 2023-08-12 15:31:37

out of interest which OS are you using? And do you have CUDA gpu?

wtcactus | karma 477 | avg karma 2.91 · | 2023-03-08 11:02:43

I'm having a hard time figuring out if I can simply install NVIDIA proprietary drivers and CUDA on Vanilla OS.

dogma1138 | karma 17831 | avg karma 2.04 · | 2017-01-15 14:57:24

Yeah I meant to write NVIDIA not CUDA, FYI you need to login to download stuff from NVIDIA, try copying the URL with the auth parameter ;)

I've made a post about how VP9 is offloaded to GPU in this thread, it's done via DXVA/MF on Windows.

reply

Iolaum | karma 1416 | avg karma 4.21 · | 2019-06-28 13:41:54

Do you have any documentation, or even random notes, on how you set that up? I have an optimus laptop and couldn't make the proprietary drivers work ( tried both rpmfusion and negativo17). I m happily using nouveau for now but at some point I ll need to use CUDA again.

jart | karma 12123 | avg karma 5.27 · | 2023-12-13 12:48:31

Yes you need to install CUDA and MSVC for GPU. But here's some good news! We just rolled our own GEMM functions so llamafile doesn't have to depend on cuBLAS anymore. That means llamafile 0.4 (which I'm shipping today) will have GPU on Windows that works out of the box, since not depending on cuBLAS anymore means that I'm able to compile a distributable DLL that only depends on KERNEL32.DLL. Oh it'll also have Mixtral support :) https://github.com/Mozilla-Ocho/llamafile/pull/82

minimaxir | karma 67739 | avg karma 7.48 · | 2017-02-21 21:30:08+00:00

Looking at the docs, it appears that the CUDA drivers have to be manually installed on the host image, which takes time/money.

Is there a GCP Image with CUDA drivers pre-installed? Or is that not possible with the hardware architecture?

reply

Rickvst | karma 16 | avg karma 1.6 · | 2023-05-10 09:03:05

In my case it was easy to setup, but I didn't figured out how to change from using my iGPU to the GPU. Unfortunetly on the iGPU it is pretty slow.

natch | karma 6856 | avg karma 1.47 · | 2020-03-20 16:19:26+00:00

I wish they would spend five minutes documenting how to use the GPU on Ubuntu. My 1080ti is just sitting idle while my CPU is busy folding. Any instructions I came across said something like “make sure you have the libraries” but then failed to describe even at a high level how to locate and install those libraries. Last time I installed any CUDA libraries it involved adding an Nvidia repo or something.

Edit: I’d be glad to be proven wrong with a link to an FAQ or some part of the docs.

reply

stavros | karma 66636 | avg karma 10.05 · | 2023-08-05 07:01:21

I have an nVidia card and I never managed to get any AI stuff working on my base system. The only thing that's worked off the box is Docker images with CUDA support.

neves | karma 3406 | avg karma 2.48 · | 2018-01-29 18:05:07+00:00

Do you have good references about how to setup the environment with a Nvidia GPU in a Windows machine?

mayeaux | karma 381 | avg karma 3.37 · | 2022-11-18 18:35:26

Hey, I know the feeling, I felt bad when I had my GPU just sitting there and it's just a little Vast server lol. If you want to use your hardware to run this software I'd be more than happy to help get it setup!

arp242 | karma 17029 | avg karma 3.74 · | 2023-11-28 11:37:52

Is that needed? Can't you run the GUI (Wayland, or X11) with Nouveau and CUDA with the nVidia driver?

Steltek | karma 3719 | avg karma 3.32 · | 2021-07-17 11:31:52

An important prerequisite was left out: Meshroom requires an Nvidia card for CUDA. Sadly, I swapped out my Nvidia card for an AMD one so I could switch to Sway/Wayland. I've contemplated putting both cards in, AMD for the desktop and the Nvidia for CUDA, but I haven't got around to it. I need my desktop intact for WFH duties.

renewiltord | karma 12072 | avg karma 1.39 · | 2024-04-12 16:03:32

Thank you! Can I run with 2 GPUs or with heterogeneous GPUs that have same RAM? I will try. Just curious if you already have tried.

petercooper | karma 48277 | avg karma 5.12 · | 2024-02-20 18:48:35

I got it after a few hours. You need to install drivers/CUDA yourself, but all very straightforward. Unfortunately due to having 20GB of VRAM, I'm limited to mixtral:8x7b-instruct-v0.1-q2_K but it runs fine, generating at about 40 tokens/s (65 tok/s for eval). As per official specs, it's running maxed out at 70W (being an SFF card).

(I've now tried running the Q4 mixtral which is 26GB. 18GB is on GPU, 8GB through CPU. Gets about 11 tok/s.)

reply

brobinson | karma 2225 | avg karma 2.3 · | 2022-04-20 07:27:20

Are you able to run X/Wayland off of the Intel GPU and leave the Nvidia GPU free for CUDA usage on that setup?

john2x | karma 610 | avg karma 1.89 · | 2023-07-06 16:52:05

Care to share some links? My lack of GPU is the main blocker for me from playing with local-only options.

I have an old laptop with 16GB RAM and no GPU. Can I run these models?

reply