If you are going to run AI models locally on Ubuntu 22.04, 24.04 or 26.04, the first step is making sure the system detects the NVIDIA card correctly.
In this entry we prepare the base: install the recommended driver, reboot and check that the GPU appears with nvidia-smi.
We are not installing Ollama or downloading models yet. That will go in separate pieces so we do not mix too many things in a single entry.
Ubuntu versions
This method is intended for Ubuntu 22.04 LTS, Ubuntu 24.04 LTS and Ubuntu 26.04 LTS.
The driver part uses ubuntu-drivers, Ubuntu’s own tool for detecting and installing recommended drivers. The CUDA Toolkit part is optional and should be checked against NVIDIA’s documentation if a specific tool requires it.
Ubuntu 20.04 can still have valid installations, but I would treat it as a maintenance or compatibility case, not as the new base for preparing a local AI machine.
Check whether Ubuntu detects NVIDIA drivers
Ubuntu includes the ubuntu-drivers tool, which helps you see which driver it recommends for the machine.
sudo ubuntu-drivers list
On servers or machines intended for compute/GPU workloads, it may also be useful to list the GPGPU variant:
sudo ubuntu-drivers list --gpgpu
Install the recommended driver
The simplest option is to let Ubuntu install the recommended driver:
sudo ubuntu-drivers install
If you are preparing a machine for GPU compute, Ubuntu can also install the recommended GPGPU variant:
sudo ubuntu-drivers install --gpgpu
After installing the driver, reboot the machine:
sudo reboot
Verify the GPU with nvidia-smi
After rebooting, check that the driver is loaded and that Ubuntu sees the card:
nvidia-smi
If everything is correct, you will see a table with the GPU, the driver version and the CUDA version supported by that driver.
That CUDA value does not mean that you have already installed the full development toolkit. For running tools such as Ollama, the important point here is usually that the driver works and that the GPU appears correctly.
Do I need to install CUDA Toolkit?
It depends on what you are going to run.
For many local AI uses, such as running models with ready-made tools, having the NVIDIA driver working may be enough.
If you need to compile software, use nvcc or run tools that explicitly require CUDA Toolkit, then install the toolkit from NVIDIA’s repositories or by following the official documentation.
The package indicated by NVIDIA for Debian/Ubuntu systems is:
sudo apt install cuda-toolkit
I would not install it by habit on every machine. First I would check what the tool you are going to use actually needs.
Common problems
If nvidia-smi does not appear or fails:
command -v nvidia-smi
If the command does not exist, check that the NVIDIA utilities package was installed correctly.
If the command exists but does not detect the GPU, check:
- that the machine was rebooted after installing the driver;
- that the GPU is compatible with the installed driver;
- whether Secure Boot is blocking the kernel module from loading;
- that there is no mix of manually installed drivers and packaged drivers.
Next piece
With this, the base is ready: Ubuntu, NVIDIA driver and GPU check.
The next logical piece would be installing Ollama and testing a small model using the GPU.