For VMs:

[PCI passthrough] enables a guest to directly use physical PCI devices on the host, even if host does not have drivers for this particular device.

https://docs.oracle.com/en/virtualization/virtualbox/6.0/admin/pcipassthrough.html

For containers:

Make sure you have installed the NVIDIA driver and Docker engine for your Linux distribution. Note that you do not need to install the CUDA Toolkit on the host system, but the NVIDIA driver needs to be installed.

https://github.com/NVIDIA/nvidia-docker