Docker, Nvidia Container 설치

유의사항

https://realblack0.github.io/2020/03/23/docker-for-gpu.html

• linux에서만 지원되며, windows는 docker container에서 gpu 사용이 불가능하다고 합니다.
• nvidia-docker2 패키지는 depreciated 될 예정 입니다.
• Docker 버전은 반드시 19.03 이후 버전이어야 합니다.

최신 Docker는 nvidia-docker를 별도 설치할 필요 없이 지원이 된다고 합니다.

 

진행 환경

• ubuntu 20.04
• cuda 11.7
• nvidia driver 510

Docker 설치

https://docs.docker.com/engine/install/ubuntu/

https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-on-ubuntu-20-04

• First, update your existing list of packages:

sudo apt update

• Next, install a few prerequisite packages which let apt use packages over HTTPS:

sudo apt install apt-transport-https ca-certificates curl software-properties-common

• Then add the GPG key for the official Docker repository to your system:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

• Add the Docker repository to APT sources:
  This will also update our package database with the Docker packages from the newly added repo.

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"

• Make sure you are about to install from the Docker repo instead of the default Ubuntu repo:

 

apt-cache policy docker-ce

• Finally, install Docker:

sudo apt install docker-ce

NVIDA Container toolkit 설치

https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html#docker

• Setup the package repository and the GPG key:

distribution=$(. /etc/os-release;echo $ID$VERSION_ID) \
      && curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
      && curl -s -L https://nvidia.github.io/libnvidia-container/$distribution/libnvidia-container.list | \
            sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
            sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

• Install the nvidia-container-toolkit package (and dependencies) after updating the package listing:

sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit

• Configure the Docker daemon to recognize the NVIDIA Container Runtime

sudo nvidia-ctk runtime configure --runtime=docker

• Restart the Docker daemon to complete the installation after setting the default runtime

sudo systemctl restart docker

• At this point, a working setup can be tested by running a base CUDA container:

sudo docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

설치 확인

• Bash 실행

sudo docker run -it --name test-cuda --gpus all nvidia/cuda:11.2.2-devel-ubuntu20.04 bash

Error 발생 시

• Error

nvcc fatal : No input files specified; use option --help for more informationE: Conflicting values set for option Signed-By regarding source https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64/ /: /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg !=

 

패스 설정 후 진행

export PATH="/usr/local/cuda-11.2/bin/:$PATH"ls /etc/apt/sources.list.d/
export LD_LIBRARY_PATH="/usr/local/cuda-11.2/lib64:$LD_LIBRARY_PATH"
source ~/.bashrc

 

• Error

E: Conflicting values set for option Signed-By regarding source https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64/ /: /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg != E: The list of sources could not be read.

 

https://stackoverflow.com/questions/42410134/how-to-fix-e-the-list-of-sources-could-not-be-read-could-not-be-read-error

아래의 경로에서 문제가 발생한 nvidia-container 제거

ls /etc/apt/sources.list.d/

 

• Error
  • 해당 버전을 찾을 수 없다는 에러

docker: Error response from daemon: pull access denied for 11.8.0-base-ubuntu22.04, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.E: Conflicting values set for option Signed-By regarding source https://nvidia.github.io/libnvidia-container/stable/ubuntu18.04/amd64/ /: /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg !=
E: The list of sources could not be read.

 

Docker Hub에서 다른 버전을 찾아 수행해줍니다.

 

• Error
  • 정말 많은 이유로 발생할 수 있는 에러인데, 한가지 고려할만한 점으로 GPU가 안켜져있어서 일 수 있다.

docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed

https://github.com/NVIDIA/nvidia-docker/issues/1648

https://www.ddengle.com/board_FAQ/2719454

 

• permission mode 켜기

sudo nvidia-smi -pm 1