nvidia/cuda:12.3.1-devel-ubuntu22.04 not working on A10, 3090 and 1080 nodes
User reported that their image based on nvidia/cuda:12.3.1-devel-ubuntu22.04 can run on A100 and A6000 nodes, but not A10, 3090 and 1080 nodes. The error message is
Warning Failed 2m33s (x4 over 3m20s) kubelet Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running hook #0: error running hook: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: requirement error: unsatisfied condition: cuda>=12.3, please update your driver to a newer version, or use an earlier cuda container: unknown
The error message suggests to update driver, but driver 535 does not work on A10 nodes.