CI/CD

Some options for running experiments automatically to save you a lot of time

At a high level, I've experimented with two options so far:

Fixed scale github actions (supports GPU):

First you need to install the docker nvidia runtime (from the instructions below and this repo):

Installing the NVIDIA Container Toolkit — NVIDIA Container Toolkitdocs.nvidia.com

Try running the following now on a desktop with GPUs:

docker run -it --gpus all nvidia/cuda:12.2.2-cudnn8
-runtime-ubi8 nvidia-smi

If this returned with the GPU info, the runtime has now been successfully installed!

Next, you'll need to register a runner on this machine by following the below documentation:

Adding self-hosted runners - GitHub DocsGitHub Docs

After that, you can go ahead and try running a test-ci run to use GPUs:

Unexpected error with integration github-files: Internal Server Error

Autoscaling github actions (doesn't support GPU)

The following script will setup an autoscaling kubernetes cluster on an Ubuntu instance with docker already installed.

The directions are taken from ARC here.

Unexpected error with integration github-files: Internal Server Error

This is an example CI file that makes use of the cluster that is setup above.

Unexpected error with integration github-files: Internal Server Error