CI/CD

Some options for running experiments automatically to save you a lot of time

At a high level, I've experimented with two options so far:

Fixed scale github actions (supports GPU):

First you need to install the docker nvidia runtime (from the instructions below and this repo):

Installing the NVIDIA Container Toolkit — NVIDIA Container Toolkitdocs.nvidia.com

Try running the following now on a desktop with GPUs:

docker run -it --gpus all nvidia/cuda:12.2.2-cudnn8
-runtime-ubi8 nvidia-smi

If this returned with the GPU info, the runtime has now been successfully installed!

Next, you'll need to register a runner on this machine by following the below documentation:

Adding self-hosted runners - GitHub DocsGitHub Docs

After that, you can go ahead and try running a test-ci run to use GPUs:

https://github.com/ezhang7423/research_project/blob/main/.github/workflows/gpu-ci.yml

name: Run on <GPU-ENABLED-RUNTIME>

on:
  push:


jobs:
  simple-hello:
    runs-on: <GPU-ENABLED-RUNTIME>
    container:
      image: pytorch/pytorch:2.1.0-cuda12.1-cudnn8-runtime
      options:
        --gpus all
    steps:
      - name: Check runtime env
        run: |
          echo "Running on ${{ runner.os }}"
          pwd -P
          whoami

      - name: Check GPU
        run: |
          nvidia-smi


  

Autoscaling github actions (doesn't support GPU)

The following script will setup an autoscaling kubernetes cluster on an Ubuntu instance with docker already installed.

The directions are taken from ARC here.

https://github.com/ezhang7423/research_project/blob/main/scripts/setup-autoscaling.bash

#!/bin/bash

NAMESPACE="arc-systems"
INSTALLATION_NAME="arc-runner-set"  #! this will be whatever is used for runs-on: in the CI file
NAMESPACE="arc-runners"
GITHUB_CONFIG_URL="https://github.com/<your_enterprise/org/repo>" #! can be set to repo, org, or enterprise
GITHUB_PAT="<PAT>" #! this is the PAT that has access to the repo, org, or enterprise


# This script will create an autoscaling group and launch configuration on Ubuntu with docker already installed
# Install minikube

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo install minikube-linux-amd64 /usr/local/bin/minikube

# Install kubectl
sudo snap install kubectl --classic

# Install helm
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash


helm install arc \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller


helm install "${INSTALLATION_NAME}" \
    --namespace "${NAMESPACE}" \
    --create-namespace \
    --set githubConfigUrl="${GITHUB_CONFIG_URL}" \
    --set githubConfigSecret.github_token="${GITHUB_PAT}" \
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set

This is an example CI file that makes use of the cluster that is setup above.

https://github.com/ezhang7423/research_project/blob/main/.github/workflows/autoscaling-ci.yml

name: autoscaling-ci
on:
  workflow_dispatch:
    inputs:
      number-nodes:
        description: 'Number of nodes'
        required: true
        default: 4
  # push:

env:
  number-nodes: 4

jobs:
  prejob:
    runs-on: arc-gke
    outputs:
      node_matrix: ${{ steps.set-matrix.outputs.node_matrix }}
    steps:
      - name: Python setup
        uses: actions/setup-python@v3
        with:
          python-version: 3.10

      - name: Use python to convert number-nodes to a list
        id: set-matrix
        run: |
          matrix_value=$(python -c "print(str(list(range(1, ${{ github.event.inputs.number-nodes || env.number-nodes }} + 1))).replace(' ', ''))")
          echo "node_matrix=$matrix_value" >> $GITHUB_OUTPUT

  test:
    runs-on: arc-gke    
    needs: prejob
    strategy:
      fail-fast: true
      matrix:
        node: ${{ fromJson(needs.prejob.outputs.node_matrix) }}
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      - name: Run tests
        run: |
          echo "Running tests on node ${{ matrix.node }}"