mirror of
https://github.com/rcourtman/Pulse.git
synced 2026-09-11 14:00:29 +00:00
rcourtman
2305131d8a
Self-hosted runners attached to a public repo are a known
severe-risk pattern: any workflow that gains the ability to run
PR-controlled code escalates to RCE on the runner host. Even
though this workflow is workflow_dispatch only today, the runner
itself sits one workflow edit away from arbitrary code execution
on whatever machine it's pinned to.
Changes:
- runs-on: self-hosted -> ubuntu-24.04. Reachability of base_url
from the GitHub-hosted runner is now the dispatcher's
responsibility. Add tailscale/* steps (like deploy-v6-preview-
demo.yml or deploy-demo-server.yml already do) if pointing at
a private Pulse instance.
- Drop the admin/admin fallback on PULSE_EVAL_USER and
PULSE_EVAL_PASS. The previous default was
${{ secrets.PULSE_EVAL_USER || 'admin' }} which silently ran
the eval as admin/admin if the secrets were missing. Added a
fail-closed precheck that refuses to proceed unless both
secrets are set.
- Route inputs.scenario, inputs.models, inputs.providers, and
inputs.base_url through env: indirection instead of inline
${{ }} interpolation into shell. Defense-in-depth against
script injection from the dispatch payload.
Net result: kills the audit's only CRITICAL finding and the
companion MEDIUM (admin/admin fallback). Eval can still run
exactly as before once secrets are set and base_url is reachable.
89 lines
3.1 KiB
YAML
89 lines
3.1 KiB
YAML
name: Pulse AI Model Matrix
|
|
|
|
on:
|
|
workflow_dispatch:
|
|
inputs:
|
|
scenario:
|
|
description: Scenario or collection to run (e.g. matrix, smoke, readonly, advanced)
|
|
required: true
|
|
default: matrix
|
|
models:
|
|
description: Comma-separated model list (e.g. gpt-4.1-mini,claude-3-5-sonnet,gemini-1.5-pro,ollama:llama3.1)
|
|
required: false
|
|
default: ""
|
|
providers:
|
|
description: Optional provider filter (e.g. openai,anthropic,gemini,ollama)
|
|
required: false
|
|
default: ""
|
|
base_url:
|
|
description: Pulse API base URL reachable from the GitHub-hosted runner. Add Tailscale or another reachability layer to the workflow if pointing at a private instance.
|
|
required: true
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
jobs:
|
|
eval:
|
|
name: Model Matrix Eval
|
|
# Moved off self-hosted: a self-hosted runner attached to a public repo
|
|
# is RCE-equivalent on the runner host as soon as one workflow gets
|
|
# mis-configured to run untrusted PR code. Reachability to the eval
|
|
# target is now the dispatcher's responsibility (pass a base_url the
|
|
# runner can reach, or add tailscale/* steps before this job).
|
|
runs-on: ubuntu-24.04
|
|
timeout-minutes: 60
|
|
|
|
steps:
|
|
- name: Checkout code
|
|
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
|
|
|
- name: Set up Go
|
|
uses: actions/setup-go@40f1582b2485089dde7abd97c1529aa768e1baff # v5
|
|
with:
|
|
go-version-file: go.mod
|
|
|
|
- name: Verify eval credentials are configured
|
|
env:
|
|
PULSE_EVAL_USER: ${{ secrets.PULSE_EVAL_USER }}
|
|
PULSE_EVAL_PASS: ${{ secrets.PULSE_EVAL_PASS }}
|
|
run: |
|
|
set -euo pipefail
|
|
if [ -z "${PULSE_EVAL_USER}" ] || [ -z "${PULSE_EVAL_PASS}" ]; then
|
|
echo "::error::PULSE_EVAL_USER and PULSE_EVAL_PASS repo secrets must be set."
|
|
echo "::error::Refusing to fall back to admin/admin defaults."
|
|
exit 1
|
|
fi
|
|
|
|
- name: Run eval matrix
|
|
env:
|
|
EVAL_REPORT_DIR: tmp/eval-reports
|
|
PULSE_EVAL_USER: ${{ secrets.PULSE_EVAL_USER }}
|
|
PULSE_EVAL_PASS: ${{ secrets.PULSE_EVAL_PASS }}
|
|
INPUT_SCENARIO: ${{ inputs.scenario }}
|
|
INPUT_MODELS: ${{ inputs.models }}
|
|
INPUT_PROVIDERS: ${{ inputs.providers }}
|
|
INPUT_BASE_URL: ${{ inputs.base_url }}
|
|
run: |
|
|
set -euo pipefail
|
|
MODEL_ARGS=("-auto-models")
|
|
if [ -n "${INPUT_MODELS}" ]; then
|
|
MODEL_ARGS=("-models" "${INPUT_MODELS}")
|
|
fi
|
|
if [ -n "${INPUT_PROVIDERS}" ]; then
|
|
export EVAL_MODEL_PROVIDERS="${INPUT_PROVIDERS}"
|
|
fi
|
|
go run ./cmd/eval \
|
|
-scenario "${INPUT_SCENARIO}" \
|
|
"${MODEL_ARGS[@]}" \
|
|
-url "${INPUT_BASE_URL}" \
|
|
-user "${PULSE_EVAL_USER}" \
|
|
-pass "${PULSE_EVAL_PASS}"
|
|
|
|
- name: Upload eval reports
|
|
if: always()
|
|
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
|
|
with:
|
|
name: eval-reports
|
|
path: tmp/eval-reports
|
|
retention-days: 14
|