Skip to main content

​
1) Sign in

  1. Open the SwarmAI app. The login screen shows Email and Password inputs plus a Sign in button.
  2. Enter your workspace credentials. After sign-in, use the project picker in the header if you belong to multiple projects—most pages require an active project.

​
2) Add a secret

  1. Go to Settings → Secrets.
  2. Click Add secret. In the “Set API secret” dialog, provide a Secret label and Secret value (your provider API key).
  3. Save. Secrets are stored securely and not shown again; create separate labels per provider (e.g., openai-prod, anthropic-eval, gemini-pro).

​
3) Register a model for inference/eval

  1. In Settings → Models, click Add model.
  2. Fill the registry entry:
    • Display Name: model ref you’ll reuse (e.g., openai-gpt4o, gemini-pro, anthropic-sonnet).
    • Model name: provider identifier (e.g., gpt-4o, gemini-1.5-pro, claude-3.5-sonnet).
    • Provider: choose the provider you’re targeting.
    • Base URL: set the provider’s completion endpoint:
      • OpenAI: https://api.openai.com/v1/chat/completions
      • Gemini: https://generativelanguage.googleapis.com/v1beta/models/gemini-1.5-pro:generateContent
      • Anthropic: https://api.anthropic.com/v1/messages
    • Temperature / Retries / Max completion tokens: tune as needed.
    • API key (secret): select the secret you just added; use Manage secrets to jump back if it’s empty.
  3. Set a Default model reference at the top of the Models card so planners/evaluators can reuse it.
  4. Optional but common: in Planner and Evaluation tabs, pick the same model ref for Model reference so automated planning and scoring use your registry entry.

​
4) Build a test plan

  1. Open Test Plans and click the small + (Create new test plan). A full-screen editor opens.
  2. In Basic Details, fill Test Plan Name, Category, Status, Version, Tags, and Description.
  3. Add requirements in the Requirements tab (Add Requirement). Each has Title, Description, Category, Priority, and Acceptance Criteria.
  4. Add fixtures in the Fixtures tab (Create Fixture) with Name, Description, and JSON Fixture content (validated as an object).
  5. Add cases in the Test Cases tab (Add Test Case):
    • Title/Description
    • Optional Starting URL (must be http/https)
    • Optional Fixture (select from Fixtures)
    • Requirements (multi-select)
    • Priority, Test Type, Estimated Duration
    • Test Steps (ordered list) and Expected Result
  6. Click Save Changes (bottom of the dialog) to persist the plan.

​
5) Execute the plan

  1. Go to Execution. In Launch New Swarm, pick your Test Plan (dropdown shows test counts) and AI Model (model refs from Settings).
  2. Toggle Auto-Eval if you want automatic scoring.
  3. Click Launch. A swarm is created and started; active swarms appear in the list with status badges.
  4. Expand runs to monitor status, view evaluator explanations, cancel runs, or switch to manual review if needed.

​
6) Generate an API key

  1. Open API Keys in the sidebar.
  2. Click Create key, supply a Label, then confirm. The secret value is shown once in a modal—copy it before closing.
  3. Use this key with the SwarmAI API or the MCP CLI to trigger plan runs programmatically.