ProDG Knowledge Base

Onboarding Guide

5 min read

ProDG Mainframe — Onboarding Guide

Version: 1.0.0
Audience: Mitch (CEO), new team members, contractors

Welcome to ProDG Mainframe

This guide gets you from zero to productive on the ProDG infrastructure in under 10 minutes.

1. Prerequisites

1.1 What You Need

  • A Mac, Linux, or Windows machine with internet access
  • SSH keypair (ed25519 recommended)
  • Tailscale client installed
  • Web browser

1.2 Install Tailscale

macOS:

brew install tailscale
sudo tailscale up --login-server https://headscale.prodg.studio

Linux:

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --login-server https://headscale.prodg.studio

Windows: Download from tailscale.com/download

1.3 Authenticate

After running tailscale up, you’ll get a URL to authenticate. Open it in your browser and follow the Headscale registration flow. Once approved, you’ll appear on the tailnet as 100.64.0.x.

Verify:

tailscale status
# Should show: 100.64.0.x  your-hostname  mitch@  active

2. Service Access

2.1 Quick Reference Table

ServicePublic URLTailnet URLCredentials
Infisical (Secrets)https://secrets.mainframe.prodg.studiohttp://100.64.0.1:8082Super admin (Mitch)
Vaultwarden (Passwords)https://vault.mainframe.prodg.studiohttp://100.64.0.1:8083Master password
Grafana (Metrics)https://metrics.mainframe.prodg.studiohttp://100.64.0.1:3000admin / [in .env]
MinIO (S3 Console)https://s3.mainframe.prodg.studiohttp://100.64.0.1:9001prodg-minio / [in .env]
Hermes APIhttps://api.mainframe.prodg.studiohttp://100.64.0.1:8000X-API-Token
Headscalehttps://headscale.prodg.studio—CLI only

When connected to the tailnet, you can access all services directly without going through the public internet:

# SSH to mainframe (no public IP needed)
ssh root@100.64.0.1
 
# Curl API internally
curl http://100.64.0.1:8000/health
 
# Open Grafana in browser
open http://100.64.0.1:3000   # macOS
# Or navigate to http://100.64.0.1:3000 in your browser

3. Infisical — Secrets Management

3.1 First Login

  1. Navigate to https://secrets.mainframe.prodg.studio
  2. Log in with your super admin credentials (Mitch)
  3. Create organizations and projects as needed

3.2 Migrating Secrets from .env

Goal: Move all secrets from /opt/prodg/compose/.env into Infisical.

  1. In Infisical, create a project called mainframe
  2. Add environments: production, staging
  3. For each secret in .env, create a corresponding Infisical secret:
    • Key: POSTGRES_PASSWORD
    • Value: [value from .env]
    • Environment: production

3.3 Using Infisical Secrets in Docker Compose

Once migrated, update docker-compose.yml to use Infisical’s dynamic secret injection instead of .env:

# Example (requires Infisical CLI in container or sidecar)
environment:
  INFISICAL_TOKEN: ${INFISICAL_SERVICE_TOKEN}

Note: Full Infisical-Docker integration is Phase 9 work. For now, .env remains the source of truth.

4. Vaultwarden — Password Vault

4.1 Setup

  1. Navigate to https://vault.mainframe.prodg.studio
  2. Create your master password (this is your vault encryption key — write it down securely)
  3. Signups are currently disabled. Contact Mitch to create accounts.

4.2 Admin Panel

https://vault.mainframe.prodg.studio/admin

Admin token is in .env as VAULTWARDEN_ADMIN_TOKEN.

4.3 Browser Extension

Install the Bitwarden browser extension and point it to https://vault.mainframe.prodg.studio as the self-hosted server URL.

5. Grafana — Monitoring Dashboards

5.1 Login

  • URL: https://metrics.mainframe.prodg.studio
  • User: admin
  • Password: See GRAFANA_ADMIN_PASSWORD in /opt/prodg/compose/.env

5.2 Pre-configured Datasources

DatasourceURLStatus
Prometheushttp://prometheus:9090✅ Default
Lokihttp://loki:3100✅

5.3 Essential Dashboards to Create

  1. Node Overview — CPU, memory, disk, network from node-exporter
  2. Docker Containers — Container CPU/memory from cadvisor
  3. Hermes Tasks — Pending/running tasks from custom metrics
  4. Loki Explorer — Log browsing for all services

5.4 Alerting

Alerts are pre-configured and sent to the ProDG Telegram group. You can modify alert rules at:

/opt/prodg/compose/grafana/provisioning/alerting/rules.yml

6. Hermes API — Agent Orchestration

6.1 Authentication

Every API request requires:

Header: X-API-Token: <token>

Token location: HERMES_API_TOKEN in .env

6.2 Registering an Agent

API_TOKEN=$(grep HERMES_API_TOKEN /opt/prodg/compose/.env | cut -d= -f2)
 
curl -X POST https://api.mainframe.prodg.studio/v1/agents/register \
  -H "X-API-Token: $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "research-agent-01",
    "tier": "tier-2-trusted",
    "capabilities": ["research", "web-search"],
    "host": "agent-01.prodg.internal",
    "tailscale_ip": "100.64.0.10"
  }'

6.3 Submitting a Task

curl -X POST https://api.mainframe.prodg.studio/v1/tasks \
  -H "X-API-Token: $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "task_type": "web-research",
    "payload": {"query": "latest AI model releases 2026"},
    "tier_required": "tier-2-trusted",
    "priority": 5,
    "timeout_seconds": 300
  }'

6.4 Worker Polling for Tasks

curl "https://api.mainframe.prodg.studio/v1/tasks/next?tier=tier-2-trusted" \
  -H "X-API-Token: $API_TOKEN"

6.5 Dispatching a Docker Container

curl -X POST "https://api.mainframe.prodg.studio/v1/dispatch/docker?image=python:3.11-slim" \
  -H "X-API-Token: $API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "command": ["python", "-c", "print(\"hello from container\")"],
    "env": {"FOO": "bar"},
    "labels": {"job": "test"}
  }'

7. Backblaze B2 Backups

7.1 What’s Backed Up

DataFrequencyRetention
PostgreSQL (all databases)Daily 03:00 UTC30 days
MinIO object storageDaily 03:00 UTC30 days
Docker Compose configsDaily 03:00 UTC90 days

7.2 Checking Backup Status

# On mainframe
tail -20 /var/log/prodg-backup.log
 
# List remote backups
source /opt/prodg/compose/.env
RCLONE_B2_KEY_ID="$B2_KEY_ID" RCLONE_B2_KEY="$B2_KEY_SECRET" \
rclone ls :b2:MainframeBackup

7.3 Manual Backup Trigger

/opt/prodg/backups/scripts/backup-all.sh

8. Troubleshooting

8.1 Can’t Access a Service

  1. Check if connected to tailnet: tailscale status
  2. Try the tailnet IP directly: http://100.64.0.1:<port>
  3. Check if service is running: docker ps | grep <service>
  4. Check Caddy logs: docker logs --tail 20 caddy

8.2 Certificate Error

If you see an SSL/certificate error:

  1. Ensure DNS records are grey-cloud (DNS-only) in Cloudflare
  2. Check Caddy has obtained certs: docker logs caddy | grep "certificate"
  3. Force renewal: docker exec caddy caddy reload --config /etc/caddy/Caddyfile

8.3 Forgot Grafana Password

# On mainframe
cd /opt/prodg/compose
docker compose stop grafana
docker run --rm -v /opt/prodg/data/grafana:/var/lib/grafana \
  grafana/grafana:latest \
  sh -c "grafana-cli admin reset-admin-password 'NewPass123!'"
docker compose start grafana

8.4 Container Won’t Start

# Check logs
docker logs --tail 50 <container>
 
# Check for port conflicts
ss -tlnp | grep <port>
 
# Check disk space
df -h /
 
# Restart from scratch (preserves data volumes)
docker compose restart <service>

9. Security Best Practices

  1. Never commit .env to git. It’s already in .dockerignore.
  2. Use Tailnet for internal access rather than public URLs when possible.
  3. Rotate API tokens quarterly. See runbook Section 6.
  4. Store personal passwords in Vaultwarden, not in notes or chats.
  5. Monitor Telegram alerts — they signal real infrastructure issues.
  6. SSH keys only — password auth is disabled on mainframe.

10. Getting Help

IssueResource
Service downCheck Grafana alerts + /var/log/prodg-backup.log
Tailnet issuestailscale status + tailscale ping 100.64.0.1
API questionscurl https://api.mainframe.prodg.studio/health
Infisical helphttps://infisical.com/docs
Vaultwarden helphttps://github.com/dani-garcia/vaultwarden/wiki

Onboarding Guide Version: 1.0.0 — Generated by Hermes Agent

Graph View

Backlinks