ProDG Mainframe — Consolidated Infrastructure Plan

Version: 2.0
Date: 2026-05-08
Owner: Mitch Ngugi, CEO ProDG Studio
Domain: prodg.studio

1. Executive Summary

This document consolidates all infrastructure planning from the original Option D migration through Phase 9 hardening, incorporating new decisions around:

Tavily as the web search API for agent research workloads
Infisical (self-hosted) as the single source of truth for all secrets
Vaultwarden (self-hosted) as the team password manager
Backblaze B2 as the offsite backup target
Tailscale as the secure networking layer for API routing
Modal for burst/untrusted inference workloads
SSH key-based access (Mainframe ed25519 key for Mitch)

Trust Model

Workload Type	Location	Container Runtime	Network
Trusted agents (research, infra)	On-box (mainframe)	Docker containers	Tailscale + Caddy
Untrusted code / burst inference	Remote Modal nodes	Modal sandbox	Tailscale egress

2. Architecture Overview

┌─────────────────────────────────────────────────────────────────────────┐
│                          ProDG Mainframe                                 │
│  (Bare metal — Ubuntu 22.04, Docker, Tailscale)                       │
├─────────────────────────────────────────────────────────────────────────┤
│  Public Edge                                                            │
│  ┌─────────┐  ┌──────────┐  ┌────────────────┐  ┌─────────────────┐  │
│  │ Caddy   │  │Headscale │  │  Infisical     │  │  Vaultwarden    │  │
│  │:80,:443 │  │:8080     │  │  :8082         │  │  :8083          │  │
│  └────┬────┘  └──────────┘  └────────────────┘  └─────────────────┘  │
│       │                                                                 │
│  Internal Services (prodg-internal network)                            │
│  ┌──────────┐ ┌─────────┐ ┌─────────┐ ┌──────────┐ ┌─────────────┐  │
│  │ Postgres │ │ Redis   │ │ MinIO   │ │ Grafana  │ │ Prometheus  │  │
│  │ :5432    │ │ :6379   │ │ :9000   │ │ :3000    │ │ :9090       │  │
│  └──────────┘ └─────────┘ └─────────┘ └──────────┘ └─────────────┘  │
│  ┌──────────┐ ┌─────────┐ ┌─────────┐ ┌──────────┐                   │
│  │ Loki     │ │Promtail │ │Node Exp │ │ cAdvisor  │                   │
│  │ :3100    │ │         │ │         │ │           │                   │
│  └──────────┘ └─────────┘ └─────────┘ └──────────┘                   │
│  ┌──────────┐                                                          │
│  │ Hermes   │ ← Tavily API (research agents)                          │
│  │ API      │ ← Backblaze B2 (offsite backups)                        │
│  │ :8000    │ ← Tailscale API routing                                 │
│  └──────────┘                                                          │
├─────────────────────────────────────────────────────────────────────────┤
│  Remote / Burst                                                         │
│  ┌────────────────────┐                                                │
│  │ Modal (Serverless) │ ← Untrusted code, burst inference              │
│  │  GPU/CPU workloads │                                                │
│  └────────────────────┘                                                │
└─────────────────────────────────────────────────────────────────────────┘

3. Service Inventory

3.1 Public-Facing Services

Service	Port	Domain	Purpose	Auth
Caddy	80, 443	`*.prodg.studio`	Reverse proxy, TLS termination	Let’s Encrypt
Headscale	8080	—	Tailscale control server	Pre-auth keys
Infisical	8082	`secrets.mainframe.prodg.studio`	Secret management	Service tokens
Vaultwarden	8083	`vault.mainframe.prodg.studio`	Password vault	Master password

3.2 Internal Services

Service	Port	Image	User (UID)	Data Dir
Postgres	5432	`postgres:16-alpine`	70:70	`/opt/prodg/data/postgres`
Redis	6379	`redis:7-alpine`	1000:1000	`/opt/prodg/data/redis`
MinIO	9000	`minio/minio:RELEASE.2026...`	1000:1000	`/opt/prodg/data/minio`
Grafana	3000	`grafana/grafana:11.6.0`	—	`/opt/prodg/data/grafana`
Prometheus	9090	`prom/prometheus:v3.3.1`	—	`/opt/prodg/data/prometheus`
Loki	3100	`grafana/loki:3.5.0`	—	`/opt/prodg/data/loki`
Promtail	—	`grafana/promtail:3.5.0`	—	—
Node Exporter	9100	`prom/node-exporter:v1.9.1`	—	—
cAdvisor	8085	`gcr.io/cadvisor:v0.52.1`	—	—

3.3 Application Services

Service	Port	Image	User (UID)	Secrets Source
Hermes API	8000	`prodg/hermes-api:v0.7.1`	1000:1000 (prodg)	Infisical → `.env`
Infisical	8082	`infisical/infisical:v0.159.27`	1000:1000 (node)	Self-managed

4. Security Model

4.1 Secrets Architecture

Single Source of Truth: Infisical (self-hosted)

Project: eb7f744a-bcd3-4ef6-b5e4-8e50a906bb92
Environment: prod
Domain: https://secrets.mainframe.prodg.studio
Access: Service token (st.ba814a0f...) with read,write scope on / path

Secrets in Infisical (18 total):

Secret	Type	Consumers
`POSTGRES_PASSWORD`	Connection	Postgres, Hermes API, Vaultwarden
`POSTGRES_USER`	Connection	Postgres
`POSTGRES_DB`	Connection	Postgres
`REDIS_PASSWORD`	Connection	Redis, Hermes API
`MINIO_ROOT_USER`	Credential	MinIO
`MINIO_ROOT_PASSWORD`	Credential	MinIO, Hermes API
`INFISICAL_AUTH_SECRET`	Auth	Infisical
`INFISICAL_JWT_SECRET`	Auth	Infisical
`INFISICAL_ENCRYPTION_KEY`	Encryption	Infisical
`VAULTWARDEN_ADMIN_TOKEN`	Admin	Vaultwarden
`GRAFANA_ADMIN_PASSWORD`	Admin	Grafana
`HERMES_API_TOKEN`	Auth	Hermes API
`TELEGRAM_BOT_TOKEN`	Integration	Backup scripts, alerts
`TELEGRAM_CHAT_ID`	Integration	Backup scripts, alerts
`B2_KEY_ID`	Cloud	Backup scripts
`B2_KEY_SECRET`	Cloud	Backup scripts
`B2_BUCKET`	Cloud	Backup scripts
`TAVILY_API_KEY`	API	Hermes API (research agents)

4.2 Access Control

Component	Method	Identity
Host SSH	ed25519 key	Mitch (`Mainframe` key)
Tailscale	Headscale + pre-auth	Node key registration
Infisical	Service token	Machine identity
Vaultwarden	Master password	Mitch (admin)
Grafana	Basic auth	admin + `GRAFANA_ADMIN_PASSWORD`
Hermes API	Bearer token	`HERMES_API_TOKEN`

4.3 Service User Model

UID	User	Services	Rationale
0	root	Caddy, Grafana, Prometheus, Loki, Promtail, Node Exporter, cAdvisor, Headscale, Vaultwarden	Requires root or not yet migrated
70	postgres	Postgres	Official image hardcoded UID
1000	prodg	Redis, MinIO, Infisical, Hermes API	Shared non-root for data access

Note: Vaultwarden was reverted to root (UID 0) during Phase 9 due to a Diesel auth loop caused by a stale DATABASE_URL. Future hardening can retry user: "1000:1000" after secrets are stable.

5. Networking

5.1 Docker Networks

Network	Driver	Services
`prodg-internal`	bridge	All services

5.2 Tailscale Integration

Headscale Control Server

Port: 8080 (internal), proxied via Caddy
Namespace: prodgroup
Users: Mitch (admin keys)

Hermes API Routing

API exposed via Tailscale for agent workloads
Agents on remote nodes connect through Tailscale tunnel
Modal workers can egress through Tailscale to reach api.mainframe.prodg.studio

5.3 Caddy Routing

*.mainframe.prodg.studio {
    # Vaultwarden
    vault.mainframe.prodg.studio → vaultwarden:8083
    
    # Infisical
    secrets.mainframe.prodg.studio → infisical:8082
    
    # Hermes API
    api.mainframe.prodg.studio → hermes-api:8000
    
    # Headscale
    ts.mainframe.prodg.studio → headscale:8080
    
    # Docs (Quartz)
    docs.mainframe.prodg.studio → file server /data/docs/public
}

6. Storage & Backup

6.1 On-Box Storage

Path	Contents	Size	Owner
`/opt/prodg/data/postgres`	PostgreSQL data	—	70:70
`/opt/prodg/data/redis`	Redis persistence	—	1000:1000
`/opt/prodg/data/minio`	S3-compatible object store	—	1000:1000
`/opt/prodg/data/vaultwarden`	Vaultwarden SQLite + attachments	—	—
`/opt/prodg/data/infisical`	Infisical encryption keys	—	1000:1000
`/opt/prodg/data/grafana`	Grafana dashboards & DB	—	—
`/opt/prodg/data/loki`	Loki log storage	—	—
`/opt/prodg/quartz`	Quartz knowledge base source	—	—
`/opt/prodg/docs`	Obsidian vault + generated site	—	—

6.2 Backup Strategy

Tool: rclone → Backblaze B2

Schedule	Contents	Destination
Daily	Postgres dump, MinIO, Vaultwarden	`b2:MainframeBackup/daily/`
Weekly	Full data directory	`b2:MainframeBackup/weekly/`
On-demand	Pre-migration snapshots	`/opt/prodg/backups/` (local)

Credential Source: Infisical → .env → backup-all.sh

No hardcoded credentials in scripts
B2 keys loaded from environment at runtime

7. Agent Workloads

7.1 Trusted Agents (On-Box)

Runtime: Docker containers on mainframe Network: prodg-internal + Tailscale Examples:

Research agents (Hermes API → Tavily)
Infrastructure automation (Docker socket access)
Log analysis (Loki queries)

Characteristics:

Long-running or cron-scheduled
Access to internal services (Postgres, Redis, MinIO)
Run as prodg UID 1000 where possible

7.2 Untrusted / Burst Agents (Remote)

Runtime: Modal serverless containers Network: Tailscale egress to mainframe Examples:

GPU inference workloads
Third-party code execution
High-compute research tasks

Characteristics:

Ephemeral (spin up ↔ spin down)
No direct access to internal Docker network
Authenticate to Hermes API via HERMES_API_TOKEN over Tailscale
Results written to MinIO or returned via API

7.3 Hermes API Integration

# Remote agent (Modal) calling trusted API
import modal
 
app = modal.App("prodg-agent")
 
@app.function(secrets=[modal.Secret.from_name("prodg-api-token")])
def research_task(query: str):
    # Connects via Tailscale to api.mainframe.prodg.studio
    response = requests.post(
        "https://api.mainframe.prodg.studio/v1/research",
        headers={"Authorization": f"Bearer {os.environ['HERMES_API_TOKEN']}"},
        json={"query": query, "search_provider": "tavily"}
    )
    return response.json()

8. Operational Runbooks

8.1 Regenerate `.env` from Infisical

/opt/prodg/scripts/load-secrets.sh
# Then restart services that need new env:
cd /opt/prodg/compose && docker compose up -d

8.2 Add a New Secret

Add to Infisical:

docker exec -e INFISICAL_TOKEN=$INFISICAL_TOKEN infisical \
  infisical --domain=https://secrets.mainframe.prodg.studio/api \
  secrets set --token=$INFISICAL_TOKEN --env=prod \
  --projectId=eb7f744a-bcd3-4ef6-b5e4-8e50a906bb92 \
  NEW_SECRET="value"

Regenerate .env:
```
/opt/prodg/scripts/load-secrets.sh
```
Restart affected services

8.3 Rotate a Service Token

Generate new token in Infisical UI
Update /opt/prodg/compose/.env.infisical
Run load-secrets.sh
Revoke old token in Infisical UI

8.4 Vaultwarden Admin Access

https://vault.mainframe.prodg.studio/admin
Token: (from Infisical → VAULTWARDEN_ADMIN_TOKEN)

8.5 SSH Access (Mitch)

ssh -i ~/.ssh/mainframe mitch@mainframe.prodg.studio
# Authenticates via ed25519 key: AAAAC3NzaC1lZDI1NTE5AAAAIKv5Y45VNS1nb2fwrmVsIe5gcPurhPoIHtXJh4gHgVSB

9. Phase Status

Phase	Status	Key Deliverables
Phase 1: Foundation	✅ Complete	Docker, networks, base compose
Phase 2: Observability	✅ Complete	Grafana, Prometheus, Loki stack
Phase 3: Headscale	✅ Complete	Tailscale control plane
Phase 4: Storage	✅ Complete	MinIO, Postgres, Redis
Phase 5: Secrets/Auth	✅ Complete	Infisical, Vaultwarden deployed
Phase 6: Hermes API	✅ Complete	API server with Tavily integration
Phase 7: Documentation	✅ Complete	Quartz site, Obsidian vault
Phase 8: Backups	✅ Complete	Rclone → B2, automated scripts
Phase 9: Hardening	🔄 17/18 done	See below

Phase 9 Remaining

Task	Status	Notes
`.env` backup	✅ Done	`/opt/prodg/backups/dotenv-20260429-132041.bak`
Data dir permissions	✅ Done	All dirs owned by correct UIDs
Redis → UID 1000	✅ Done	Healthy
MinIO → UID 1000	✅ Done	Healthy
Infisical → UID 1000	✅ Done	Healthy
Hermes API → UID 1000	✅ Done	Rebuilt `v0.7.1`
Backup scripts → no hardcoded	✅ Done	Sources from `.env`
Service token created	✅ Done	`st.ba814...`
Secrets migrated to Infisical	✅ Done	All 18 secrets pushed
`load-secrets.sh` created	✅ Done	Uses container CLI, self-hosted domain
`.env` regenerated from Infisical	✅ Done	Validated, compose config passes
Vaultwarden root revert	✅ Done	SQLite fallback after DB auth loop
~~Infisical CLI on host~~	~~N/A~~	Using container CLI instead
Remove legacy `.env`	⏳ Pending	Awaiting decision on caching strategy
Quartz cron re-enable	⏳ Pending	User requested pause during changes

10. Decisions Log

Date	Decision	Rationale
2026-04-27	Option D selected (self-hosted)	Sovereignty over cloud dependencies
2026-04-27	Obsidian + Quartz for docs	Markdown-native, git-tracked, static site
2026-04-28	Tavily for web search	Better citation quality than Serper/Brave for agents
2026-04-28	Backblaze B2 for offsite	Cost-effective S3-compatible cold storage
2026-04-29	Infisical for secrets	Self-hosted, Docker-native, service token auth
2026-04-29	Vaultwarden for passwords	Bitwarden-compatible, lightweight, self-hosted
2026-04-29	`prodg` UID 1000 for services	Shared non-root identity for cross-service data access
2026-05-08	Postgres stays UID 70	Official image hardcodes UID; changing breaks DB
2026-05-08	Vaultwarden reverted to root	Diesel auth loop with stale `DATABASE_URL` in config.json
2026-05-08	Remove `DATABASE_URL` from compose	Allowed Vaultwarden to start with built-in SQLite
2026-05-08	Use container CLI for Infisical	Host CLI install failed (404/no npm package); container has v0.41.89
2026-05-08	`.env` as transient cache	Docker Compose requires `.env` file for variable interpolation; regenerated from Infisical via script

11. Service Account Reference

Mitch (You)

Role: CEO, ProDG Studio
SSH Key: ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKv5Y45VNS1nb2fwrmVsIe5gcPurhPoIHtXJh4gHgVSB Mainframe
Added to mainframe: Yes
Infisical account: mitch@prodg.studio

Service Tokens

Token	Scope	Permissions	Expires
`st.ba814a0f...`	`prodg-mainframe-core` prod /	read, write	2026-05-09

⚠️ Note: Service token expires 2026-05-09. Plan rotation before expiry.

12. Quick Reference

File Paths

File	Purpose
`/opt/prodg/compose/docker-compose.yml`	Stack definition
`/opt/prodg/compose/.env`	TRANSIENT — regenerated from Infisical
`/opt/prodg/compose/.env.infisical`	GOLD — service token only, mode 600
`/opt/prodg/scripts/load-secrets.sh`	Regenerate `.env` from Infisical
`/opt/prodg/backups/scripts/backup-all.sh`	Daily backups to B2
`/opt/prodg/docs/index.md`	Wiki homepage
`/opt/prodg/docs/05-phase-9-plan.md`	Phase 9 detailed plan

One-Liners

# Stack status
docker compose -f /opt/prodg/compose/docker-compose.yml ps
 
# Regenerate env from Infisical
/opt/prodg/scripts/load-secrets.sh
 
# View Infisical secrets (from container)
docker exec -e INFISICAL_TOKEN=$(cat /opt/prodg/compose/.env.infisical | cut -d= -f2) \
  infisical infisical --domain=https://secrets.mainframe.prodg.studio/api \
  export --token=$INFISICAL_TOKEN --env=prod \
  --projectId=eb7f744a-bcd3-4ef6-b5e4-8e50a906bb92
 
# Manual backup
/opt/prodg/backups/scripts/backup-all.sh
 
# Check Tailscale status
docker exec headscale headscale nodes list

Document generated by Hermes Agent. Last updated: 2026-05-08.
For updates, edit /opt/prodg/docs/plan.md and run the Quartz rebuild job.

ProDG Knowledge Base

Explorer

plan