Deploy & Test on Thor

Thor is the GPU lab server used for integration testing of cyber-inference. It is accessible via SSH and hosts the production-like test environment.

Connection Details

Deploy Workflow

Follow these steps in order. Each depends on the previous.

1. Commit & Push (local machine)

bash
1git add -A && git commit -m "<message>" && git push

2. Pull on Thor (remote)

bash
1ssh matt@thor.lab "cd /home/matt/Local/cyber-inference && git pull"

3. Start the Server (remote)

The server runs via start.sh which handles uv sync and auto-restart.

bash
1# Interactive (see logs live) - use for debugging
2ssh -t matt@thor.lab "cd /home/matt/Local/cyber-inference && ./start.sh"
3
4# Background (detached) - use for long-running tests
5ssh matt@thor.lab "cd /home/matt/Local/cyber-inference && nohup ./start.sh > /tmp/cyber-inference.log 2>&1 &"

CUDA PyTorch wheels are verified automatically when NVIDIA hardware is detected.

4. Verify the Server

bash
1# Health check
2curl -s http://thor.lab:8337/health
3
4# List models
5curl -s http://thor.lab:8337/v1/models | python3 -m json.tool
6
7# System status
8curl -s http://thor.lab:8337/admin/status | python3 -m json.tool
9

The web GUI is available at: http://thor.lab:8337

5. Test Inference

bash
1# Chat completion
2curl -s http://thor.lab:8337/v1/chat/completions \
3  -H "Content-Type: application/json" \
4  -d '{"model": "<model_name>", "messages": [{"role": "user", "content": "Hello"}]}' \
5  | python3 -m json.tool
6
7# Embeddings
8curl -s http://thor.lab:8337/v1/embeddings \
9  -H "Content-Type: application/json" \
10  -d '{"model": "<model_name>", "input": "test text"}' \
11  | python3 -m json.tool

Quick One-Liner Deploy

Pull latest and restart in one command:

bash
1ssh -t matt@thor.lab "cd /home/matt/Local/cyber-inference && git pull && ./start.sh"

Troubleshooting

• Server won't start: Check logs with ssh matt@thor.lab "tail -50 /tmp/cyber-inference.log"
• Port in use: Kill existing process with ssh matt@thor.lab "pkill -f 'cyber-inference serve'"
• Check running processes: ssh matt@thor.lab "ps aux | grep cyber-inference"
• GPU/CUDA issues: ssh matt@thor.lab "nvidia-smi"