Kamal Deployment (kw-app)

Overview

kw-app uses Kamal for zero-downtime deployments to:

• Staging: Raspberry Pi 4 (ARM64, ~4GB RAM)
• Production: VPS (x86_64)

Key Features:

• Rolling restarts (zero downtime)
• Automated migrations before deploy
• Docker-based deployment
• SSH-based remote execution

Prerequisites

Local Machine Requirements:

• Ruby 3.2.2 (via chruby) - NOT Docker!
• Kamal gem installed globally or in bundle
• SSH keys configured for deployment servers

Architecture

Deployment Flow

Local Machine (chruby 3.2.2)
    ↓
Run: kamal deploy -d staging/production
    ↓
1. Build Docker image (ARM64 or x86_64)
2. Push to registry
3. SSH to target server
4. Run migrations (if needed)
5. Pull new image
6. Rolling restart (zero downtime)
7. Health checks

Why Native Ruby for Kamal?

• Kamal runs on host machine, not in Docker
• Needs native Ruby 3.2.2 (via chruby)
• Docker is used ONLY for the app container being deployed

Configuration Files

config/
├── deploy.yml               # Base config (not used directly)
├── deploy.staging.yml       # Staging (Raspberry Pi)
└── deploy.production.yml    # Production (VPS)

Key Settings

Staging (deploy.staging.yml):

• Server: Raspberry Pi 4
• Architecture: ARM64
• Memory: ~4GB (use batches for large operations)
• Branch: develop

Production (deploy.production.yml):

• Server: VPS
• Architecture: x86_64
• Branch: main

Common Commands

Deployment Commands

All Kamal commands run with native Ruby (chruby):

bash
1# Deploy to staging
2zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal deploy -d staging'
3
4# Deploy to production
5zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal deploy -d production'
6
7# Deploy specific branch
8zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal deploy -d staging --version=feature-branch'
9
10# Check deployment status
11zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app details -d staging'
12
13# View logs
14zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app logs -d staging'
15
16# Rollback to previous version
17zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal rollback -d production'

Console Access

Staging Console:

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app exec -d staging -i --reuse "bin/rails console"'

Production Console:

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app exec -d production -i --reuse "bin/rails console"'

Flag Meanings:

• -d staging/production - Destination environment
• -i / --interactive - Keep terminal interactive (required for console)
• --reuse - Reuse existing SSH connection (no registry auth needed)

One-off Commands

Run rake task:

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app exec -d staging --reuse "bin/rails db:migrate:status"'

Run Rails runner:

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app exec -d staging --reuse "bin/rails runner \"puts User.count\""'

Deployment Workflow

1. Pre-Deployment Checklist

Before deploying:
- [ ] All tests passing locally
- [ ] Migrations reviewed and tested
- [ ] Environment variables updated in credentials
- [ ] Staging tested (for production deploys)
- [ ] Backup taken (for production)

2. Standard Deployment

bash
1# 1. Check current status
2zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app details -d staging'
3
4# 2. Deploy
5zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal deploy -d staging'
6
7# 3. Verify deployment
8zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app logs -d staging'
9
10# 4. Smoke test via console
11zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app exec -d staging -i --reuse "bin/rails console"'

3. Migration Deployment

Kamal runs migrations automatically before deploying new containers.

If migration fails, deployment stops (zero-downtime preserved).

Manual migration (if needed):

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app exec -d staging --reuse "bin/rails db:migrate"'

4. Emergency Rollback

bash
1# Rollback to previous version
2zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal rollback -d production'
3
4# Check status
5zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app details -d production'

Troubleshooting

Issue: Deployment Hangs

Symptoms: Kamal stuck on "Waiting for health check"

Solutions:

1. Check app logs:

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app logs -d staging'

2. Check container status:

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app details -d staging'

3. SSH to server and check Docker:

bash
1ssh staging-server
2docker ps
3docker logs <container-id>

Issue: "Registry Authentication Failed"

Cause: Trying to use --reuse without an active connection.

Solutions:

1. Remove --reuse for first connection:

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app exec -d staging -i "bin/rails console"'

2. Or authenticate first:

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal registry login -d staging'

Issue: Out of Memory (Staging Pi)

Symptoms: Deployment succeeds but app crashes on Raspberry Pi.

Cause: Pi has only ~4GB RAM.

Solutions:

1. Reduce Sidekiq concurrency in staging
2. Use database batches for large operations
3. Deploy during low-traffic times
4. Consider upgrading Pi memory

Issue: "Wrong Ruby Version"

Symptoms:

Your Ruby version is 2.6.10, but your Gemfile specified 3.2.2

Cause: Running Kamal without chruby.

Solution: Always wrap in zsh -c 'source ~/.zshrc && chruby 3.2.2 && ...'

Issue: Migrations Fail During Deployment

Symptoms: Deployment stops with migration error.

Solution:

1. Check migration on local:

bash
1docker-compose exec -T app bundle exec rake db:migrate

2. Fix migration and redeploy
3. OR manually run on server:

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app exec -d staging --reuse "bin/rails db:migrate"'

CI/CD Integration

GitHub Actions (Automated)

Staging deploys automatically on push to develop:

yaml
1name: Deploy to Staging
2on:
3  push:
4    branches: [develop]
5jobs:
6  deploy:
7    runs-on: self-hosted  # Runs on Pi
8    steps:
9      - uses: actions/checkout@v2
10      - name: Deploy
11        run: bundle exec kamal deploy -d staging

Production deploys on push to main:

yaml
1name: Deploy to Production
2on:
3  push:
4    branches: [main]
5jobs:
6  deploy:
7    runs-on: ubuntu-latest
8    steps:
9      - uses: actions/checkout@v2
10      - name: Deploy
11        run: bundle exec kamal deploy -d production

Security Best Practices

✅ Always Do

• Use SSH keys (never passwords)
• Store secrets in Rails encrypted credentials
• Use --reuse flag to avoid exposing registry credentials
• Review migrations before production deploy
• Test on staging first
• Keep Kamal gem updated

⚠️ Ask First

• Modifying deploy.*.yml files
• Changing server SSH configurations
• Deploying to production outside of CI/CD
• Running destructive rake tasks on production

🚫 Never Do

• Commit master.key files
• Hardcode passwords in deploy configs
• Deploy directly to production without staging test
• Skip migrations testing
• Force deploy without health checks

Monitoring

Post-Deployment Verification

bash
1# 1. Check app is running
2zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app details -d production'
3
4# 2. Check logs for errors
5zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app logs -d production --tail 100'
6
7# 3. Test critical paths (console)
8zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal app exec -d production -i --reuse "bin/rails console"'
9> User.count
10> # Test other critical models
11
12# 4. Check Sidekiq
13# Visit: https://your-domain.com/sidekiq

Monitoring Endpoints

• Sidekiq Web UI: https://domain.com/sidekiq
• Health Check: https://domain.com/health (if configured)
• App Logs: Via Kamal app logs command

Quick Reference

Command Pattern

ALL Kamal commands:

bash
1zsh -c 'source ~/.zshrc && chruby 3.2.2 && bundle exec kamal <command> -d <env> [flags]'

Common Operations

Key Flags

• -d <env> - Destination (staging/production)
• -i / --interactive - Interactive mode (console)
• --reuse - Reuse SSH connection
• --version=<branch> - Deploy specific branch

Additional Resources

• Kamal Docs: https://kamal-deploy.org/
• kw-app Ansible Setup: ansible/README.md
• Server Provisioning: ansible/playbooks/
• Deploy Configs: config/deploy.*.yml

Version: 2.0
Last Updated: 2024-01
Maintained By: kw-app team