https://nuteo.info/en/tags/ops/Recent content in Ops on nuteo.infoHugoen© 2026 NuteoMon, 03 Aug 2026 00:00:00 +0000https://nuteo.info/en/blog/postmortem-postgres-outage/Mon, 03 Aug 2026 00:00:00 +0000https://nuteo.info/en/blog/postmortem-postgres-outage/<h1 id="postmortem-the-day-postgres-took-down-production">Postmortem: the day Postgres took down production</h1> <blockquote> <p><strong>TL;DR</strong>: A missing index on a foreign key column, combined with a cron job running during peak hours, caused a 47-minute outage on a billing API.</p> </blockquote> <h2 id="timeline-all-times-utc">Timeline (all times UTC)</h2> <ul> <li><strong>14:00</strong> — Cron job starts: <code>cleanup_old_invoices.sh</code></li> <li><strong>14:02</strong> — Customer reports slow checkout</li> <li><strong>14:05</strong> — PagerDuty alert: API latency p99 &gt; 5s</li> <li><strong>14:08</strong> — On-call engineer (me) opens laptop</li> <li><strong>14:14</strong> — Identify: Postgres CPU 100%, hundreds of <code>DELETE</code> queries waiting</li> <li><strong>14:22</strong> — Kill the cron job manually → queries drain in ~30s</li> <li><strong>14:25</strong> — Service recovering</li> <li><strong>14:47</strong> — Full recovery, replication caught up</li> </ul> <p><strong>Total impact</strong>: 47 minutes of degraded service, ~120 failed transactions.</p>https://nuteo.info/en/work/homelab-50-services/Mon, 03 Aug 2026 00:00:00 +0000https://nuteo.info/en/work/homelab-50-services/<h1 id="production-homelab-50-services-on-3-nodes">Production homelab: 50+ services on 3 nodes</h1> <p>Three Ubuntu boxes. One Caddy reverse proxy. Fifty-plus self-hosted services running the show — email, git, CI/CD, dashboards, password manager, photo backup, and a small Kubernetes cluster for the things that need it.</p> <h2 id="the-stack">The stack</h2> <table> <thead> <tr> <th>Node</th> <th>Role</th> <th>Specs</th> </tr> </thead> <tbody> <tr> <td><code>edge-01</code></td> <td>Reverse proxy + firewall</td> <td>2 vCPU, 4GB RAM, public IP</td> </tr> <tr> <td><code>app-01</code></td> <td>Application workloads</td> <td>8 vCPU, 32GB RAM, NVMe</td> </tr> <tr> <td><code>data-01</code></td> <td>Databases + storage</td> <td>8 vCPU, 32GB RAM, 2×4TB HDD (RAID1)</td> </tr> </tbody> </table> <h2 id="what-runs-where">What runs where</h2> <p><strong>edge-01</strong> (public-facing):</p>https://nuteo.info/en/work/server-inventory/Mon, 03 Aug 2026 00:00:00 +0000https://nuteo.info/en/work/server-inventory/<h1 id="server-inventory--deployment-automation">Server inventory &amp; deployment automation</h1> <p>When you have more than 5 servers, you stop remembering which one runs what. This tool fixes that. Single source of truth for: server, role, deploy status, uptime, and alerts.</p> <h2 id="what-it-does">What it does</h2> <ul> <li>📋 <strong>Inventory</strong> — name, IP, role, environment, tags</li> <li>🚀 <strong>Deploy queue</strong> — what&rsquo;s deploying where, status</li> <li>📊 <strong>Dashboard</strong> — uptime, last deploy, alerts</li> <li>🔔 <strong>Notifications</strong> — Slack/Telegram when deployments fail</li> </ul> <h2 id="stack">Stack</h2> <ul> <li><strong>Python</strong> (FastAPI) for the API</li> <li><strong>SQLite</strong> for local store (Postgres optional)</li> <li><strong>HTMX</strong> for the dashboard (no React, no build step)</li> <li><strong>systemd</strong> + cron for agent-side collection</li> </ul> <h2 id="architecture">Architecture</h2> <pre tabindex="0"><code>┌─────────────┐ ┌──────────────┐ ┌─────────────┐ │ each host │ │ central │ │ dashboard │ │ │ │ API │ │ │ │ agent.py ──┼──▶ │ FastAPI ────┼──▶ │ /dashboard │ │ (cron 5m) │ │ SQLite │ │ (HTMX) │ └─────────────┘ └──────────────┘ └─────────────┘ │ ▼ ┌─────────────┐ │ Telegram │ │ alerts │ └─────────────┘ </code></pre><p>The agent is a 50-line Python script that runs via cron every 5 minutes, collects <code>uptime</code>, <code>df</code>, <code>docker ps</code>, and POSTs to the central API.</p>