backend

Prometheus TargetDown on 10th fleet server: node_exporter missing, Ansible deploy failed due to SSH MaxAuthTries burned by ed25519 key order. Fix = SSH hardening first (key-only, key order), then monitoring — all idempotent.

TL;DR: A TargetDown alert revealed a missing node_exporter, but SSH access failed due to MaxAuthTries exhaustion from multiple key attempts. The fix prioritized SSH hardening — key-only authentication with cleaned key order — before deploying monitoring. This idempotent playbook sequence ensures stable access as a prerequisite for reliable fleet monitoring.

The 10th fleet server shattered the automation illusion: SSH MaxAuthTries burned, cloud-init overrode hardening, node_exporter missing. Lesson: fleet discipline means idempotent pipeline, not manual one-offs.

TL;DR: Adding the tenth fleet server exposed SSH hardening failures from MaxAuthTries limits and cloud-init config overrides, plus a missing node_exporter. The Ansible playbook now manages drop-in SSH configs and enforces uniform monitoring rollout. True automation means the eleventh server deploys without manual intervention.

A backfilled next_review went stale in three days. The fix: compute derived fields at read time and keep manual overrides.

TL;DR: A hand-run backfill for next_review went stale the moment one new file hit the index, exposing it as a derived field that shouldn't be stored. The real fix computes it at read time, only filling blanks so manual overrides survive. SQLite and PostgreSQL settled this long ago: derive on read, store only for overrides or heavy read performance.

A server monitoring dashboard becomes noise when attention items pile up with no organization. Three complementary fixes transform a flat grid into a usable workflow.

TL;DR: Dashboard Guy Eighteen flat grid of alerts made it impossible to find critical issues across nine servers. Backend search with debounce, severity filter buttons, and collapsible server groups solved the noise problem by pushing all filtering logic server-side. Three tools together let you triage from hundreds of items in two taps or three words.

Mass-deleting Redis keys felt right until I read the docs. Versioned keys with INCR bumps make stale chat answers unreachable instead.

TL;DR: Editing an article left the Redis-cached AI answers stale, and the author's first fix—scanning and deleting every ask: key—turned out to solve the wrong problem. SCAN is non-blocking; the real pain is tracking key families across workers forever. Versioned keys fix it: bake versions into key names, bump with INCR on edits, and let TTL clean up.

Pure-Python cosine, vectors in MariaDB postmeta, embeddings computed locally. Semantic features with no new service on a small server.

TL;DR: Running a blog on a 1GB VPS rules out a vector database, so the author computes bge-m3 embeddings locally via Ollama and stores them as JSON in MariaDB postmeta. Related-article matching and dup-checks use pure-Python cosine similarity, fast enough for around 100 articles. Lesson: for small blogs, skip Qdrant or pgvector until scale demands it.

The knowledge seed failed on every container start although the script was in the image. Bare python resolved to the system interpreter.

TL;DR: The CI seed script failed because bare python resolved to the system interpreter, which had none of the project's packages or venv site-packages on its path. Running a script file by path skips CWD, so cd didn't help either. Fix: call the venv interpreter directly with PYTHONPATH set to the project root.

Hand-typed knowledge entries never reached the VPS. A repo-tracked seed file, upserted on every container start, keeps both databases converged.

TL;DR: Hand-typed knowledge entries lived only in the local database, so the VPS was missing four. The fix is a seed file plus a small seeder running on every container start, using INSERT ... ON DUPLICATE KEY UPDATE keyed on slug.

A correct regex run against stream fragments matches nothing when the marker spans two deltas. Strip the joined answer instead.

TL;DR: A CTA_FLAG marker leaked into a Redis-cached chat answer because the model split it across two stream chunks, so the per-delta regex strip never matched a full pattern. The fix adds a cleanup pass on the joined answer right before caching. Lesson: any regex run per-chunk is blind to split patterns, so sanitize at the last point before persistence.