The three building blocks
Monitoring: metrics, logs and user-perspective checks, alerting only on events that require action.
NOC or on-call: a NOC is staffed around the clock; on-call staff are woken by alerts. Both need clear responsibilities.
Runbooks: documented steps for known incidents so that the right action is taken quickly, even at night.