A notification platform that had gone quiet
AlertAny lets an organization raise an alert from any phone: dial in, enter an extension and a passcode, and the system fans the message out to that extension's people by SMS and email. It had stopped working. Callers reached the system and were told no extensions were available. Separately, a second module had failed weeks earlier and nobody had noticed.
The silent failure came first. A service certificate had auto-renewed in the key vault, but the renewed public key was never pushed to the identity provider — so on the day the old key expired, authentication broke. Nothing raised a flag, because the subscription carried no credential-expiry monitoring at all.
The phone system was three faults stacked on each other. The call handler asked for available extensions without passing the number that had been dialed, so the list came back empty every single time. The stored phone number carried a typo — one extra digit. And five extensions were missing the owner record that makes them visible in the first place. Once callers could finally get through, a fourth issue surfaced: the notification step was silently skipping admin-owned extensions, because the code that resolves which account sends the message only understood the customer hierarchy.
All four defects were fixed and shipped as a versioned container image, with previous versions retained for instant rollback. Call-in and passcode entry were verified by the customer, and the notification path was repaired in the same pass. Because the platform provides no alerting on credential expiry, we scheduled a recheck ahead of the next renewal rather than leaving it to be discovered the same way.