Summary: Between 20:46 UTC on September 1 and 14:32 UTC on September 2, TEFCA record discovery through CommonWell failed for every consent that reached it. Patients completed IAL2 identity verification normally, then saw "We had trouble locating your records." The certificates we use to authenticate to CommonWell had renewed automatically, and the renewed certificates were not yet registered with CommonWell.
Timeline (UTC):
Sep 1, 12:18 — AWS Certificate Manager automatically renews our test-mode CommonWell certificate.
Sep 1, 19:57 — AWS Certificate Manager renews our live-mode certificate.
Sep 1, 20:46 — A routine deploy restarts our services onto the renewed certificates. Live-mode requests to CommonWell begin failing as unauthorized; within minutes all are failing.
Sep 2, 13:51 — A customer reports IAL2 authorizations failing in test mode.
Sep 2, 13:52 — Incident opened. Investigation identifies certificate rejections in both modes.
Sep 2, 13:57 — Test-mode certificate re-registered. Test mode recovers within a minute.
Sep 2, ~14:10 — Live-mode certificate re-registered.
Sep 2, 14:12 — First live-mode requests succeed.
Sep 2, 14:32 — Last rejection at 14:31; all requests succeed from 14:32. Full recovery.
What happened: We authenticate to CommonWell with a client certificate that we must register with them before use. The certificates are managed in AWS Certificate Manager, which renews them automatically ahead of expiry, and a renewed certificate must be registered again. Both renewed on schedule on September 1. Registration is a manual step, and nothing tied it to the renewal.
Two factors extended the incident. Running services kept using the previous certificate until that evening's deploy restarted them, so the failure surfaced at the restart rather than at the renewal. And our monitors were not watching for this failure — affected requests completed without raising an error — so it went undetected for 17 hours.
Customer impact:
Consents that reached record discovery during the window saw the error above and retrieved no records; identity verification itself succeeded in every case. Those patients were not connected and will need to re-consent.
Scheduled background refreshes of existing CommonWell connections did not complete during the window and resumed on their normal schedule afterward. Records already retrieved remained available throughout.
Direct payer and provider connections were unaffected.
Remediation:
Alert on each certificate renewal in AWS Certificate Manager and register the renewed certificate with CommonWell before it goes into service.
Alarm directly on QHIN authorization failures, which would have caught this within minutes.
