Flexpa Status - Incident history https://flexpastatus.com Flexpa Thu, 7 May 2026 23:51:00 +0000 Degraded Performance Type: Incident Duration: 12 minutes Affected Components: link.flexpa.com, api.flexpa.com May 8, 11:24:53 GMT+0 - Postmortem - S**ummary** On May 7 between approximately 7:51 PM and 8:08 PM ET, Flexpa customers experienced elevated response times and request timeouts on hot-path API endpoints, including the Flexpa OAuth Authorization URL. The root cause was an AWS infrastructure failure in us-east-1, which AWS has acknowledged in their AWS Health Dashboard ([AWS event](https://health.aws.amazon.com/health/status)). **Timeline (ET)** * **7:20 PM** — AWS data center experienced a thermal event in us-east-1 Availability Zone use1-az4 causing instance impairments due to loss of power. * **7:36 PM** — Internal monitoring detected increased Redis command latency on background job workers. * **7:51 PM** — Customer-facing API requests began hitting timeouts as the impaired Redis primary became progressively unresponsive. Status page updated to "Monitoring." * **8:06 PM** — Amazon ElastiCache's automatic Multi-AZ failover completed, promoting a replica in an unaffected Availability Zone to primary. * **8:08 PM** — Full service recovery confirmed; latency and error rates returned to baseline. * **8:25 PM** — AWS publicly acknowledged the AZ-level impairment. **What happened** Flexpa runs Redis as a Multi-AZ ElastiCache replication group, with a primary node and replicas distributed across multiple Availability Zones for exactly this kind of failure. When AWS lost power in use1-az4, our Redis primary was hosted in that AZ and progressively degraded over \~15 minutes before AWS's automatic failover triggered. During that window, requests that depended on Redis (rate limiting, session state, queue operations) stalled and eventually timed out at the request layer. Once the failover completed and traffic shifted to a replica in a healthy AZ, the service recovered immediately. **Customer impact** * Window of degraded service: approximately 17 minutes (7:51 PM – 8:08 PM ET). * Affected: hot-path API endpoints, OAuth authorization redirect, and queued background jobs. * No data was lost. Failed requests are safe to retry. **What we're doing** * We are reviewing our Redis client retry behavior to shorten the recovery window in future Multi-AZ failovers from \~90 seconds to under 30 seconds. * We are reviewing direct alerting under this scenario so that paging occurs sooner May 7, 23:51:00 GMT+0 - Monitoring - We observed degraded response times and a Redis failover at 00:06 UTC against several hot-path API endpoints including the Flexpa OAuth Authorization URL. The service is currently stable and we are investigating. May 8, 00:02:39 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 12 minutes

Affected Components: ,

<p><small>May <var data-var='date'> 8</var>, <var data-var='time'>11:24:53</var> GMT+0</small><br><strong>Postmortem</strong> - S**ummary** On May 7 between approximately 7:51 PM and 8:08 PM ET, Flexpa customers experienced elevated response times and request timeouts on hot-path API endpoints, including the Flexpa OAuth Authorization URL. The root cause was an AWS infrastructure failure in us-east-1, which AWS has acknowledged in their AWS Health Dashboard ([AWS event](https://health.aws.amazon.com/health/status)). **Timeline (ET)** * **7:20 PM** — AWS data center experienced a thermal event in us-east-1 Availability Zone use1-az4 causing instance impairments due to loss of power. * **7:36 PM** — Internal monitoring detected increased Redis command latency on background job workers. * **7:51 PM** — Customer-facing API requests began hitting timeouts as the impaired Redis primary became progressively unresponsive. Status page updated to "Monitoring." * **8:06 PM** — Amazon ElastiCache's automatic Multi-AZ failover completed, promoting a replica in an unaffected Availability Zone to primary. * **8:08 PM** — Full service recovery confirmed; latency and error rates returned to baseline. * **8:25 PM** — AWS publicly acknowledged the AZ-level impairment. **What happened** Flexpa runs Redis as a Multi-AZ ElastiCache replication group, with a primary node and replicas distributed across multiple Availability Zones for exactly this kind of failure. When AWS lost power in use1-az4, our Redis primary was hosted in that AZ and progressively degraded over \~15 minutes before AWS's automatic failover triggered. During that window, requests that depended on Redis (rate limiting, session state, queue operations) stalled and eventually timed out at the request layer. Once the failover completed and traffic shifted to a replica in a healthy AZ, the service recovered immediately. **Customer impact** * Window of degraded service: approximately 17 minutes (7:51 PM – 8:08 PM ET). * Affected: hot-path API endpoints, OAuth authorization redirect, and queued background jobs. * No data was lost. Failed requests are safe to retry. **What we're doing** * We are reviewing our Redis client retry behavior to shorten the recovery window in future Multi-AZ failovers from \~90 seconds to under 30 seconds. * We are reviewing direct alerting under this scenario so that paging occurs sooner.</p> <p><small>May <var data-var='date'> 7</var>, <var data-var='time'>23:51:00</var> GMT+0</small><br><strong>Monitoring</strong> - We observed degraded response times and a Redis failover at 00:06 UTC against several hot-path API endpoints including the Flexpa OAuth Authorization URL. The service is currently stable and we are investigating..</p> <p><small>May <var data-var='date'> 8</var>, <var data-var='time'>00:02:39</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Thu, 7 May 2026 23:51:00 +0000 https://flexpastatus.com/incident/cmow6nafk029tzswduiji0ixk https://flexpastatus.com/incident/cmow6nafk029tzswduiji0ixk Intermittent API errors Type: Incident Duration: 41 minutes Affected Components: api.flexpa.com Apr 29, 18:30:00 GMT+0 - Investigating - We are currently investigating this incident. Apr 29, 19:01:19 GMT+0 - Identified - Issue identified and a fix is being deployed Apr 29, 19:06:58 GMT+0 - Monitoring - Fix has been deployed and we are currently monitoring Apr 29, 19:11:06 GMT+0 - Resolved - This incident has been resolved. During the incident period 2.9% of requests to Flexpa API failed. The degraded state made the issue difficult to diagnose at first. We're reviewing observability improvements to reduce this in the future. Type: Incident

Duration: 41 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 29</var>, <var data-var='time'>18:30:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 29</var>, <var data-var='time'>19:01:19</var> GMT+0</small><br><strong>Identified</strong> - Issue identified and a fix is being deployed.</p> <p><small>Apr <var data-var='date'> 29</var>, <var data-var='time'>19:06:58</var> GMT+0</small><br><strong>Monitoring</strong> - Fix has been deployed and we are currently monitoring.</p> <p><small>Apr <var data-var='date'> 29</var>, <var data-var='time'>19:11:06</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. During the incident period 2.9% of requests to Flexpa API failed. The degraded state made the issue difficult to diagnose at first. We're reviewing observability improvements to reduce this in the future..</p> ]]> Tue, 29 Apr 2025 18:30:00 +0000 https://flexpastatus.com/incident/cma2v7o6n001jcscbu96yt2hy https://flexpastatus.com/incident/cma2v7o6n001jcscbu96yt2hy Pagination Resolution Type: Incident Duration: 3 hours and 1 minute Affected Components: api.flexpa.com Feb 5, 21:24:45 GMT+0 - Monitoring - A fix has been deployed and we are monitoring. Feb 5, 21:23:24 GMT+0 - Identified - \`Bundle.link\[0\].next\` values are returning incorrect base URL / hostname values. Requests made to these values as documented in our pagination led to timeouts in the period of Jan 20 to Feb 5. We are currently deploying a fix. Feb 6, 00:24:03 GMT+0 - Resolved - We have completed our monitoring and can validate resolution of this issue. Type: Incident

Duration: 3 hours and 1 minute

Affected Components:

<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>21:24:45</var> GMT+0</small><br><strong>Monitoring</strong> - A fix has been deployed and we are monitoring..</p> <p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>21:23:24</var> GMT+0</small><br><strong>Identified</strong> - \`Bundle.link\[0\].next\` values are returning incorrect base URL / hostname values. Requests made to these values as documented in our pagination led to timeouts in the period of Jan 20 to Feb 5. We are currently deploying a fix..</p> <p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>00:24:03</var> GMT+0</small><br><strong>Resolved</strong> - We have completed our monitoring and can validate resolution of this issue..</p> ]]> Wed, 5 Feb 2025 21:23:24 +0000 https://flexpastatus.com/incident/cm6sf15n00024mufbh58glk8e https://flexpastatus.com/incident/cm6sf15n00024mufbh58glk8e CORS error when accessing Flexpa Link Type: Incident Duration: 48 minutes Affected Components: link.flexpa.com, api.flexpa.com Oct 28, 14:52:41 GMT+0 - Investigating - We are currently investing an issue with our API that is resulting in a CORS error when opening Flexpa Link Oct 28, 15:40:36 GMT+0 - Resolved - The incident has been resolved. Our investigation confirmed that the impact was confined to local development environments, with no effect on production systems. The root cause was identified as a firewall configuration update that inadvertently blocked [localhost](http://localhost) requests. All systems are now functioning normally. Type: Incident

Duration: 48 minutes

Affected Components: ,

<p><small>Oct <var data-var='date'> 28</var>, <var data-var='time'>14:52:41</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investing an issue with our API that is resulting in a CORS error when opening Flexpa Link.</p> <p><small>Oct <var data-var='date'> 28</var>, <var data-var='time'>15:40:36</var> GMT+0</small><br><strong>Resolved</strong> - The incident has been resolved. Our investigation confirmed that the impact was confined to local development environments, with no effect on production systems. The root cause was identified as a firewall configuration update that inadvertently blocked [localhost](http://localhost) requests. All systems are now functioning normally..</p> ]]> Mon, 28 Oct 2024 14:52:41 +0000 https://flexpastatus.com/incident/cm2t51i9e00469h5dxdnbntrl https://flexpastatus.com/incident/cm2t51i9e00469h5dxdnbntrl Past access tokens incorrectly appearing as revoked Type: Incident Duration: 56 minutes Affected Components: api.flexpa.com Jul 16, 21:49:49 GMT+0 - Resolved - We have resolved the issue with our API. Jul 16, 20:53:36 GMT+0 - Investigating - We are currently investigating this incident. Jul 16, 21:19:45 GMT+0 - Identified - We are continuing to work on a fix for this incident. Jul 16, 21:42:25 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Type: Incident

Duration: 56 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 16</var>, <var data-var='time'>21:49:49</var> GMT+0</small><br><strong>Resolved</strong> - We have resolved the issue with our API. .</p> <p><small>Jul <var data-var='date'> 16</var>, <var data-var='time'>20:53:36</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jul <var data-var='date'> 16</var>, <var data-var='time'>21:19:45</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Jul <var data-var='date'> 16</var>, <var data-var='time'>21:42:25</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> ]]> Tue, 16 Jul 2024 20:53:36 +0000 https://flexpastatus.com/incident/clyow51dx140759hqoq5o6i40le https://flexpastatus.com/incident/clyow51dx140759hqoq5o6i40le Issue with API Type: Incident Duration: 28 minutes Affected Components: api.flexpa.com Jun 20, 19:14:30 GMT+0 - Investigating - We are currently investigating this incident. Jun 20, 19:42:42 GMT+0 - Resolved - This incident has been resolved. **What happened?** An update (migration) to our database took longer than expected and began to saturate connection resources. This resulted in services such as API not being able to make new connections. **What are we doing to remediate?** We are setting up increased monitoring in order to triage database connection issues faster. We are also optimizing our database lookups by using an index on the table that caused the issue. Type: Incident

Duration: 28 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 20</var>, <var data-var='time'>19:14:30</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jun <var data-var='date'> 20</var>, <var data-var='time'>19:42:42</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. **What happened?** An update (migration) to our database took longer than expected and began to saturate connection resources. This resulted in services such as API not being able to make new connections. **What are we doing to remediate?** We are setting up increased monitoring in order to triage database connection issues faster. We are also optimizing our database lookups by using an index on the table that caused the issue..</p> ]]> Thu, 20 Jun 2024 19:14:30 +0000 https://flexpastatus.com/incident/clxnn5h5e61956bmmwa2cefbq3 https://flexpastatus.com/incident/clxnn5h5e61956bmmwa2cefbq3 Elevated 500 Responses on FHIR requests Type: Incident Duration: 1 hour and 16 minutes Affected Components: api.flexpa.com Jun 13, 17:03:55 GMT+0 - Investigating - We are currently investigating this incident. Jun 13, 17:52:03 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Jun 13, 18:20:12 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 1 hour and 16 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 13</var>, <var data-var='time'>17:03:55</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jun <var data-var='date'> 13</var>, <var data-var='time'>17:52:03</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Jun <var data-var='date'> 13</var>, <var data-var='time'>18:20:12</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. .</p> ]]> Thu, 13 Jun 2024 17:03:55 +0000 https://flexpastatus.com/incident/clxdiejyo109633bgn3xjpn9d8m https://flexpastatus.com/incident/clxdiejyo109633bgn3xjpn9d8m Elevated 500 Responses Type: Incident Duration: 37 minutes Affected Components: api.flexpa.com Jun 10, 15:24:10 GMT+0 - Investigating - We are currently investigating this incident. Jun 10, 16:01:09 GMT+0 - Resolved - This incident has been resolved. We experienced a process failure during a routine database update, which caused some users to encounter errors when accessing the Flexpa API. Why did it happen? A database migration did not complete as expected, causing discrepancies between the application and the database. This was primarily due to the migration being prematurely stopped by our automated health checks. How to prevent it in the future? We are improving our deployment process to ensure that the application remains in a consistent state, even if a database update does not complete successfully. This change is intended to prevent similar errors from occurring in the future. Type: Incident

Duration: 37 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>15:24:10</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jun <var data-var='date'> 10</var>, <var data-var='time'>16:01:09</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. We experienced a process failure during a routine database update, which caused some users to encounter errors when accessing the Flexpa API. Why did it happen? A database migration did not complete as expected, causing discrepancies between the application and the database. This was primarily due to the migration being prematurely stopped by our automated health checks. How to prevent it in the future? We are improving our deployment process to ensure that the application remains in a consistent state, even if a database update does not complete successfully. This change is intended to prevent similar errors from occurring in the future..</p> ]]> Mon, 10 Jun 2024 15:24:10 +0000 https://flexpastatus.com/incident/clx94ipzd96587bcolwde0bhw0 https://flexpastatus.com/incident/clx94ipzd96587bcolwde0bhw0 Elevated 429 responses Type: Incident Duration: 3 hours and 49 minutes Affected Components: api.flexpa.com Jan 22, 14:15:00 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Jan 22, 13:51:01 GMT+0 - Investigating - We are currently investigating this incident. Jan 22, 17:39:52 GMT+0 - Resolved - This incident has been resolved. Type: Incident

Duration: 3 hours and 49 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 22</var>, <var data-var='time'>14:15:00</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Jan <var data-var='date'> 22</var>, <var data-var='time'>13:51:01</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Jan <var data-var='date'> 22</var>, <var data-var='time'>17:39:52</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> ]]> Mon, 22 Jan 2024 13:51:01 +0000 https://flexpastatus.com/incident/clrozjobs0209b3n14xahw03u https://flexpastatus.com/incident/clrozjobs0209b3n14xahw03u Stuck authorizations on Flexpa Link Type: Incident Duration: 1 day, 2 hours and 7 minutes Affected Components: link.flexpa.com Jan 10, 14:10:33 GMT+0 - Resolved - This incident has been resolved. The incident impacted successful Flexpa Link authorizations. Why did it happen? We pushed a bug into production code related to the generation of system Patient IDs upon a successful Flexpa Link authorization. Due to a test which passed incorrectly, the bad logic was not caught in development. How to prevent in future? By creating re-usable test fixtures, specifically as it relates to our patient authorization model, we can make it easier for developers to avoid writing false positive test cases. Jan 10, 13:44:27 GMT+0 - Investigating - We are currently investigating reports of stuck authorizations on Flexpa Link. Jan 10, 13:57:58 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Type: Incident

Duration: 1 day, 2 hours and 7 minutes

Affected Components:

<p><small>Jan <var data-var='date'> 10</var>, <var data-var='time'>14:10:33</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. The incident impacted successful Flexpa Link authorizations. Why did it happen? We pushed a bug into production code related to the generation of system Patient IDs upon a successful Flexpa Link authorization. Due to a test which passed incorrectly, the bad logic was not caught in development. How to prevent in future? By creating re-usable test fixtures, specifically as it relates to our patient authorization model, we can make it easier for developers to avoid writing false positive test cases..</p> <p><small>Jan <var data-var='date'> 10</var>, <var data-var='time'>13:44:27</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating reports of stuck authorizations on Flexpa Link..</p> <p><small>Jan <var data-var='date'> 10</var>, <var data-var='time'>13:57:58</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> ]]> Wed, 10 Jan 2024 13:44:27 +0000 https://flexpastatus.com/incident/clr7u106n18166bvltco1adv2u https://flexpastatus.com/incident/clr7u106n18166bvltco1adv2u MyFlexpa Test mode outage Type: Incident Duration: 23 hours and 39 minutes Affected Components: my.flexpa.com Dec 6, 19:00:03 GMT+0 - Investigating - We are currently investigating this incident. Dec 6, 19:23:41 GMT+0 - Resolved - This incident has been resolved. This did not affect any customer implementations or integrations with Flexpa Link. Why did it happen? We promoted an environment change in MyFlexpa's dev mode, and had to complete unexpected manual steps to redeploy the MyFlexpa application. How to prevent in future? Improve our internal change process for updating MyFlexpa's environment configuration by removing unnecessary manual steps. Type: Incident

Duration: 23 hours and 39 minutes

Affected Components:

<p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>19:00:03</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>19:23:41</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. This did not affect any customer implementations or integrations with Flexpa Link. Why did it happen? We promoted an environment change in MyFlexpa's dev mode, and had to complete unexpected manual steps to redeploy the MyFlexpa application. How to prevent in future? Improve our internal change process for updating MyFlexpa's environment configuration by removing unnecessary manual steps..</p> ]]> Wed, 6 Dec 2023 19:00:03 +0000 https://flexpastatus.com/incident/clpu4w1la25465bcodktgimx0i https://flexpastatus.com/incident/clpu4w1la25465bcodktgimx0i Issue with Link Type: Incident Duration: 1 day, 2 hours and 4 minutes Affected Components: link.flexpa.com Nov 13, 19:04:43 GMT+0 - Monitoring - We implemented a fix and are currently monitoring the result. Nov 13, 18:59:38 GMT+0 - Identified - We are continuing to work on a fix for this incident. Nov 13, 19:20:41 GMT+0 - Resolved - This incident has been resolved. Why did it happen? Flexpa released a fix to accommodate a scenario where some payer authentication flows were incompatible with Safari. However the fix, which uses cookies to handle communication between Flexpa Link and Flexpa API, introduced a bug which was undetectable in the development environment because the development environment shares the same domain across services. How do we prevent it in the future? - Broaden test suite across browsers. Safari, Chrome, Firefox etc. - Test across multiple domains in the development environment Nov 13, 18:54:38 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 1 day, 2 hours and 4 minutes

Affected Components:

<p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>19:04:43</var> GMT+0</small><br><strong>Monitoring</strong> - We implemented a fix and are currently monitoring the result..</p> <p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>18:59:38</var> GMT+0</small><br><strong>Identified</strong> - We are continuing to work on a fix for this incident..</p> <p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>19:20:41</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. Why did it happen? Flexpa released a fix to accommodate a scenario where some payer authentication flows were incompatible with Safari. However the fix, which uses cookies to handle communication between Flexpa Link and Flexpa API, introduced a bug which was undetectable in the development environment because the development environment shares the same domain across services. How do we prevent it in the future? - Broaden test suite across browsers. Safari, Chrome, Firefox etc. - Test across multiple domains in the development environment.</p> <p><small>Nov <var data-var='date'> 13</var>, <var data-var='time'>18:54:38</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Mon, 13 Nov 2023 18:54:38 +0000 https://flexpastatus.com/incident/clox9kikq93035b7ois1tbz7au https://flexpastatus.com/incident/clox9kikq93035b7ois1tbz7au Issue with Link Type: Incident Duration: 15 hours and 11 minutes Affected Components: link.flexpa.com Oct 26, 18:40:57 GMT+0 - Resolved - This incident has been resolved. Why did it happen? The new parameter in Flexpa Link was checked if it was defined, and so the word "undefined" was being serialized into the Flexpa Link application state, and the application was crashing on this unexpected value. How do we prevent it in the future? - Include Flexpa Link (package) in our CICD test suites. - Implement a UX Monitoring alarm on MyFlexpa to test that Link is loading successful Oct 26, 18:25:46 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 15 hours and 11 minutes

Affected Components:

<p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:40:57</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. Why did it happen? The new parameter in Flexpa Link was checked if it was defined, and so the word "undefined" was being serialized into the Flexpa Link application state, and the application was crashing on this unexpected value. How do we prevent it in the future? - Include Flexpa Link (package) in our CICD test suites. - Implement a UX Monitoring alarm on MyFlexpa to test that Link is loading successful.</p> <p><small>Oct <var data-var='date'> 26</var>, <var data-var='time'>18:25:46</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Thu, 26 Oct 2023 18:25:46 +0000 https://flexpastatus.com/incident/clo7im1da23590bjn9828cg7hj https://flexpastatus.com/incident/clo7im1da23590bjn9828cg7hj Issue with API Type: Incident Duration: 1 day, 19 hours and 57 minutes Affected Components: flexpa.com/docs, link.flexpa.com, api.flexpa.com, my.flexpa.com, portal.flexpa.com Oct 3, 19:14:27 GMT+0 - Investigating - We are currently experiencing an issue processing patients. Patients will see an error during a patient link: "Patient Init". Oct 3, 19:58:24 GMT+0 - Resolved - We have resolved the issue with our API. The database cache our apps connect to was destroyed in the course of a configuration change today. This prevented a service from being online. That service being offline meant that every new Patient Access Token transitioned to an error state. An error state was shown visibly to patients / our customers and Links do not occur successfully. This impacted all customers (excepting those opted out from our data cache) because every new Patient Access Token requires this step to successfully complete. We completed resolution of this event once we had all applications pointed at the new data cache. Flexpa takes downtime such as this event seriously. We have completed a post-mortem procedure and will be introducing changes to our process to prevent this issue from re-occurring. Type: Incident

Duration: 1 day, 19 hours and 57 minutes

Affected Components: , , , ,

<p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>19:14:27</var> GMT+0</small><br><strong>Investigating</strong> - We are currently experiencing an issue processing patients. Patients will see an error during a patient link: "Patient Init"..</p> <p><small>Oct <var data-var='date'> 3</var>, <var data-var='time'>19:58:24</var> GMT+0</small><br><strong>Resolved</strong> - We have resolved the issue with our API. The database cache our apps connect to was destroyed in the course of a configuration change today. This prevented a service from being online. That service being offline meant that every new Patient Access Token transitioned to an error state. An error state was shown visibly to patients / our customers and Links do not occur successfully. This impacted all customers (excepting those opted out from our data cache) because every new Patient Access Token requires this step to successfully complete. We completed resolution of this event once we had all applications pointed at the new data cache. Flexpa takes downtime such as this event seriously. We have completed a post-mortem procedure and will be introducing changes to our process to prevent this issue from re-occurring..</p> ]]> Tue, 3 Oct 2023 19:14:27 +0000 https://flexpastatus.com/incident/clnap821d81437b4n52gcej3o1 https://flexpastatus.com/incident/clnap821d81437b4n52gcej3o1 Issue with API Type: Incident Duration: 3 hours and 33 minutes Affected Components: flexpa.com/docs, link.flexpa.com, api.flexpa.com, my.flexpa.com, portal.flexpa.com Aug 18, 14:38:46 GMT+0 - Resolved - This incident has been resolved. Post mortem summary: **What happened?** api.flexpa.com was unavailable and 503s were returning from the load balancer. **Root cause analysis** We tried to deploy a Task Definition that was pointing to secrets that were not in the parameter store, because those secrets were removed after the deploy was initiated. **Action items to prevent its reoccurrence** Ensure that removing a secret from Doppler should regenerate a task definition and redeploy. Aug 18, 14:35:14 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 3 hours and 33 minutes

Affected Components: , , , ,

<p><small>Aug <var data-var='date'> 18</var>, <var data-var='time'>14:38:46</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. Post mortem summary: **What happened?** api.flexpa.com was unavailable and 503s were returning from the load balancer. **Root cause analysis** We tried to deploy a Task Definition that was pointing to secrets that were not in the parameter store, because those secrets were removed after the deploy was initiated. **Action items to prevent its reoccurrence** Ensure that removing a secret from Doppler should regenerate a task definition and redeploy..</p> <p><small>Aug <var data-var='date'> 18</var>, <var data-var='time'>14:35:14</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Fri, 18 Aug 2023 14:35:14 +0000 https://flexpastatus.com/incident/cllgozsje6038bnojyq5ojjk3 https://flexpastatus.com/incident/cllgozsje6038bnojyq5ojjk3 API Issue Type: Incident Duration: 1 minute Affected Components: api.flexpa.com Jul 27, 22:23:00 GMT+0 - Resolved - This incident has been resolved. We believe this was an incorrect alarm. Jul 27, 22:22:12 GMT+0 - Investigating - We are currently investing an issue with our API. Type: Incident

Duration: 1 minute

Affected Components:

<p><small>Jul <var data-var='date'> 27</var>, <var data-var='time'>22:23:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. We believe this was an incorrect alarm..</p> <p><small>Jul <var data-var='date'> 27</var>, <var data-var='time'>22:22:12</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investing an issue with our API..</p> ]]> Thu, 27 Jul 2023 22:22:12 +0000 https://flexpastatus.com/incident/clklpzkdm1256144eo46flt73je https://flexpastatus.com/incident/clklpzkdm1256144eo46flt73je API Issue Type: Incident Duration: 46 minutes Affected Components: api.flexpa.com Jul 21, 20:58:05 GMT+0 - Monitoring - We have identified the source of the issue. The API is operational and available during this time. We are working on a definitive resolution. Jul 21, 21:43:55 GMT+0 - Resolved - This incident has been resolved. We discovered an uncaught error in responses to our /fhir endpoint and implemented a fix. Type: Incident

Duration: 46 minutes

Affected Components:

<p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>20:58:05</var> GMT+0</small><br><strong>Monitoring</strong> - We have identified the source of the issue. The API is operational and available during this time. We are working on a definitive resolution..</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>21:43:55</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved. We discovered an uncaught error in responses to our /fhir endpoint and implemented a fix..</p> ]]> Fri, 21 Jul 2023 20:44:12 +0000 https://flexpastatus.com/incident/clkd1uf6d75979c9olck2vrpaq https://flexpastatus.com/incident/clkd1uf6d75979c9olck2vrpaq API Issue Type: Incident Duration: 1 hour Affected Components: api.flexpa.com Jul 21, 20:39:42 GMT+0 - Investigating - We are currently investing an issue with our API. Jul 21, 20:40:42 GMT+0 - Resolved - We have resolved the issue with our API. Type: Incident

Duration: 1 hour

Affected Components:

<p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>20:39:42</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investing an issue with our API..</p> <p><small>Jul <var data-var='date'> 21</var>, <var data-var='time'>20:40:42</var> GMT+0</small><br><strong>Resolved</strong> - We have resolved the issue with our API..</p> ]]> Fri, 21 Jul 2023 20:39:42 +0000 https://flexpastatus.com/incident/clkd1on5e66048czolnj39m7yn https://flexpastatus.com/incident/clkd1on5e66048czolnj39m7yn API Issue Type: Incident Duration: 31 minutes Affected Components: api.flexpa.com Jun 14, 08:14:12 GMT+0 - Investigating - We are currently investing an issue with our API. Jun 14, 08:14:42 GMT+0 - Resolved - We have resolved the issue with our API. Type: Incident

Duration: 31 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 14</var>, <var data-var='time'>08:14:12</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investing an issue with our API..</p> <p><small>Jun <var data-var='date'> 14</var>, <var data-var='time'>08:14:42</var> GMT+0</small><br><strong>Resolved</strong> - We have resolved the issue with our API..</p> ]]> Wed, 14 Jun 2023 08:14:12 +0000 https://flexpastatus.com/incident/clivfrep8606073icn5r05f1f26 https://flexpastatus.com/incident/clivfrep8606073icn5r05f1f26 API Issue Type: Incident Duration: 3 hours and 1 minute Affected Components: api.flexpa.com Jun 6, 23:25:42 GMT+0 - Investigating - We are currently investing an issue with our API. Jun 6, 23:28:42 GMT+0 - Resolved - We have resolved the issue with our API. Jun 12, 03:46:08 GMT+0 - Resolved - ### Post Incident Summary Availability of Flexpa is critically important to our customers. As part of our normal incident response, we have conducted a post-incident summary as to the source of this incident. On June, we deployed new logging mechanisms in order to more effectively troubleshoot customer issues. The deployment was completed successfully and internal testing showed systems were operationally normal. On June 6, Flexpa experienced three brief API outages cumulatively lasting 6 minutes. During these outages, Flexpa was completely unavailable as services restarted At 6:31 PM EST, it was observed that the MyFlexpa application was behaving erratically. At 7:22 PM EST, a crash of Flexpa applications was observed and noted internally. At 7:23 PM EST, Flexpa's applications had automatically restarted and recovered. At 8:05 PM EST, a root cause had been identified and a fix was deployed to update logging to prevent future crashes. We have been operationally normal since this time. At 8:26 PM EST, an update was applied to ensure future alerting and logging was routed to the appropriate location. Moving forward from this incident we are: * Improving testing environment for logging updates to detect issues in advance of production deployment * Continue investigation into logging libraries Flexpa depends on in order to understand failure more deeply and prevent similar issues in the future. * Re-affirming our commitment to operational excellence Type: Incident

Duration: 3 hours and 1 minute

Affected Components:

<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>23:25:42</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investing an issue with our API..</p> <p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>23:28:42</var> GMT+0</small><br><strong>Resolved</strong> - We have resolved the issue with our API..</p> <p><small>Jun <var data-var='date'> 12</var>, <var data-var='time'>03:46:08</var> GMT+0</small><br><strong>Resolved</strong> - ### Post Incident Summary Availability of Flexpa is critically important to our customers. As part of our normal incident response, we have conducted a post-incident summary as to the source of this incident. On June, we deployed new logging mechanisms in order to more effectively troubleshoot customer issues. The deployment was completed successfully and internal testing showed systems were operationally normal. On June 6, Flexpa experienced three brief API outages cumulatively lasting 6 minutes. During these outages, Flexpa was completely unavailable as services restarted At 6:31 PM EST, it was observed that the MyFlexpa application was behaving erratically. At 7:22 PM EST, a crash of Flexpa applications was observed and noted internally. At 7:23 PM EST, Flexpa's applications had automatically restarted and recovered. At 8:05 PM EST, a root cause had been identified and a fix was deployed to update logging to prevent future crashes. We have been operationally normal since this time. At 8:26 PM EST, an update was applied to ensure future alerting and logging was routed to the appropriate location. Moving forward from this incident we are: * Improving testing environment for logging updates to detect issues in advance of production deployment * Continue investigation into logging libraries Flexpa depends on in order to understand failure more deeply and prevent similar issues in the future. * Re-affirming our commitment to operational excellence.</p> ]]> Tue, 6 Jun 2023 23:25:41 +0000 https://flexpastatus.com/incident/clikwss0y197077i0ogkf9nqi6b https://flexpastatus.com/incident/clikwss0y197077i0ogkf9nqi6b API Issue Type: Incident Duration: 1 hour and 1 minute Affected Components: api.flexpa.com Jun 6, 23:22:12 GMT+0 - Investigating - We are currently investing an issue with our API. Jun 6, 23:23:12 GMT+0 - Resolved - We have resolved the issue with our API. Type: Incident

Duration: 1 hour and 1 minute

Affected Components:

<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>23:22:12</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investing an issue with our API..</p> <p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>23:23:12</var> GMT+0</small><br><strong>Resolved</strong> - We have resolved the issue with our API..</p> ]]> Tue, 6 Jun 2023 23:22:12 +0000 https://flexpastatus.com/incident/clikwoab3188281htog2wdqr7lx https://flexpastatus.com/incident/clikwoab3188281htog2wdqr7lx API Issue Type: Incident Duration: 30 minutes Affected Components: api.flexpa.com Jun 6, 22:44:12 GMT+0 - Investigating - We are currently investing an issue with our API. Jun 6, 22:44:42 GMT+0 - Resolved - We have resolved the issue with our API. Type: Incident

Duration: 30 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>22:44:12</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investing an issue with our API..</p> <p><small>Jun <var data-var='date'> 6</var>, <var data-var='time'>22:44:42</var> GMT+0</small><br><strong>Resolved</strong> - We have resolved the issue with our API..</p> ]]> Tue, 6 Jun 2023 22:44:12 +0000 https://flexpastatus.com/incident/clikvbf8z107605idognj0vujxw https://flexpastatus.com/incident/clikvbf8z107605idognj0vujxw API Issue Type: Incident Duration: 31 minutes Affected Components: api.flexpa.com Jun 3, 16:32:12 GMT+0 - Investigating - We are currently investing an issue with our API. Jun 3, 16:32:42 GMT+0 - Resolved - We have resolved the issue with our API. Type: Incident

Duration: 31 minutes

Affected Components:

<p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>16:32:12</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investing an issue with our API..</p> <p><small>Jun <var data-var='date'> 3</var>, <var data-var='time'>16:32:42</var> GMT+0</small><br><strong>Resolved</strong> - We have resolved the issue with our API..</p> ]]> Sat, 3 Jun 2023 16:32:12 +0000 https://flexpastatus.com/incident/clig7pgy2315616hpoql2ax5cpa https://flexpastatus.com/incident/clig7pgy2315616hpoql2ax5cpa Database incident Type: Incident Duration: 9 minutes Affected Components: api.flexpa.com Apr 20, 21:30:00 GMT+0 - Investigating - We are currently investigating this incident. Apr 20, 21:39:00 GMT+0 - Resolved - This incident has been resolved. Apr 26, 19:50:38 GMT+0 - Resolved - ### Post Incident Summary Availability of Flexpa is critically important to our customers. As part of our normal incident response, we have conducted a post-incident summary as to the source of this incident. On April 19, we deployed a new database proxy / bouncer (PG Bouncer) as a new piece of infrastructure to assist in a migration effort. The deployment was completed successfully and internal testing showed systems were operationally normal. On April 20, Flexpa experienced two brief API outages cumulatively lasting 17 minutes. We continued to experience high latency throughout the day. At 12:17 PM EST it was observed that an internal application was not available. At 12:22 PM EST our active database client connections began to drop dramatically from normal levels. At the same time, an internal responder was not able to successfully connect to the new database proxy infrastructure to investigate. At 12:43 PM EST an availability monitor on api.flexpa.com was triggered and automatically posted a public status alarm. The internal responder escalated an an incident response call began. At 12:51 PM EST the responding team restarted the database proxy and client connections successfully resumed. While external availability to api.flexpa.com was restored at this time, internal applications continued to be unavailable due to database connectivity. Investigation continued with a review of the proxy logs. The investigating team was able to determine that the application code used to connect to the database proxy had been misconfigured. At 5:30 PM EST the incident response team attempted to deploy a configuration change. This configuration change was not successful and resulted in an additional 9 minutes of downtime before being reverted. As a result, over the course of the evening of April 20, the incident response team executed work to remove the database proxy from our infrastructure, completing their work at 11:00 PM EST. We have been operationally normal since this time. Moving forward from this incident we are: * Committed to additional testing of any database proxy infrastructure we add to our system in the future * Improving alarms for internal services, that can serve as a warning sign for external availability issues * Re-affirming our commitment to operational excellence Type: Incident

Duration: 9 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>21:30:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> <p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>21:39:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 26</var>, <var data-var='time'>19:50:38</var> GMT+0</small><br><strong>Resolved</strong> - ### Post Incident Summary Availability of Flexpa is critically important to our customers. As part of our normal incident response, we have conducted a post-incident summary as to the source of this incident. On April 19, we deployed a new database proxy / bouncer (PG Bouncer) as a new piece of infrastructure to assist in a migration effort. The deployment was completed successfully and internal testing showed systems were operationally normal. On April 20, Flexpa experienced two brief API outages cumulatively lasting 17 minutes. We continued to experience high latency throughout the day. At 12:17 PM EST it was observed that an internal application was not available. At 12:22 PM EST our active database client connections began to drop dramatically from normal levels. At the same time, an internal responder was not able to successfully connect to the new database proxy infrastructure to investigate. At 12:43 PM EST an availability monitor on api.flexpa.com was triggered and automatically posted a public status alarm. The internal responder escalated an an incident response call began. At 12:51 PM EST the responding team restarted the database proxy and client connections successfully resumed. While external availability to api.flexpa.com was restored at this time, internal applications continued to be unavailable due to database connectivity. Investigation continued with a review of the proxy logs. The investigating team was able to determine that the application code used to connect to the database proxy had been misconfigured. At 5:30 PM EST the incident response team attempted to deploy a configuration change. This configuration change was not successful and resulted in an additional 9 minutes of downtime before being reverted. As a result, over the course of the evening of April 20, the incident response team executed work to remove the database proxy from our infrastructure, completing their work at 11:00 PM EST. We have been operationally normal since this time. Moving forward from this incident we are: * Committed to additional testing of any database proxy infrastructure we add to our system in the future * Improving alarms for internal services, that can serve as a warning sign for external availability issues * Re-affirming our commitment to operational excellence.</p> ]]> Thu, 20 Apr 2023 21:30:00 +0000 https://flexpastatus.com/incident/clgy3kxfc123894bnncbmpxzrnm https://flexpastatus.com/incident/clgy3kxfc123894bnncbmpxzrnm Database incident Type: Incident Duration: 8 minutes Affected Components: api.flexpa.com Apr 20, 16:51:00 GMT+0 - Resolved - This incident has been resolved. Apr 20, 16:43:00 GMT+0 - Investigating - We are currently investigating this incident. Type: Incident

Duration: 8 minutes

Affected Components:

<p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>16:51:00</var> GMT+0</small><br><strong>Resolved</strong> - This incident has been resolved..</p> <p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>16:43:00</var> GMT+0</small><br><strong>Investigating</strong> - We are currently investigating this incident..</p> ]]> Thu, 20 Apr 2023 16:43:00 +0000 https://flexpastatus.com/incident/clgy3k2uy122696bnncad9v8mwb https://flexpastatus.com/incident/clgy3k2uy122696bnncad9v8mwb Planned migration Type: Maintenance Duration: 13 minutes Affected Components: link.flexpa.com, api.flexpa.com, my.flexpa.com, portal.flexpa.com Apr 20, 03:30:01 GMT+0 - Identified - Maintenance is now in progress Apr 20, 03:30:00 GMT+0 - Identified - We are planning for a scheduled maintenance during this window. We expect the window to last 15 minutes. During the window, new authorizations via Flexpa Link will not be possible, nor will data retrieval requests from Flexpa API. Apr 20, 03:42:34 GMT+0 - Completed - Maintenance has completed successfully. Type: Maintenance

Duration: 13 minutes

Affected Components: , , ,

<p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>03:30:01</var> GMT+0</small><br><strong>Identified</strong> - Maintenance is now in progress.</p> <p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>03:30:00</var> GMT+0</small><br><strong>Identified</strong> - We are planning for a scheduled maintenance during this window. We expect the window to last 15 minutes. During the window, new authorizations via Flexpa Link will not be possible, nor will data retrieval requests from Flexpa API..</p> <p><small>Apr <var data-var='date'> 20</var>, <var data-var='time'>03:42:34</var> GMT+0</small><br><strong>Completed</strong> - Maintenance has completed successfully..</p> ]]> Thu, 20 Apr 2023 03:30:00 +0000 https://flexpastatus.com/maintenance/clgnz6tw5201431kjohv60jj8li https://flexpastatus.com/maintenance/clgnz6tw5201431kjohv60jj8li