Degraded Performance | 500 Errors Experienced Within Megaphone
Incident Report for Megaphone
Postmortem

Degraded Performance | 500 Errors Experienced Within Megaphone

Date: 12/5/19

Status: Complete, action items in progress

Summary: Megaphone CMS Internal Server Error Some Users | 180 Mins

Impact: No revenue loss.

Detection: Internal resources detected unusual levels of HTTP 500s.

Root Causes: There was a regression in a database query introduced by the production deploy this afternoon, related to soft deletes of orders.

Dear Valued Customer,

On December 5th 2019, some users experienced a “500 Internal Server Error” message. We deeply apologize for any inconvenience. Please know that we value your time and ability to access the CMS at your convenience. It is important to mention, no delivery of audio was interrupted in the incident.

After a swift analysis we identified the source of the error and have implemented policies to mitigate the likelihood of this type of incident in the future.

Please do not hesitate to let us know if you have any questions.

Sincerely,

Megaphone Team

Posted Dec 05, 2019 - 19:59 EST

Resolved
This incident has been resolved.
Posted Dec 05, 2019 - 19:17 EST
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Dec 05, 2019 - 18:09 EST
Identified
Certain users may encounter 500 page errors. Our engineers have identified this cause and are implementing a fix. We certainly apologize for any inconvenience and thank you for your patience. We will keep you posted with developments.
Posted Dec 05, 2019 - 17:01 EST
This incident affected: Megaphone CMS.