Fastly Implements Fix For Software Bug That Caused Outage

Fastly Inc., which operates a content delivery network, announced that it quickly detected a software bug that caused an outage on Tuesday (June 8), and that most of its network was back to running normally within 49 minutes, according to a blog post.

“We detected the disruption within one minute, then identified and isolated the cause, and disabled the configuration. Within 49 minutes, 95 percent of our network was operating as normal,” Senior Vice President of Engineering and Infrastructure Nick Rockwell said in the post.

In describing the cause of the outage, Rockwell wrote in the post that the company “began a software deployment” on May 12 “that introduced a bug that could be triggered by a specific customer configuration under specific circumstances.” Then, early on June 8, a client “pushed a valid configuration change that included the specific circumstances that triggered the bug.” Fastly then “created a permanent fix for the bug and began deploying it” later that day, according to Rockwell.

Going forward, Rockwell noted that the company is implementing the fix for the bug throughout its network “as quickly and safely as possible.” In addition, he said that the company is holding a full “post mortem” of the “processes and practices” the company adhered to during the incident.

Rockwell added that the firm would “figure out why we didn’t detect the bug during our software quality assurance and testing processes” and “evaluate ways to improve our remediation time.”

News and government websites went down early Tuesday (June 8) amid the outage, as previously reported. The sites were down for approximately an hour starting at 6 a.m. Eastern Time.

“We provide mission-critical services, and we treat any action that can cause service issues with the utmost sensitivity and priority. We apologize to our customers and those who rely on them for the outage and sincerely thank the community for its support,” Rockwell said in the post.