Following the outage on June 15th, which drew both anxiety and anger from both customers and regulatory officials alike, T-Mobile’s Neville Ray has detailed what precisely went wrong with its network. It turns out, a single leased circuit from a third-party provider in a specific area was to blame, but the problem quickly spiraled into a network-wide overload.
According to the combination transparency report/apology, problems started when a single leased fiber circuit from an unnamed third-party provider in the southeast US failed. By itself, that shouldn’t be a problem, T-Mobile has backups and redundancies meant to gracefully reroute traffic in the event of such failures — but for whatever reason, they didn’t work. The resulting overload resulted in what T-Mobile calls an “IP traffic storm” that affected its entire IP multimedia core network for VoLTE calls.
I want to be fully transparent about what happened yesterday with our network. We did not meet our own bar for excellence. We have taken the necessary steps to avoid reoccurrence and truly apologize for any inconvenience we created. https://t.co/sDXZemXRsK
— Neville (@NevilleRay) June 17, 2020
Ray says that only VoLTE service was affected, but reports indicate that customers ran into issues regarding other aspects of service, including data and SMS.
Although the problem has since resolved, Ray claims the company has worked with its vendors to increase safeguards that might prevent a recurrence of similar issues. In the meantime, the company still doesn’t know what caused that initial circuit to fail.
The T-Mobile network outage is unacceptable. The @FCC is launching an investigation. We’re demanding answers—and so are American consumers.
— Ajit Pai (@AjitPaiFCC) June 16, 2020
It remains to be seen if this explanation is enough for the FCC, though. FCC chairman Ajit Pai previously expressed that his agency was opening an investigation into the outage. While these details are a sort of answer (outside the initial trigger, anyway), who knows if it will be enough to smooth over regulatory concerns, which could be mounting following the Sprint merger — especially in the wake of so many lost jobs, contrary to the original “more jobs from day one” claim.
