First day internet outages were caused by off-campus failure
On the first day of online classes for the spring 2022 semester, countless Teams calls dropped midstream as Apogee’s MyResNet wifi disconnected throughout Canyon Creek, University Village and the Residence Halls. Students were left scrambling to get back to class, as the outage took several hours to resolve.
As a Peer Advisor in Residence Hall North, criminology senior Carol Lehavan fielded calls from frantic first-years looking to get reconnected, sending them to the Hall’s lobby and other areas that retained CometNet access during the outage.
“The January outage was inconvenient, considering it was the first day of classes,” Lehavan said. “I know a lot of people exhausted hotspots or went to other buildings on campus just to do their classes, which kind of defeats the purpose of having virtual classes that you do from your room.”
Connection was restored around 8:11 p.m. that night after a nine hour outage. Matthew Grief, associate vice president for Student Affairs and director of Housing Operations, said that the issue was due to an off-campus equipment malfunction.
“I want to stress that it was not something on campus that was causing the problem. There’s an off-campus internet provider—or fiber provider company—called Zayo,” Grief said.
Apogee uses Zayo Group’s fiber optics cables to communicate data quickly and securely to UTD. When their equipment malfunctioned, it took out residential Wi-Fi connections on campus, with no clear timeline for when they would be restored. To ensure students could resume classes in the meantime, Grief reached out to UTD’s Office of Information Technology, which worked with Apogee to create a “temporary internet solution” as described in a Housing email to residents that evening.
“They were able to do the cross connect that evening through a little bit of magic, and they also were able to allow that connection to continue until Zayo restored their network,” Grief said.
UTD transitioned back to Zayo’s fiber connection with only a five-minute break in service just after midnight on Jan. 25. OIT is leaving this failsafe in place to be used for similar situations in the future—at least until a more permanent fix is complete.
“Both of us left that configuration in place after this last incident. So, it will be a relatively quick cut over, but still requires manual intervention. Our engineers must coordinate that activity with their engineers,” said Brian Dourty, associate vice president and chief technology officer at OIT. “Having a second circuit to campus for Apogee, that will be a permanent solution.”
Leaving that framework for connection through OIT means that, in case of a future outage, residents will have less of a wait before a temporary fix in in place. The path is already paved. Moving forward, however, a second circuit would prevent that disconnection in the first place, with interruptions solved quickly by reverting to a different pathway to campus entirely.
Dourty has advocated since last February’s freeze for the installation of that kind of backup network for campus wifi, which would kick in automatically. Apogee’s new plan is to contract with another fiber provider for their second path to campus, making it independent of another Zayo outage, and vice versa.
“Students wouldn’t feel a disruption if one of those two paths were to go down,” he said. “Now, in the odd chance that both of those paths would go down, we would still be able to provide sort of that final failover connectivity, as we did both in this instance and the instance back in February of last year.”
Apogee had begun work on such a network during Summer 2021, but complications with the original partner company set them back to square one at the start of the fall semester. The current timeline for the implementation of that secondary failsafe is tentative.
“The latest I heard from Apogee is they have reached somewhat of an agreement with the outside internet provider—or the fiber provider, I should say,” Grief said. “Once we get all the approvals, then it’s usually about 30 to 45 days [to implement].”
UTD’s contract with Apogee will soon be up for renewal. While Housing, OIT and Apogee have worked together well over the past few years in resolving issues like this most recent outage, Grief said that it is too early to determine whether that contract will be renewedrenewal will actually occur.
“A lot of evaluation has to go into it, and there are a lot of people involved in that,” he said. “So right now, our service provider will begin evaluating heavily within the next few months… we want to make sure that they’re providing what we’re asking them to provide our students.”
After all, as Apogee’s residential services page states: “While lost connectivity may have once been a survivable annoyance, a class lecture dropped midstream is a failure of your mission.”