The Daily Life of a Network Operator

18/10/2021

As part of the LACNIC 36 LACNOG 2021 event, LACNOG organized a panel under the title of “The daily Life of a Network Operator” and invited two operators working in the region to share their day-to-day experiences and real-life problems, as well as the solutions they had applied and the lessons they had learned during the process.

Moderated by Tomás Lynch, Jaime Olmos, Network Operations Center manager at the University of Guadalajara, and Christian Cojitambo Terán, Project Engineer at a fixed services ISP in Ecuador, shared their experiences “from the trenches,” as Lynch put it.

“Problems and their solutions are almost always the same regardless of whether the network is large or small. The idea is to find common ground,” Lynch explained.

The Ecuadorean case. Cojitambo Terán described the process used by the ISP where he works to solve a communication problem they experienced when implementing massive Internet service. “It was a pilot project (to the router) to complement communications. We had two servers, and one of them handled user authentication,” he said.

In the process, we detected that the client had no Internet. After analyzing the problem, we found that the router was sending a packet containing a badly formed Option 82 field. “We decided on two solutions, a global solution and partial solution. The first consisted of asking the manufacturer to develop a patch that could be applied to correct the router’s behavior. This involved development time so, in the meantime, we had to apply a partial solution, which was to ask each server administrator to provide another server that could be used while the manufacturer developed their solution,” said Cojitambo Terán.

Once the patch had been developed and installed, the router’s behavior improved.

Massive attack. Olmos de la Cruz then shared that last December the University of Guadalajara had received a massive denial of service attack targeting very specific services. “We began to see incidents where the University’s web pages were unreachable, services such as email and databases, and this alerted us to the fact that something was going on,” Olmos explained.

“The problem was not the devices, but the attacks we received from Europe and particularly from China,” he added.

The magnitude of the attack was unprecedented for the University. Given that there was little experience in solving this type of issues, what the University did was to sacrifice one of the ISPs via BGP, identifying the IPv4 prefix that was the object of the attack. This allowed putting an end to the incident.

Jorge Villa of LACNOG thanked both operators for sharing their experiences and noted they would help others to be aware of potential situations that might come up in their operations. “Operations include multiple services, fixed, mobile… so any experience from other operators will be of great help. This was the first participation in this type of event, which will help us learn from the experiences of others,” Villa concluded.