CENIC’s Network Operations Center
Once an institution is connected to CalREN, CENIC’s Network Operations Center (NOC) is the first point of contact for any issues or questions that arise. NOC engineers monitor the network 24 hours a day, 7 days a week, 365 days a year. CENIC members who experience a problem notify the NOC and are placed in immediate contact with an experienced, highly trained network engineer who records the issue in the NOC’s ticketing system and begins to troubleshoot with the CENIC member who has called. Engineers have experience sufficient to resolve most issues, and fewer than 1% of incoming tickets are escalated to a senior engineer.
Ju Kim, Senior Manager of Network Operations at UCLA and current chair of one of CENIC’s technical advisory committee's, notes that “the NOC performs proactive analysis on maintaining the CalREN network around the clock, and is doing a good job of recognizing, isolating, and resolving faults. When a problem is identified, maintenance is initiated and communicated to its member institutions in a timely and appropriate manner to ensure minimal interruption to production. I appreciate NOC’s participation in TAC meetings and the way it provides the team information on updates, progress, and changes to the CENIC network.”
When you walk into the NOC you see a large, open environment. Often you will find engineers huddled around a whiteboard discussing various ongoing tickets. You may also find a group huddled around a cubicle during a conference call, each offering expertise on how to resolve the issue at hand. Problems are frequently solved through this kind of collaboration.
The engineers who staff the NOC also diagnose and resolve problems identified by network monitoring tools, which generate an alarm when an issue is detected on the network. Engineers often detect problems before the users themselves and work with them to quickly resolve the issue.
The ticketing system tracks issues until they are resolved. Many can be resolved on the same day, while some are more complex and take longer. In 2016, the NOC:
- Processed 19,803 new tickets
- Assessed 85,410 alarms
NOC engineers also coordinate and communicate third-party maintenance and repair activities, perform equipment updates, replacements, and upgrades, decommission retired equipment, and handle tasks in support of circuit installations, service upgrades, and backbone upgrades. In 2016, the NOC:
- Performed 1,516 maintenance procedures
- Operationalized 196 new devices
- Deployed and operationalized 289 new circuits
Michael Kilpatrick, Technology Program Director at the University of California Health Systems, says “The CENIC Network Operations team has consistently been excellent for UC Health, delivering great value. Each of their engineers demonstrates a rapid response culture, an eagerness to engage, and a high level of professionalism and customer service.”
CENIC’s network engineers continuously strive to improve processes and documentation, and actively share technical knowledge and experience within the team. In addition to face-to-face interaction, internal mailing lists allow members to receive maintenance and outage announcements as well as resolution notices.
Engineers who work in the NOC gain broad experience and a deep understanding of CENIC. As a result, they become valuable assets and are frequently promoted elsewhere within the organization, bringing their expertise to departments such as Internet Services, Provisioning, and Core Engineering.
Engineers are available to assist CENIC members 24 hours per day, 7 days per week, including holidays (24/7/365).
The CENIC NOC can be reached at:
Telephone: (714) 220-3494 (714-220-FIXIT)