A Five-Fold Path Toward Network Orchestration and Automation

Featured article by Olivier Huynh Van, CTO and Co-founder, Glue Networks

Network operators today have a tremendous challenge on their hands. Wide area networks (WAN) have become extremely complex, and operators must work within “brownfield” environments comprising both old and new equipment. And the staggering array of network equipment in large enterprises doesn’t help matters.

Business users are asking for specific network throughput guarantees when it comes to their critical applications, legal departments require compliance with mandated regulatory frameworks, and operations are asked to do more with shrinking budgets. All these requirements do not easily align with existing network architectures; hence, network operators are continuously faced with a slew of granular parameter change requests, trying to meet ongoing network requirement changes without having the proper tools in place.

It’s no longer workable for engineers to use hop-by-hop or router-to-router management to

manage configurations due to rapid changes to hardware and software. Network complexity today requires control of the entire environment from end to end, and the ability to apply policy changes with the utmost precision across a whole network. Let’s consider the path to making these changes:

– Is the device connecting wirelessly to the network, or is it hard-wired?

– Is the device coming from a remote site, or is it on a corporate LAN?

– Where is the end user’s device, whether it be a laptop or a PC?

– Will changes to the device involve other ops teams (i.e. telco and VoIP)?

– Does the end user need access to public or private cloud (or both)?

These questions need answers, after which change requests must be completed to meet the user’s request and satisfy the network performance requirement of the application while having minimal effect on the other services running on the network. The next hurdle is keeping up with those changes. It isn’t unusual for teams to fall behind as the changes roll in, but when operations is given a complex request, only to realize there is no current network documentation, the results can be disastrous. Often senior network architects, designers and engineers must collaborate to minimize potential side effects, which slows the process even more.

Engineering Challenges

Engineers work under difficult circumstances; their job has been likened to rebuilding a jet engine while in flight. In corporate networks, disruptions can be catastrophic to the business. Hence, all changes must undergo stringent verification and approval processes. Add in changes that must be made across domains—for example, security, disaster recovery, video and VoIP—and a disparate knowledge base between the different departments involved often leads to conflicting or incomplete change requests.

What further complicates matters is the lack of highly skilled “full-stack engineers,” professionals who can program software and make configuration changes with networking on the fly, regardless of the application or the equipment. Operations personnel tend to be generalists, and without a detailed skillset, they must rely upon subject matter experts who are well versed in various subsets of technology. Alternatively, they must wait on configuration changes to be released by the hardware or software vendor, or engage a contractor from a third party, a slow and painful process at best.

Most networks grow and evolve over time – the old brownfield analogy. This contrasts with the less common “greenfield” scenario, where the engineers can pick the newest router or product with the most up-to-date feature set. The average replacement cycle is usually dictated by the manufacturer’s support cycle, typically five to 10 years. Such a long cycle can create a significant mismatch of feature sets supported, since new firmware is issued every six to 12 months on average, and existing devices are only updated when necessary. Today, engineers can’t think of each router or device as being the same but instead must consider what version of firmware was installed, what hardware plug-in extensions have come and gone, and then mix and match configurations that work from end to end.

So Many Priorities

Customers have come to expect—rightly or wrongly—that apps will always be available, from anywhere. The performance of today’s critical apps leaves no room for error, failure or downtime. This brings us back to the “jet in flight” analogy, but it’s not just a single jet – it’s more like fixing multiple jet engines from multiple manufacturers while in flight.

This environment tends to become impossible, even if enterprises try to standardize deployed hardware. With nearly 30 device types per vendor on the market, all featuring a variety of firmware, and two or three vendors in deployment, the numbers are stacking up. Given that each piece of equipment comes with a unique command line or user interface to successfully configure a device, it becomes a nightmare for even the savviest network engineer.

Of course, security must be a major consideration. The security policy of a “vanilla” access list on a router is now a thing of the past. Simple firewalling and access control lists are no longer sufficient; they must be extended with more sophisticated intrusion detection and prevention systems. And using public internet transport for low-cost bandwidth requires additional layers of secured virtual networks.

It’s also now important to consider capacity planning. Many enterprises are now using Software as a Service (SaaS) and Infrastructure as a Service (IaaS) such as AWS and Azure. Network bandwidth usage and flow patterns have significantly changed, and are evolving rapidly with the introduction of additional services such as rollouts of Salesforce or Office 365, creating unique demands on networks that were originally designed for “internal use only” usage and security concepts. The reality is that adding new equipment to today’s complex networks can’t be done overnight.

Toward Automation

By way of alleviating some of the stress, some organizations have implemented a self-service IT portal to provide help desk functionality and auto-ticketing. However, these services are typically reserved for simpler, specific tasks, such as accessing a network drive or adding access to a printer, that don’t require a change to network functions.

Even though a primary goal of IT help desk ticketing systems is to assign incoming change requests to the proper engineer, requests that are handled manually can take three to five days to implement if there is no new equipment. When new equipment like a new circuit or router is involved, turnaround time can take 30 to 60 days or more and may require senior architects and networking engineers to implement.

Clearly, more must be done. Here are five tips to help level the path to automation:

– Take control. Leadership usually defines the strategy, and it relates to network features like scalability, reliability, security etc. Beginning to roll out standard policies in your enterprise, starting from simple enforcement of standards for DNS and NTP to routing and tunneling, will go a long way. These policies define the rules for each so that centralized control and policy management can be enforced. This is the start of taking control of your network.

– Aim for incremental success. Start by automating specific pain points, such as a QoS policy first. A small success will help in the progression of applying an automation strategy when you move to the next more painful solution.

– Create a robust rule set. Abstraction and modeling of standard features and node and site configuration are vendor-agnostic and can be implementedno matter what device you are working on. Consider what you want the network to do, then build it into the rule set to develop your model. Having models for network features, node types, site type and more will help tremendously when implementing the models in an automation/orchestration platform.

– Define normal and deploy changes. The challenging part here is getting started early. Conducting a discovery exercise on an existing network can be complex. Understanding exactly what the current configuration state is, based on an internal configuration management database or existing documentation versus what is really there in terms of validating devices and firmware, can add to this complexity. Once the network is known, it is possible to deploy changes against your modeled functionality and migrate it to the desired state. After this is complete, it is essential to immediately protect the network against unauthorized changes, automatically monitoring the configuration state to ensure no other changes are applied.

– Automate and orchestrate. User requirements and applications using the network are ever-evolving. New sites are being deployed. Devices are being upgraded. To enable and maintain agility to service these requests, centralized model-driven automation and orchestration are necessary. This type of control of the network will ensure new devices are provisioned correctly. Policies and best practices must be maintained throughout the network. To enable this, engineering and operations must not be slowed down by automation and orchestration tools and must enable DevOps to quickly develop, test and deploy new features into the production network.

Uniquely Orchestrated

Managing complexity has become job one for network engineers today. Each network has its own configuration, be it a unique collection of older and newer technologies or a greenfield environment. Best-in-class organizations put all networks’ nodes in a config-synchronized state that falls in line with the approved network model, and using the five tips listed above can greatly help in achieving that.

About the author:

Olivier Huynh Van is the visionary inventor of Gluware technology and leads R&D for Glue Networks. Previously, Olivier was the former CTO of Yelofin Networks, and has 20 years experience designing and managing mission critical global networks for ADM Investor Services, Groupe ODDO & Cie, Natixis, Oxoid and Deutsche Bank. Olivier holds a Master’s Degree in Electronics, Robotics and Information Technology from ESIEA in Paris, France.