Stages of Network Automation

Featured article by Leon Adato, Head Geek^™, SolarWinds

Larry Wall, the creator of Perl^™, once said the three attributes of a great programmer are laziness, impatience, and hubris. Automation allows you to indulge in all three, as long as you don’t mind the work, know that it may take some time to perfect, and can admit that you may not be as good at automation when you first start out as you think you’ll be.

Benefits of Automation

Sarcasm aside, there are always benefits to automation of any kind, beyond the “do my work for me, robot” trope that continues to plague the technology industry. The process of automating often forces elements of standardization and consistency into the environment, because you can’t automate something that happens irregularly. Put simply, in the process of automating, you discover and identify the consistency and standardization within your organization, or you enforce both of these things.

Specific to networking, automation tees up the mindset and the technical framework for larger-scale enhancements, such as software-defined networking (SDN). By creating an environment that can be automated—for example, rolling out 27,000 devices at once through automated processes rather than individually—you also improve your chances at a more stable and secure environment, counterintuitive as that may sound.

Put simply, if you’re automating a response to an issue, it needs to not only be fixed, but also understood and explained. And because you’re automating away the drudgery that occurs every day, you can make the decision not to deal with an issue that crops up at 2 a.m.; rather, you can deal with it when you arrive at the office at the beginning of the day, when you’re settled into work (and caffeinated). In addition, you can automate alerts to provide information when a security-related issue appears, so you can either decide to automate the solution or deal with it separately.

A driving force in the implementation of automation, however, is understanding when it’s a boon and when it’s a drawback.

Is Automation Always in Vogue?

Before even thinking about automating your problems away, you must already be leveraging a thorough monitoring solution that provides detailed network alerts and information. If you can say that’s not something currently being implemented in your environment, hold off until that can be remedied. Moreover, the importance of a strong network team cannot be overstated. There are several instances where this team is critical to seeing automation through:

1. If you have a monitoring solution, but your team is constantly in firefighting mode, don’t try to start automation processes, because you’ll end up being unsuccessful.

2. If there’s high turnover on the network team, automation will not work. The process requires organizational memory—“this issue happened 27 times last year; we should automate a solution”—and it can be difficult to architect a solution properly if you can’t compare past and present.

3. Finally, and perhaps most importantly, if your team isn’t clear on how firewalls—or wireless, or routing, or switching, etc.—work, raise the need for education. A cautionary tale for itinerant automation enthusiasts is The Sorcerer’s Apprentice. Put simply, if you don’t understand the environment, you may flood the sorcerer’s network.

Key Considerations

Now that we’ve determined when it makes sense to implement network automation practices, there are two key considerations to reflect on. Firstly, consider the bang you’re getting for your buck, both in dollars and in effort. For example, automating unforeseeable black swans, such as the Dyn^® DDoS attack, is unlikely to yield results due to the rarities of these events. The lesson here is not to spend time and effort automating an unlikely event. What is worth the time and effort, however, is to automate an often-recurring process, such as when disks are full.

The second key consideration is that the benefits of automation must be monitorable and measurable in order to justify why you’re spending time on it. This is especially true of networks, because they’re difficult to monitor and it can be difficult to know what the improvements in the environment are without a comprehensive solution to provide empirical evidence. If your automation is invisible at layers one through four of the OSI model, it needs to be revisited.

Stages and Best Practices of Network Automation

With these key considerations in mind, let’s talk about the steps your organization can take to move towards network automation: create, test, conduct more tests, and finally, test. These may seem a tad redundant, but it’s critical to understand what problem or action you’re attempting to automate. Here are some helpful tips:

– Gain understanding. If you’re automating a response to a problem, ensure you understand exactly what the steps are to remediate, alleviate, and/or inform when the problem occurs. In many cases, the problem can’t be fixed or avoided without a human. And while alleviating and remediating are self-explanatory for IT professionals who need to find a solution to a problem, informing is less obvious; it refers to supplemental information provided by the automated report that helps you make a decision. If the automated process serves up valuable information and metrics at the time of the event (e.g., there were these top ten processes taking place on the server, top ten protocols, route tables on that device right now, etc.) that will get injected into the ticket, report, or documentation, it’s incredibly beneficial for you, the one who needs to decide whether to fix the issue right now, or three hours from now.

– Conjure up the problem or event. Next, recreate the event at will during testing. If you’re unable to do that, your managers or team may make the argument that you don’t understand the problem in the first place.

– Test it on one system. When it’s successfully recreated to match the real-world event, test it on a single, real system to make sure it follows the Hippocratic Oath: do no harm.

– Expand the scope of systems. If there was no harm done in the single test, roll it out to a smaller pilot set of systems. Then, you stop. Be patient, and wait for it to occur again in the wild to test your theories. With frequency in mind and focusing on time and effort, if you picked the right problem or event to automate, it will occur again that same day, or a few days later.

– Monitoring is key. Lastly, monitor the event or problem so you can clearly show improved metrics. It’s especially powerful if an automated process can save an hour a day, or 365 hours per year. This is also where the comprehensive monitoring toolset plays a role—if management can’t see documented evidence of improvement due to automation, it’s unlikely to take root.

Conclusion

Although the steps and best practices above are important for making informed decisions about events on the network, it still requires you and your team to investigate and address the root cause the next morning. And as automation becomes more advanced, the future of the technology will take many different forms. The simple answer to the future of network automation is of course SDN, although the reality is that many organizations don’t need it to be successful.

The more complex, yet more factual uses of automation are: configuration centric, which deals with backing up, comparing, alerting, and automatically correcting configurations; performance-centric, which is similar to SDN and deals with actions like pushing a traffic shaper down to a router when you notice traffic patterns change, or automatically quarantining a new and unexpected device on the network; and finally, cloud-centric, which is associated with Platform-as-a-Service, and automates paths and connections between cloud-based assets and internal assets.

Ultimately, good network automation habits are enabled by good monitoring. When done correctly, it’s automation the way automation is meant to work.