Getting Past Denial: Addressing Misconceptions About SSH Key Management

Featured article by Matthew McKenna, Chief Strategy officer, SSH Communications Security

Many organizations are unaware that the back door to their networks is swinging wide open due to poor SSH user key management. When they become aware, they are likely to enter a three-stage process of denial, resistance and adaptation.

Awareness begins with the answer to this question: Who is responsible for the provisioning, revocation and recertification of SSH user key-based access in your organization? If you answered this question with a shoulder shrug or a forced grin, you may be in denial or partial denial – and that can spell disaster for your network.

So, it’s time to take a hard look at what SSH user keys are for and why they need your attention. These keys are a means of authentication, similar in some ways to a password. Whether it is access to our Oracle databases, SAP financial systems, payment processing systems, trading platforms, network devices, Linux-based IOT devices, our new agile DevOps processes or Amazon cloud, SSH and SSH user key-based access is pervasively used.

This access joins processes together to enable application-to-application data transfers, and it is the means by which our administrators and developers can conveniently access their environments without having to retype their passwords each time.

These are powerful capabilities that must be secured. Here are more reasons to be concerned:

– It’s not easy to assess which person or what process an SSH key pair is associated with because keys don’t necessarily correlate themselves to an identity.

– Anyone with an SSH client in your organization and access to a server can generate an SSH key pair.

– SSH user keys don’t expire, meaning they continue to provide access unless they are removed from the locations where they reside.

These easily generated and hard-to-track keys grant access to our most critical infrastructure, meaning that SSH key non-management is completely out of control. But those “shoulder shruggers” who are in denial will say, “If SSH user key-based access is such a big problem, why don’t I hear more about breaches caused by their use?”

Denial Becomes Downfall

Good question. After all, if there are no real-world examples, this is just an exercise in fear-mongering. However, we are seeing breaches in which the misuse of SSH has played a significant role in exfiltration of critical data. In fact, in the Snowden and Sony breaches, SSH user keys played a major role. In both cases, there is significant evidence that SSH user keys were extensively used to gain access to critical servers and data, move laterally to additional assets within the network and exfiltrate data. Poor to non-existent SSH key management and encrypted access control were large contributing factors to both of these major headlines.

So then, if we don’t have any inventory of SSH user keys and are not monitoring their usage and how they are provisioned, revoked and recertified, how would an enterprise know if SSH user keys had been used to gain unauthorized access to servers and exfiltrate data? The fact of the matter is that we wouldn’t and we don’t.

Another nail in the coffin is the fact that many organizations have little to no visibility into their encrypted SSH and SFTP traffic, thereby rendering our intrusion detection systems, data loss prevention systems, anti-virus and SIEM tools ineffective to identify threats in real time where SSH is potentially being used maliciously.

Pushing Past Resistance

Even when organizations let go of denial and admit there’s a problem, the battle isn’t over. Resistance comes in, usually in the form of a sentiment such as:

1. We are moving to the cloud, so this won’t be a problem.

2. Don’t scan my environment to show me the size of the problem, because then I will have to fix it.

3. There are too many stakeholders involved to solve this.

4. I already use Kerberos tickets or another privileged access management solution, so we don’t have a problem.

5. But there are so many other more important initiatives.

Denial prevents progress, and so does resistance. Let’s go through each of these sentiment so we can move to the phase of adapting our organizations to acceptance of this massive hole in our access management postures.

1. We are moving to the cloud, so this won’t be a problem.

If only. SSH will continue to play a key role when it comes to access and the movement of data, even in the cloud. Many large enterprises engage in what is known as a “lift and shift” when it comes to the first step of cloud migration. When it comes to SSH user key-based access, this means that we have simply migrated the access challenges we faced on-premises into our cloud.

In the automation stack of the DevOps process, SSH user key-based access is the developers’ tool of choice to move code from source code repositories like Github, to test and build automation tools like Jenkins, all the way through to how we upload the code to a cloud application. We must take into consideration, in the quickly evolving world of cloud transformation, that developers continue to have access through the supply chain closer than ever to development environments. This access also needs to be controlled effectively.

2. Don’t scan my environment to show me the size of the problem, because then I will have to fix it.

It’s time to wake up and smell the lurking security disaster. A scan of the environment will provide insight into risk around segregation of duty access controls from non-production to production environment. It will demonstrate gaps in how we manage third-party access to our environments. It will give us insight into access that has not been recertified in two or more years and demonstrate where we have key-based access with weak encryption. It will highlight how deeply a malicious actor could penetrate into our environment with a single stolen private key and show us how big the gap is. But the size of the issue is honestly the lesser concern. The greater concern is how we gain control of this access in terms of how we provision it, revoke it and recertify it.

3. There are too many stakeholders involved to solve this.

SSH unites organizations in terms of access and data flows. It cuts across our on-premises assets and is used by third-party suppliers to gain access to our systems remotely and transfer data to our systems. It is used by our developers to move code throughout the DevOps supply chain as we create new applications. It is used to gain access to our cloud and to transfer data and connect applications between our midrange servers and our mainframe.

It’s a positive irony that the problem of SSH-related access that organizations face actually has the chance to bring many organizational functions closer together. First and foremost, SSH user key and encrypted access is an identity and access management issue. However, cryptographic services, security operations and infrastructure all play a significant role in effectively addressing the issue. It’s not really an issue of too many stakeholders; it’s more about identifying who has the ownership to solve the problem.

4. I already use Kerberos tickets or another privileged access management solution, so we don’t have a problem.

Interactive access and role-based access to infrastructure assets must be managed, and

Kerberos tickets and traditional PAM solutions are excellent for this purpose. However, what these solutions do not cover well is the management of automated machine-to-machine and application-to-application-based access. When we look at the greater picture of SSH user key-based access, over 80 percent of the SSH user key-based access in our environments is application to application in nature.

5. But there are so many other more important initiatives.

Are you sure about that? If so, please describe in detail what could be more important than securing the primary back-down in terms of access to our most critical infrastructure. We are often too focused on what is coming through the front door when we should be thinking what is happening at the back door. How can we continue to ignore hundreds of thousands to millions of access credential to our most critical infrastructure? What is more important than securing the crown jewels of our enterprises?

Adapting to a New Security Posture

The reasons for resistance have been addressed, so now it’s time to adapt by adopting

effective ways to gain visibility, control and ongoing governance of our SSH user key-based access.

The first order of business is to solve the challenge of visibility, because in most cases SSH user key-based access has not been part of the overall provisioning processes of organizations’ identity management frameworks. Visibility consists of developing a mapping of private and public keys to their corresponding interactive or application to application connection. This can be achieved through scripted approaches, some of the privileged access management solutions and those dedicated to managing SSH user keys.

Verbose logging, activated in the SSH daemon, can be implemented to gain an understanding of how frequently keys are authenticating and from where. This information is essential because it provides us the needed intelligence as to what key-based access in our environment is valid and what access may no longer be needed. It also can help us identify if private keys are authenticating from IPs, which are not recognized by our network.

One of the main challenges is staunching the flow of unauthorized SSH user key-based provisioning. Through a process called “lockdown,” where authorized keys are relocated from users’ home directories to a central root-owned location, we can create an effect where now only users with root-level access are able to generate new SSH user keys going forward. But be careful here; if you lock down access without having a provisioning process in place, you may frustrate many administrators, developers and application owners in your organization. Communication of the process and oversight is essential to the success of this process.

Automation is mandatory in light of the ongoing and rapid adoption of DevOps and cloud environments; it’s the same for SSH user keys. On average, if a skilled user sets up an SSH user key-based trust and validates that the connection is working, it may take them five minutes. An unskilled SSH user, on the other hand, may take 30 minutes or more. If we do the math, we are generating thousands, tens of thousands or hundreds of thousands SSH user keys a year, and the costs really begin to add up. Automation can be achieved through scripting, solutions dedicated to SSH user key management and even orchestration tools.

It can be a difficult journey to travel from denying a problem to creating a strategy to overcome it. But in the case of SSH user keys, it’s a journey that organizations must take in order to close the dangerous back door to their networks. Fortunately, others have taken this path and created best practices for us to build a solid key management framework, including continuous monitoring and greater visibility. The ability to adapt to the new normal that SSH threats represent will separate the winners from the losers in the security game.

About the author:

Matthew brings over 15 years of high technology sales, marketing, and management experience to SSH Communications Security and drives strategy, key account sales, and evangelism. His expertise in strategically delivering technology solutions that anticipate the marketplace has helped the company become a market leader.

Prior to joining the company, Matthew served as a member of the executive management team of ADP Dealer Services Nordic and Automaster Oy, where he was responsible for international channel operations and manufacturer relations. In addition, he was responsible for key accounts including Mercedes-Benz, General Motors, and Scania CV. Before this, Matthew played professional soccer in Germany and Finland.

Matthew holds a Bachelor of Arts degree in German from the University of South Carolina and an MBA from the Helsinki School of Economics and Business Administration.