Escaping VMware’s NSX Edge OS Jailed Shell

Intro

In this blog post we provide details about a CLI injection vulnerability that we discovered during a penetration testing engagement against VMware Cloud Director. The vulnerability was identified on NSX Edge OS version < 6.4.13.

According to VMware's advisory the following products are affected by this vulnerability:

• NSX Data Center for vSphere [version <6.4.13]
• Cloud Foundation (NSX-V) [version 3.x]

VMware Cloud Director is a platform that allows the management of large-scale cloud infrastructures. It is possible to create Virtual Data Centers (VDC) hosted on a physical data center or on multiple data centers distributed across different geographical locations. Moreover, it allows to provision network, storage and compute resources in a flexible fashion, allowing easy scalability and resilience.

The following diagram shows how the VMware components work together to provide cloud services and shows where VMware Cloud Director stands with respect to the other components.

source

From the diagram, vCenter is the component that manages the NSX Data Center for vSphere virtual infrastructure hosts, virtual machines and virtual resources (CPU, memory, storage, and network).

vCloud Director (vCD) sits higher in the cloud infrastructure: it can traverse multiple virtual infrastructures and further abstract virtual infrastructure resources into software-defined storage, networking, security, and availability.

One of the components that allows network management for a VDC is the NSX Edge Router, which is a virtual router running on a Linux kernel that implements all the common features of a real physical router. The purpose of this router is to allow communication between the VDC and the outside world, standing on the edge of the tenant network. In VMWare Cloud Director, users with “Organization Administrator” privileges are allowed to enable SSH on the NSX Edge Router, which would grant them access to a restricted Linux shell that they can use to configure the router’s services.

It is clear that the NSX Edge Router is a key component of the cloud network infrastructure and it is important to harden and protect it from attackers since a compromise of the router might lead to the compromise of internal resources sitting behind it.

NSX Edge Router - jailed shell

The NSX Edge Router runs on NSX Edge OS which implements a jailed shell on top of the standard Linux shell; meaning that it only allows the execution of a subset of standard UNIX commands as well as custom configuration commands that are needed for network management. All other commands are blocked.

A jailed shell is a command shell on a computer system that exists on an isolated subset of the larger system. In essence, it provides a "system within a system" which protects the containing system from being adversely affected by anything that occurs in the subsystem.

source

The NSX Edge shell has different modes that grant different permissions to the operator.

The following table summarises the CLI command modes:

source

For the purpose of this blog post only the following two modes are of interest.

• Basic. Basic mode is a read-only mode. To access configuration commands, you must enter privileged mode.
• Privileged. Privileged mode commands allow support-level options such as debugging and system diagnostics.

It must be noted that all the default NSX modes are restricted. They only allow network management commands that interact with the running services using script helpers (special commands).

Other than the default modes, there is also an additional mode called Engineering mode. This mode is used for technical support and is therefore not listed in the command modes. This mode exposes a fully functional UNIX root shell. VMWare advises to “use this mode only in conjunction with a support call to prevent breaking the virtual infrastructure” (https://kb.vmware.com/s/article/2149630). This mode is disabled by default.

If engineering mode is not enabled, users are not permitted to execute any low level UNIX commands/diagnostics on the underlying operating system or to make changes to the appliance.

Jail Escape

During the engagement, we identified that it was possible to escape from a restricted SSH shell provided by the NSX Edge Router to a root shell on the underlying operating system.

This not only escapes the jailed shell but also completely bypasses engineering mode if it is enabled.

To exploit the issue, an attacker would need network access to the NSX Edge router’s SSH service and valid (non-root) credentials. The latter could potentially be retrieved via a brute force or guessing attack, identified in the target network, intercepted, etc.

After authenticating to the SSH server, the user can execute commands in a jailed shell as explained above.

We identified that a number of script helpers (commands) were vulnerable to OS command injection; thus allowing an attacker to obtain a fully interactive shell.

In order to obtain an interactive shell, the attacker would append the following to any of the vulnerable commands:

The following commands were identified to be vulnerable and could be abused by appending the string above to obtain an interactive root shell:

In addition, the following vulnerable command could be abused to obtain an interactive unprivileged (admin – not root) shell:

After running some of the aforementioned commands, the terminal needed to be reset (using the reset command) to allow correct handling of shell input and output and be able to use a new interactive shell.

If an attacker gains SSH access to a device running a vulnerable version of the NSX Edge Operating System, they could exploit this vulnerability to obtain root access to the underlying operating system.

The previous screenshot showcases the exploitation of the vulnerability from an unprivileged shell with engineering mode disabled.

Impact

As mentioned, for this vulnerability to be exploitable:

• The attacker must be able to reach the SSH service.
• The attacker must have valid credentials for any account on the device. Although it is not necessarily trivial to obtain these, if weak / guessable credentials are in place or if the credentials are obtained via some other attack vector, exploiting this vulnerability could be possible.

Exploitation of this issue could have adverse effects on the confidentiality, integrity and availability of the systems. For example, it could lead to:

• Unrestricted access to the underlying operating system
• Unrestricted network access to virtual servers, including network traffic capture and potentially MITM attacks
• Installation of malicious software on the virtual device

Mitigation

Since we reported the vulnerability to VMware, it has been assigned a CVE number (CVE-2022-22945) and VMware advisory (VMSA-2022-0005).

The vulnerability has been patched in version 6.4.13 of VMware NSX Data Center.

Therefore, the best way to remediate the issue is to patch the affected system.

Additionally, it is important not to expose the SSH service running on the NSX Edge router to the Internet. Management ports should only be accessible to staff who require them to administer the device. IP restrictions to only allow trusted addresses should be used if the virtual device needs to be managed over the Internet.