In December 2019, Citrix announced that their flagship product, Citrix Application Delivery Controller (ADC) and Gateway, had a vulnerability that would allow code execution to take place on affected devices without any authentication. This vulnerability (designated CVE-2019-19781) was severe - on a scale of 1 to 10 it was deemed a 9.8 meaning that an attacker able to exploit this vulnerability could do serious damage. An estimated 80,000 companies were affected, and by the time you read this piece, most of them will already have been attacked.
What’s involved?
The Citrix ADC and Gateway are also known to many people as the Netscaler ADC and Netscaler Gateway. ADCs are used in enterprise environments to improve the performance of applications by, for example, load balancing traffic, caching data and handling file compression. The Citrix Gateway traditionally provided remote access to the Citrix environment but is now commonly used to provide VPN access to various products, including the ADC management portal. As a result, these two products are often bundled together when purchased, and hosted on the same machine in production environments.
The vulnerability lies in this access point and allows attackers to access files on the host machine without authentication. As the host machines are on-premise, gaining access to them gives attackers a foothold in the corporate network.
What is this vulnerability?
In many operating systems, directories can be navigated using dots and slashes:
At the heart of the Citrix issue is a directory traversal vulnerability. This is where the web server does not properly check the file paths requested by users, allowing attackers to request files and folders they were never meant to access.
How is this exploited?
To exploit this vulnerability, attackers first scan the internet for Citrix Gateways. When they find one, they are greeted with a screen asking them to login:
The Citrix exploit specifically takes advantage of Perl scripts in the Citrix ADC which allow file uploads. Attackers make a POST request to the vulnerable Perl Scripts via the directory traversal vulnerability, the parameters of which contain malicious commands. The result is a malicious file placed on the host machine.
A GET request for that file will cause the system to execute it. The file can contain code that does anything from opening reverse shells to launching ransomware attacks.
Exploit scripts can be downloaded from Github allowing anyone with basic technical skills to use this vulnerability successfully. Furthermore, exploits have been folded into the popular pen-testing toolkit Metasploit, allowing adversaries to automate most of the exploitation steps.
What is the mitigation?
Happily, when Citrix published the vulnerability, they also released some code which can be used to mitigate these attacks. It is critical that system administrators run the mitigation command on their vulnerable Citrix servers.
The mitigation specifically aims to block HTTP requests which include “/vpns/” or “/../”. URLs containing those strings are returned a 403 Error.
Citrix has committed to deploying a patch by the end of January 2020. A patch schedule has been published with the earliest patches becoming available on the 20th of January.
Is mitigation enough? (Hint: It’s not)
The reality is that if you were vulnerable, there is a significant possibility that you have already been compromised. Many security researchers have noted that as soon as the Citrix mitigation was released in mid-December, it was fairly easy for skilled hackers to reverse-engineer an exploit. Therefore, while an exploit was first made public in January, it is likely sophisticated attackers were using this vulnerability at least since December. Furthermore, these vulnerabilities have been in the Citrix code base since Citrix NetScaler ADC and NetScaler Gateway version 10.5 which was released in 2014. This makes this vulnerability 5 years old; numerous attackers could have found this problem and successfully used it to compromise networks.
A prevailing mindset among many is that if you have employed the mitigation, you are safe. It cannot be stressed enough that this is not the case. Applying the mitigation after you have already been hacked does not kick the attacker out of your network, it does not return stolen files and it does not close malicious backdoors to your network. Once attackers use this vulnerability to get in the door, they use other tactics to set up shop and stay connected to your environment even after you apply the mitigation and the patch. Therefore, beyond mitigation and eventual patching, organisations are also strongly advised to do some forensic analysis to figure out if they have been compromised.
How do I find out if I have a problem?
If your network has perimeter defenses in place (e.g. IDS, IPS, Firewalls), logs should be checked thoroughly for HTTP Requests involving the string “/vpns/” or directory traversal attempts. Additionally, the Citrix ADC’s web server keeps logs in /var/log/httpaccess.log. The content of HTTP and HTTPS GET and POST requests are stored in plain text so these logs can be investigated for signs of infiltration as well as suspect activity such as the injection of XML or Perl scripts.
Secondly, you can look for suspicious file changes, particularly in the /var/tmp/netscaler/ folder. Several legitimate Perl scripts can be found in these folders, but these may have been overwritten by attackers while new scripts may have been added. Users should test hashes of these files to ensure they are legitimate. Security Researcher x1sec released a list of valid Perl files and their MD5 hash, which can be used as baseline to discover what may have been compromised.
This CVE requires more than just mitigation and patching, it requires system owners to take an active role in determining if they have been compromised in the first place. While some attacks will have clear indicators of compromise, others may be more subtle and require more work to discover. If you find yourself needing assistance in determining your exposure, ThreatSpike is always here to help.