In this post, we’ll provide recommendations from the Google Cybersecurity Action Team and discuss solutions available to Google Cloud customers and security teams to manage the risk of the Apache “Log4j 2” vulnerability (CVE-2021-44228).
Please visit Google Cloud’s advisory page for the latest updates on our assessment of CVE-2021-44228, and the potential impact of the vulnerability for Google Cloud products and services.
Background
The Apache Log4j 2 utility is an open source Apache framework that is a commonly used component for logging requests. On December 9, 2021, a vulnerability was reported that could allow a system running Apache Log4j version 2.14.1 or below to be compromised and allow an attacker to execute arbitrary code on the vulnerable server. On December 10th, 2021, NIST published a critical CVE in the National Vulnerability Database identifying this as CVE-2021-44228. The official CVSS base severity score has been determined as a severity of 10. We strongly encourage customers who manage environments containing Log4j to update to the v2.15.0 or take the mitigation actions outlined in this post. The Cybersecurity and Infrastructure Security Agency (CISA) released additional recommendations for immediate steps regarding this vulnerability.
Technical Details
The “Log4j 2 vulnerability” can be exploited by sending specially crafted log messages into Log4j 2. The attacker does this by leveraging the Java Naming and Directory Interface (JNDI) – which is a Java abstraction layer included by default in the Java SE Platform that allows a Java application to retrieve data from directory services (such as LDAP). JNDI uses HTTP URIs to resolve to a specific directory service – and the URI can be adjusted as needed to resolve to the right directory location. Using Log4j 2, an attacker sending a log message with a specially crafted URI can cause the application to execute arbitrary code (such as by providing a Base64 encoded script in the path).
This is due to specific behavior in Log4j 2 that allows for the input of variable data into the log (called Lookups). In a Lookup, the reference is queried and evaluated for input into the log. By using this feature during an exploit, the attacker uses the URI input to instruct Log4j 2 to resolve an object they input (such as an encoded script). This issue has been resolved in Log4j 2.15.0.
Updates and Recommendations to Identify, Detect & Protect
Customers should look to upgrade to v2.15.0 of Log4j as soon as possible. If they cannot upgrade to the updated version quickly, customers should look to mitigate by setting the “No Lookups property (log4j2.formatMsgNoLookups)” to true. Taking these actions (and following additional advice from the Apache foundation) is your best immediate action.
In addition to updating Log4j 2, some Google Cloud Security products can help detect and temporarily mitigate exploitation until the patch can be applied. Additionally, if you have third party WAF, IDS/IPS, and/or NDR solutions deployed in Google Cloud, please consult with your vendors for more guidance related to this vulnerability.
To identify potentially impacted systems we recommend the use of a vulnerability scanner (e.g., Tenable, Qualys or scanners of a similar capability class). These tools have reported identifying the vulnerability referenced in National Vulnerability Database (NVD) and will help to locate impacted systems.
Where possible, we recommend implementing all of the below for a layered defense in depth strategy.
Cloud Armor WAF Log4j 2 Detection and Blocking Rules
Relative to Log4j 2, Cloud Armor helps mitigate threats for applications or services behind external HTTP(S) load balancers. You can enable Cloud Armor through Cloud Console > Network Security, or via API. Cloud Armor’s WAF rules can be configured to detect, or detect and block requests. Cloud Armor customers can now deploy a new preconfigured WAF rule that will help detect and, optionally, block commonly attempted exploits of CVE-2021-44228 while you are patching your systems. To learn more about addressing the Apache Log4j 2 vulnerability with Cloud Armor, please read this blog article.
Chronicle
Threat Hunting and investigation tools can be used to look at historical data and determine if exploitation was attempted – or can be used as vehicles for monitoring active exploitation. Chronicle is Google’s threat hunting tool that provides extended event collection across cloud and on-prem (such as EDR logs, firewall logs, etc). If you are using Chronicle for log ingestion/SIEM and have historical event data stored (Chronicle retains 12 months of data by default) you can search for historical exploit attempts. Customers should search for events that contain “jndi” and a combination of strings that follow including “ldap”, “rmil”, “ldaps”, or “dns”, generated from HTTP requests, which correspond to possible Log4j 2 exploitation patterns.
For example – this syntax could be used in a Yara-L rule:
Chronicle customers can also look at external communication events for low-prevalence destinations that could be an indication of impacted systems reaching out for remote code execution. More information can be found in Chronicle documentation here.
Cloud IDS Network-based Threat Detection
Cloud IDS has been updated to help detect common types of Log4j 2 exploit attempts. These new detections are on by default for any existing or newly added deployments of Cloud IDS. Positive detections will appear in Cloud IDS alert logs, visible in terse mode in the Cloud IDS UI of Cloud Console, and in verbose mode in Cloud Logging, or by API. Cloud IDS is enabled and configured through Cloud Console > Network Security, or via API. Please see this blog for more information.
Cloud Logging
You can use the Logs Explorer to help detect potential attacks on your service exploiting the Log4j 2 vulnerability. If you are using Cloud Logging to log requests to your service, you can check httpRequest fields with user generated content for potential exploits that have string tokens like ${jndi:ldap://.
There are multiple variations on how this vulnerability is being exploited, and there are many ways to use unicode or escapes to avoid detection. This is an introduction to using a regex query to help detect some of the common exploits:
The above query matches many of the obfuscated variations of the string “${jndi:” in HTTP Load Balancer request logs. You can use similar regular expressions to scan request logs in other services by changing the resource.type. The query may take a long time to complete if you are scanning a large volume of logs. To make your queries run faster, you can make use of indexed fields like resource.type, resource.labels, or logName to narrow your query to a set of specific services or log streams.
Detecting matching log entries does not indicate that there has been a successful compromise. It may indicate that someone is probing to exploit the vulnerability within your project or workload. There could also be false positives if your application uses patterns like “${jndi:” in the http request fields.
Cloud Logging query results only include logs that have already been ingested into Cloud Logging and are also within the user specified retention limits. While most Google Cloud services have logs enabled by default, logs that were disabled or excluded will not be included in this search. If you are using an HTTP(S) Load Balancer, logging needs to be enabled for the request logs to be available in Cloud Logging. Similarly, if you have web servers like Apache or NGINX running on a VM, but have not installed the logging agent, those logs will not be accessible within Cloud Logging.
We will continue to actively monitor this event and will provide updates to this blog post on relevant mitigation steps and detection mechanisms. Please visit our security advisory page for updates on our Google Cloud security assessment.
Cloud BlogRead More
