On Friday 10 December and Monday 13 December, Pinewood issued bulletins about a critical vulnerability that exists in Log4j, a Java-based logging framework which is used by many different applications and websites. Since then, exploitation of the vulnerability has continued and, in this bulletin, we would like to share an update on the insights that we’ve gained over the past few days.
The vulnerability
For more information on the vulnerability, see our previous bulletins. In conclusion a critical vulnerability in Log4j version 2, which is part of many different software solutions, allows attackers to remotely compromise vulnerable systems by injecting malicious content into the logs of these systems. If an attacker succeeds in inserting a specific variable into the log, he can then initiate a so-called Java Naming and Directory Interface (JNDI) lookup which can then be used to load and execute the malicious content from the remote (attacker controlled) server.
Affected Products and Solutions
As there are many vulnerable software solutions, it’s impossible to sum up all of these solutions, let alone the patches that have been released to fix this issue. As this situation is quickly evolving, Pinewood therefore advises to regularly check the articles from the different vendors that are used in your environment for the status and available patches. In addition, we advise you to regularly check the overview from the Dutch National Cyber Security Center (NCSC-NL) which is continuously updated and very complete; you can find the overview here.
Attention points
There are several attention points that we’ve witnessed over the last few days that we would like to share:
- Scanning for vulnerable systems is very challenging. Our customers have asked us to launch vulnerability scans against their networks and we found that running uncredentialed scans against these networks delivers highly unreliable results, irrespective the type of vulnerability scanner used. These scans do rely on specific callbacks once the vulnerability is triggered. However, these triggers are very application-specific (e.g. malicious content should be injected in specific fields, specific pages on a website should be contacted, etc.) and therefore require specific vulnerability scanning plug-ins for specific software solutions.
- One of the work-arounds is not effective. Two workarounds were previously advised by Apache to prevent exploitation of the vulnerability: 1) to change the system property log4j2.formatMsgNoLookups or the environment variable LOG4J_FORMAT_MSG_NO_LOOKUPS to TRUE or 2) to remove the JndiLookup class from the classpath. It turned out that the first work-around does not prevent exploitation of the vulnerability in all cases and should therefore not be relied upon. Apache now only advises the second workaround.
- Broad scale scanning continues. We see continuous efforts to find vulnerable systems by injecting malicious strings into requests towards internet-facing IP addresses. If a vulnerable system is/was connected to the internet, it is highly likely that it was hit by such a scan.
- Limiting outbound communication can greatly reduce the risk of exploitation attempts. The exploit attempts that we’ve seen, rely on connections back to attacker-controlled servers. If a vulnerable machine can only receive connections from internet-connected systems but cannot initiate connections back, this can greatly reduce the chances of a vulnerable systems getting compromised.
- New exploits are quickly used once they are published. As described earlier, the attack vector may vary amongst different software solutions. Exploits showing how a vulnerability can be exploited in a specific software solution are regularly published on the internet. As soon as such an exploit is published, it will only take a few minutes or hours before active use of it starts.
Advise
Based on our experiences with this vulnerability and its exploitation attempts we advise you the following:
- Compile an inventory of systems running Log4j version 2. Start with internet-facing systems first and then continue with the systems that are not directly internet-connected. Create this overview by issuing specific commands on all your systems, by running a credentialed vulnerability scan against your systems or by utilizing existing tooling within your network that has software inventory capabilities (e.g. Defender for Endpoint). If you prefer to run commands, you can use these:For Windows (Powershell):
gci ‘C:\’ -rec -force -include *.jar -ea 0 | foreach {select-string “JndiLookup.class” $_} | select -exp PathFor Linux (find):
find / 2>/dev/null -regex “.*.jar” -type f | xargs -I{} grep JndiLookup.class “{}” - Disconnect vulnerable systems from the internet ASAP. Because scans happen continuously and around the clock, you should assume that the vulnerability was already exploited or will be exploited very soon. It is therefore essential to disconnect a vulnerable system from the internet ASAP or – if this is not possible – either install the patch provided by your software vendor or implement the workaround if such as patch is not yet available.
- Check your logging on vulnerable systems. Because attacks rely on injecting malicious payloads in your logs, you should be able to find these attacks by checking your logs for signs of attacks. Logs of attacked systems typically contain entries with a string starting with “${jndi:“ and finding such a string in your logs is an indicator of attack. Be reminded that having such a string in your logs does not mean that your system is compromised per se, it’s “just” an attempt to do so that should be investigated further.
Questions
If you have any questions regarding this issue please contact Pinewood Support by phone 015 251 36 33 or via e-mail support@pinewood.nl.
