How Hackers Prevent Linux Malware From Being Removed

Malware On Linux

Background

Here at Quttera, we clean infected websites on a daily basis and this weekend our automated cure procedure failed to remove malicious files. The customer actually had more than 170 websites on the Linux server which was heavily infected. The automatic cure is executed with root permissions so we started investigating what was this all about.

Investigation 

We logged-in into the infected server and executed few commands with root user such as 

and we got a "Permission denied".

At the beginning we thought that there is an issue with a parent directory permissions but after a quick verification we confirmed that the permissions were OK.

The next step was looking for a running process prevents a file removal, however the lsof output did not confirm this.

It is not a surprise that hackers are constantly changing tactics and improving techniques. This allows them to:

1. Keep an infection undetected on a compromised server or website as much as possible. 
2. Make it harder to cleaned/remove it. 

And one of the main challenges is staying on top of those threats as they evolve.

We continued with further investigation and the next step was to use lsattr command which lists file attributes on a Linux second extended file system. We executed the lsattr command on the locked file and the file had "a" and "i" attributes.

You can always search for the lsattr command description through 'man' in Linux. So, we came across chattr command that led us to the following:

1. chattr change file attributes on a Linux second extended file system
2. File with the "i" attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file and no data can be written to it. Only a super-user or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear the attribute.
3. File with the "a" attribute set can be opened in append mode only for writing. Only the super-user or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear the attribute.

So to make the long story short, the solution was to run:

and

which allowed to remove file without any problem

What is "Linux second extended file system"?

The second extended file system (or Ext2) is Ext4 file system grandparent used by Linux kernel.
The Ext2 used as default file system by several Linux distributions and it was replaced by Ext3.
You can find more information about the Ext2 here

If you suspect your website was compromised or would like us to remove the malware, please select from ThreatSign - website monitoring and malware clean-up plans. To run free remote scan of your websites: http://quttera.com/website-malware-scanner

For other questions, do not hesitate to contact Quttera's help-desk.