Introduction
The auditd utility can be an extremely valuable tool for monitoring what's happening on your Linux server. It is designed to integrate with the kernel and monitor system calls that the system makes.
In this guide we will learn how to check if auditd is installed, install it if it is not, check to make sure the daemon is running, create a simple audit rule, and check the logs to see what our example rule detected.
Checking for, and installing, auditd
Your cPanel server should already have a couple audit-libs* packages installed, but it will not have the actual audit package that provides the daemon that we need.
A default installation should look like this:
Since it is not installed on this server, let's install it:
Now we can see the actual audit package:
This will install the daemon process and it will be enabled to start on server reboot, but it will not be running:
So let's start it so it will be running when we create our rules:
Adding rules
Now that we have started the service, the 'auditctl' command can be used to list/create/remove rules:
So let's create our first rule, which takes the format of:
Let's add a rule to watch /etc/passwd for writes(w) or attribute changes(a) and set the search key to 'passwd' to make it easy to find the entries regarding the /etc/passwd file being changed:
Now we can see the rule with auditctl and within the audit.rules file:
Note: This rule will exist as long as the server is not rebooted. If you wish to make the rule permanent, you need to add it to the /etc/audit/rules.d/audit.rules file. This is how it will look by default:
To add our example rule, simply add it to the bottom of the file in the exact same way that it appears when running 'auditctl -l':
Searching audit logs
The 'ausearch' command is used to search the audit logs, so we'll use the following to search for the "passwd" search key:
That entry is showing the rule was added and it is currently the only entry since /etc/passwd hasn't been modified since we created this rule.
Now let's add a cPanel account to the server so we can see how that operation modifies /etc/passwd:
The "-i" will print the actual named system call, rather than the number, making it far easier to know what system calls are taking place. The entries we're most concerned with here are the second and last entries. Within those, specifically we're looking mostly at the "syscall" and "exe" values.
The second entry shows that /etc/passwd was opened using the "open" system call ("syscall=open") to the "whostmgr5" executable file ("exe=/usr/local/cpanel/whostmgr/bin/whostmgr5")
A system call is how a program interacts with the kernel, so these are important to look at because they tell us whether a program is requesting to, for example, open a file, write to a file, read from a file, etc.
The last entry then shows that there was a "rename" call also by the same WHM process, and looking a little above that, you can see that a temporary file had been created "name=/etc/passwd.work.d21f6ce1" then it was renamed to /etc/passwd, thus replacing the old file with the new one.
Knowing that this change was done by the WHM process, one could then check cPanel's access log (/usr/local/cpanel/logs/access_log) to find the IP address and username that made the request that caused this file to be written to:
(The time difference comes from the access_log being written to in UTC and the audit logs using the system's timezone which is set to US/Central time so there's a 5 hour difference)
So we now know that on the morning of October 17th, someone at 10.7.65.81 logged into WHM as root and made a POST request to wwwacct which modified the /etc/passwd file. In this specific instance, one could then reference the /var/cpanel/accounting.log file and see the account that was created:
Deleting rules
Lastly, once you've gathered whatever information you're after with your audit rule, you should delete this watch rule using the same format that was used to add it, except replacing -w with -W:
The above command will remove the rule from the current running audit session. You will also need to remove the entry from /etc/audit/rules.d/audit.rules if you added it there.
It is important to delete the rules once they've served their purpose, as audit logs can potentially grow very large in size if they're left to run for extended periods of time.
The auditd utility can be an extremely valuable tool for monitoring what's happening on your Linux server. It is designed to integrate with the kernel and monitor system calls that the system makes.
In this guide we will learn how to check if auditd is installed, install it if it is not, check to make sure the daemon is running, create a simple audit rule, and check the logs to see what our example rule detected.
Note: Auditd requires access to the kernel which is not available in containers such as Virtuozzo.
You can install the package but attempts to start it will fail due to not having direct kernel access:
The systemd journal will show:
You can install the package but attempts to start it will fail due to not having direct kernel access:
Code:
# systemctl start auditd.service
Job for auditd.service failed because the control process exited with error code. See "systemctl status auditd.service" and "journalctl -xe" for details.
Code:
Oct 17 08:37:07 <HOSTNAME> auditd[4819]: Cannot change priority (Operation not permitted)
Oct 17 08:37:07 <HOSTNAME> auditd[4819]: The audit daemon is exiting.
Oct 17 08:37:07 <HOSTNAME> systemd[1]: auditd.service: control process exited, code=exited status=1
Oct 17 08:37:07 <HOSTNAME> systemd[1]: Failed to start Security Auditing Service.Your cPanel server should already have a couple audit-libs* packages installed, but it will not have the actual audit package that provides the daemon that we need.
A default installation should look like this:
Code:
#rpm -qa | grep audit
audit-libs-2.8.1-3.el7_5.1.x86_64
audit-libs-python-2.8.1-3.el7_5.1.x86_64
Code:
# yum install audit
Loaded plugins: fastestmirror, universal-hooks
Loading mirror speeds from cached hostfile
* EA4: 208.74.121.35
* cpanel-addons-production-feed: 208.74.121.35
* epel: fedora-epel.mirror.lstn.net
* remi-safe: repo1.dal.innoscale.net
Resolving Dependencies
--> Running transaction check
---> Package audit.x86_64 0:2.8.1-3.el7_5.1 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
=============================================================================================================================================================================
Package Arch Version Repository Size
=============================================================================================================================================================================
Installing:
audit x86_64 2.8.1-3.el7_5.1 updates 247 k
Transaction Summary
=============================================================================================================================================================================
Install 1 Package
Total download size: 247 k
Installed size: 621 k
Is this ok [y/d/N]: y
Downloading packages:
audit-2.8.1-3.el7_5.1.x86_64.rpm | 247 kB 00:00:00
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : audit-2.8.1-3.el7_5.1.x86_64 1/1
Verifying : audit-2.8.1-3.el7_5.1.x86_64 1/1
Installed:
audit.x86_64 0:2.8.1-3.el7_5.1
Complete!
Code:
# rpm -qa | grep audit
audit-libs-2.8.1-3.el7_5.1.x86_64
audit-2.8.1-3.el7_5.1.x86_64
audit-libs-python-2.8.1-3.el7_5.1.x86_64
Code:
# systemctl status auditd.service
● auditd.service - Security Auditing Service
Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Code:
# systemctl start auditd.service
# systemctl status auditd.service
● auditd.service - Security Auditing Service
Loaded: loaded (/usr/lib/systemd/system/auditd.service; enabled; vendor preset: enabled)
Active: active (running) since Wed 2018-10-17 08:34:59 CDT; 1s ago
Docs: man:auditd(8)
https://github.com/linux-audit/audit-documentation
Process: 27739 ExecStartPost=/sbin/augenrules --load (code=exited, status=0/SUCCESS)
Process: 27729 ExecStart=/sbin/auditd (code=exited, status=0/SUCCESS)
Main PID: 27731 (auditd)
CGroup: /system.slice/auditd.service
└─27731 /sbin/auditd
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: lost 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: enabled 1
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: failure 1
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: pid 27731
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: rate_limit 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog_limit 8192
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: lost 0
Oct 17 08:34:59 <HOSTNAME> augenrules[27739]: backlog 1
Oct 17 08:34:59 <HOSTNAME> systemd[1]: Started Security Auditing Service.Now that we have started the service, the 'auditctl' command can be used to list/create/remove rules:
Code:
# auditctl -l
No rulesSo let's create our first rule, which takes the format of:
Code:
# auditctl -w PathToFile -p Permissions -k SearchKey
Code:
# auditctl -w /etc/passwd -p wa -k passwd
Code:
# auditctl -l
-w /etc/passwd -p wa -k passwd
Code:
# cat /etc/audit/rules.d/audit.rules
## First rule - delete all
-D
## Increase the buffers to survive stress events.
## Make this bigger for busy systems
-b 8192
## Set failure mode to syslog
-f 1
Code:
# cat /etc/audit/rules.d/audit.rules
## First rule - delete all
-D
## Increase the buffers to survive stress events.
## Make this bigger for busy systems
-b 8192
## Set failure mode to syslog
-f 1
-w /etc/passwd -p wa -k passwdSearching audit logs
The 'ausearch' command is used to search the audit logs, so we'll use the following to search for the "passwd" search key:
Code:
# ausearch -k passwd
----
time->Wed Oct 17 08:45:23 2018
type=CONFIG_CHANGE msg=audit(1539783923.295:77): auid=0 ses=51297 op=add_rule key="passwd" list=4 res=1Now let's add a cPanel account to the server so we can see how that operation modifies /etc/passwd:
Code:
# ausearch -i -k passwd
----
type=CONFIG_CHANGE msg=audit(10/17/2018 08:45:23.295:77) : auid=root ses=51297 op=add_rule key=passwd list=exit res=yes
----
type=PROCTITLE msg=audit(10/17/2018 09:04:15.572:155) : proctitle=whostmgr5 - wwwacct
type=PATH msg=audit(10/17/2018 09:04:15.572:155) : item=1 name=/etc/passwd inode=1308128 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.572:155) : item=0 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=CWD msg=audit(10/17/2018 09:04:15.572:155) : cwd=/usr/local/cpanel/whostmgr/docroot
type=SYSCALL msg=audit(10/17/2018 09:04:15.572:155) : arch=x86_64 syscall=open success=yes exit=10 a0=0x4ffa230 a1=O_RDWR|O_CREAT a2=0644 a3=0x7f9899e7cedd items=2 ppid=32113 pid=32209 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=whostmgr5 - www exe=/usr/local/cpanel/whostmgr/bin/whostmgr5 key=passwd
----
type=CONFIG_CHANGE msg=audit(10/17/2018 09:04:15.573:156) : auid=unset ses=unset op=updated_rules path=/etc/passwd key=passwd list=exit res=yes
----
type=PROCTITLE msg=audit(10/17/2018 09:04:15.573:157) : proctitle=whostmgr5 - wwwacct
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=4 name=/etc/passwd inode=1308152 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=3 name=/etc/passwd inode=1308128 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=2 name=/etc/passwd.work.d21f6ce1 inode=1308152 dev=fd:01 mode=file,644 ouid=root ogid=root rdev=00:00 objtype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=1 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=PATH msg=audit(10/17/2018 09:04:15.573:157) : item=0 name=/etc/ inode=68 dev=fd:01 mode=dir,755 ouid=root ogid=root rdev=00:00 objtype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0
type=CWD msg=audit(10/17/2018 09:04:15.573:157) : cwd=/usr/local/cpanel/whostmgr/docroot
type=SYSCALL msg=audit(10/17/2018 09:04:15.573:157) : arch=x86_64 syscall=rename success=yes exit=0 a0=0x580a770 a1=0x6a583f a2=0x7f989a106980 a3=0x3 items=5 ppid=32113 pid=32209 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=whostmgr5 - www exe=/usr/local/cpanel/whostmgr/bin/whostmgr5 key=passwdThe second entry shows that /etc/passwd was opened using the "open" system call ("syscall=open") to the "whostmgr5" executable file ("exe=/usr/local/cpanel/whostmgr/bin/whostmgr5")
A system call is how a program interacts with the kernel, so these are important to look at because they tell us whether a program is requesting to, for example, open a file, write to a file, read from a file, etc.
The last entry then shows that there was a "rename" call also by the same WHM process, and looking a little above that, you can see that a temporary file had been created "name=/etc/passwd.work.d21f6ce1" then it was renamed to /etc/passwd, thus replacing the old file with the new one.
Knowing that this change was done by the WHM process, one could then check cPanel's access log (/usr/local/cpanel/logs/access_log) to find the IP address and username that made the request that caused this file to be written to:
Code:
10.7.65.81 - root [10/17/2018:14:03:52 -0000] "POST /cpsess3914673904/scripts5/wwwacct HTTP/1.1" 200 0 "https://<HOSTNAME>:2087/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_6)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36" "s" "-" 2087So we now know that on the morning of October 17th, someone at 10.7.65.81 logged into WHM as root and made a POST request to wwwacct which modified the /etc/passwd file. In this specific instance, one could then reference the /var/cpanel/accounting.log file and see the account that was created:
Code:
Wed Oct 17 09:04:16 2018:CREATE:root:root:auditdexample.tld:172.16.0.93:auditdexampleLastly, once you've gathered whatever information you're after with your audit rule, you should delete this watch rule using the same format that was used to add it, except replacing -w with -W:
Code:
# auditctl -W /etc/passwd -p wa -k passwd
# auditctl -l
No rulesIt is important to delete the rules once they've served their purpose, as audit logs can potentially grow very large in size if they're left to run for extended periods of time.