Assumptions: Everything here is tested on Red Hat/Cent OS 5, and 2.6.24.XX kernel
auditd is the userspace side of kernel auditing functions. It can be used to watch file accesses, monitory system calls, log events etc.The closest to a home page it has
http://people.redhat.com/sgrubb/audit/. It has some nice presentations done at different Red Hat conferences and some other articles.
Installation
Installing auditd is straightforward but to use it efficiently you need to tune it to your needs. To install use
yum install audit
To start the service do
/etc/init.d/auditd start
Configuration
Installation is a breeze but configuring to your needs requires some tweaking.
auditd.conf can be used to tweak auditd's behaviour. Common settings to review/tune are :-
- log_file (default:/var/log/audit/audit.log) & log_format(default:raw)
Set the log file name and the format. The format can be either raw or nolog.
- num_logs(default:0)
Auditd does its own rotation and the default is no rotation. This keyword specifies the number of log files to keep if rotate is given as the max_log_file_action. Note, there is a small catch to this setting. Turns out when Auditd is rotating logs it won't process any logs. So on a high volume server you might miss logs during rotation. To avoid this raise the kernel backlog buffers by adding -b <num> to /etc/audit/audit.rules. The default is 64 1K buffers. This cannot be greater than 99.
- max_log_file (default:5MB) & max_log_file_action (default:ROTATE)
The first sets the maximum log size in megabytes. Action defines what to do when it reaches that state. Valid values are ignore, syslog, suspend, rotate and keep_logs. If set to ignore, the audit daemon does nothing. syslog means that it will issue a warning to syslog. suspend will cause the audit daemon to stop writing records to the disk. The daemon will still be alive. The rotate option will cause the audit daemon to rotate the logs.
- space_left (default:75 MB) & space_left_action (default:SYSLOG)
- admin_space_left (default:75 MB) & admin_space_left_action (default:SUSPEND)
- disk_full_action (default: SUSPEND) & disk_error_action (default:SUSPEND)
The first 2 can be explained as a warning threshold + action and critical threshold + action.
The 3rd one defines what to do if the disk becomes full or disk errors occur when writing
The valid actions are
ignore - do nothing, syslog - log to syslog, email - send email to action_mail_acct,
exec - execute a script, suspend - stop audit logging, single - single user mode, and halt - halt system
- action_mail_acct
Valid email address for email actions from above. This require /usr/lib/sendmail
- flush & freqThe default is to use incremental and a count of 20 which means it will flush to disk after 20 events. Valid values are none, incremental, data, and sync. This is a trade off between disk I/O and how much log data you can afford to lose if the server loses power or abruptly reboots. The data forces data to be synced at all times & the sync forced both data and meta data to be synced at all times. The sync option will off course cause the most amount of I/O. If you can afford to lose some log data, then leaving it at default would be fine.
- name_format & name
This inserts the node name/host name to the log line. Default is to insert nothing. It can be set to hostname, IP address or FQDN. This makes sense if you are passing logs from several machine to a centralized location.
- tcp_listen_portYou can tell auditd to listen for events from other machines.
Auditd rules
The main use of auditd is derived from the rules. /etc/audit/audit.rules is used to set configuration like watch rules, buffers etc.
auditctl can also be used to change rules on a live system.
General Settings
- Backlog (default: 64 1K buffers)
Turns out when Auditd is rotating logs it won't write any logs to file. So on a high volume server you might miss logs during rotation. To avoid this raise the kernel backlog buffers by adding -b <num> to /etc/audit/audit.rules. The default is 64 1K buffers. If the limit is reached the failure flag is consulted
- Rate
Messages/sec can be set by -r. if the rate is exceeded the failure flag is consulted.
- Failure flag (default: 1=printk)
Determines how the kernel will handle errors. 0 - silent, 1 - printk , 2 - panic. Even though panic sounds like a terrible idea, it can be important in high security environments, especially, if someone is trying to circumvent the logging.
- Locking rules
By adding -e 2 to end of the rules file, the configuration is locked and cannot be changed by the auditctl command. Attempts to do so will be logged and denied. Note, reverting this will require a reboot. This is another feature that is suited for very high security environments.
- Listing current rules
auditctl -l
Watching Files
-w path
Wildcards are not supported and will generate a warning. The way that watches work is by tracking the inode internally. Unlike other syscall rules, watches do not impact performance. For more granular options like audit a specific user accessing a file, use the syscall auditing
-p r|w|x|a
Set permissions filter for a file system watch. r=read, w=write, x=execute, a=attribute change. They are not standard file permissions but the syscall the gets executed for them
-k <key>
The -k adds a key making it easy to search if there are several watches. One use for -k is to define different alert levels; eg. low, medium, high and then tag different rules with those levels.
eg.
Watch yum.conf for write and attribute changes
Watch /sbin/service for execution
Watch /etc/shadow for read, write and attribute changes
Note: The same can be added to /etc/audit/audit.rules without the auditctl command
auditctl -w /etc/yum.conf -p wa -k yum_watch
auditctl -w /usr/bin/nmap -p x -k nmap_watch
auditctl -w /etc/shadow -p rwa -k shadow_watch
To report on watched files. Date format is local to the server's date format.
aureport -f
aureport -f --start 02/18/10 17:42:00
aureport -f --start 02/18/10 17:00:00 --end 02/18/10 17:10:00
aureport -f -ts this-week
aureport -f -ts today
Output will be similar to this
14742. 04/20/10 12:40:01 /etc/shadow 2 yes /usr/sbin/crond -1 977855
14743. 04/20/10 12:40:01 /etc/shadow 2 yes /usr/sbin/crond -1 977851
14744. 04/20/10 12:40:24 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977863
14745. 04/20/10 12:40:50 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977864
14746. 04/20/10 12:41:16 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977865
14747. 04/20/10 12:41:42 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977866
The 1st column is just an index
The 2nd column is the date
The 3nd column is the time
The 4rd column is the file name
The 5th column is the system call number. To convert this number into name use
aureport -f -i
The 6th column is the result of the system call. Success of failure. You can use
aureport --failed to list only failed
The 7th column is the process accessing it
The 8th column is the Actual/Audit UID (AUID). If you login as user joe(uid 500) and su to root (uid=0). The AUID is 500.
The 9th column is the event number. You can use ausearch -a <event no> to look up further details
Note :
-i is useful as it converts to more human readable value. However the conversion is done at the time of viewing. The log will contain numeric values only. So if uid 500 belongs to john at the time of logging and at a later date, uid 500 is assigned to user joe, there will be a discrepancy.
The above produces more of a summary. You can use 'ausearch' to get more details.
Let's take a closer look.
nmap is set to 0700 permissions.
[root@slice01 ~]# ls -l /usr/bin/nmap
-rwx------ 1 root root 3580248 Jul 6 2009 /usr/bin/nmap
Access by root yields the following log
# ausearch -i -k nmap_watch
time->Fri Apr 16 16:10:18 2010
node=slice01 type=PATH msg=audit(04/16/10 16:10:18.044:954405) : item=0 name=/usr/bin/nmap inode=721629 dev=08:01 mode=file,700
ouid=root ogid=root rdev=00:00
node=slice01 type=CWD msg=audit(04/16/10 16:10:18.044:954405) : cwd=/root
node=slice01 type=EXECVE msg=audit(04/16/10 16:10:18.044:954405) : a0=nmap
node=slice01 type=SYSCALL msg=audit(04/16/10 16:10:18.044:954405) : arch=x86_64
syscall=execve per=400000 success=yes exit=0 a0=6e2bd0 a1=6e5160 a2=7ae510 a3=0 items=1
ppid=7318 pid=14772 auid=sri uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0
comm=nmap exe=/usr/bin/nmap key=nmap_watch
Access by user with uid 500 who has been granted access results in this
----
time->Fri Apr 16 16:11:44 2010
node=slice01 type=PATH msg=audit(04/16/10 16:11:44.470:954410) : item=0 name=/usr/bin/nmap inode=721629 dev=08:01 mode=file,700
ouid=root ogid=root rdev=00:00
node=slice01 type=CWD msg=audit(04/16/10 16:11:44.470:954410) : cwd=/root
node=slice01 type=SYSCALL msg=audit(04/16/10 16:11:44.470:954410) : arch=x86_64
syscall=execve success=no exit=-13(Permission denied) a0=6f26e0 a1=6f2260 a2=6cf8e0 a3=0 items=1
ppid=13286 pid=15089 auid=sri uid=sri gid=sri euid=sri suid=sri fsuid=sri egid=sri sgid=sri fsgid=sri tty=pts1
comm=bash exe=/bin/bash key=nmap_watch
success=<yes|no> shows if the system call was successful
Again the
-i lists output in more human readable format and converts syscall numbers and user ids into names.
To remove a rule using auditctl you can use the
-W switch
auditctl -W /etc/shadow -p rwa -k shadow_watch
Another way to watch files is via system calls. See below.
Watching system calls
You can use this to watch any system call. There are various options available here that can be combined to audit different types of events.
Required switches with syscall monitoring are the
-a list,action -A list,action
-a appends to the end of the list
-A appends to the start of the list
The corresponding switch to the
-a is the
-d switch which can delete rules of a certain type
The
list values can be
entry/exit - This determines when to log with respect to the system call invocation
exclude - This can be used to filter events
The
action values can be :-
never - No audit records will be generated. The order of rules in the audit.rules file matters. Normally you would put suppressions at the top
always - Always write out a record
The
-S switch defines the system call to watch. It can be name or the system call number
The next switch is the
-F which builds a rule field. Several of these can be grouped together to filter various stuff.
Common things to use with this are :-
arch - cpu architecture
auid/uid - The original ID the user logged in with & the user id
euid/egid - Effective user and group ID
path/dir - Full path of the file or directory to watch. Directory watches are recursive
filetype - Type of the file. file, dir, socket, symlink, char, block, or fifo
perm - permission filter for file operation
pid/ppid - process id and parent process id
Watching for ptrace system call. Utilities like strace use it
auditctl -a entry,always -F arch=b64 -S ptrace -k info_scan
Suppressing 32bit clock_gettime & fstat64 system calls
-a entry,never -F arch=b32 -S clock_gettime -k clock_gettime
-a entry,never -F arch=b32 -S fstat64 -k fstat64
Audit files opened by a specific user. The first rule will audit all open files. It is a good idea to watch for both auid and uid.
Also the architecture is required with system calls like these
auditctl -a exit,always -S open -F auid=2010
auditctl -a exit,always -F arch=b64 -F auid=2010 -F uid=2010 -F path=/etc/hosts -S open
To search you can use the key or use the
-sc switch
ausearch -k info_scan -i
ausearch -sc ptrace -i
To search by user id
ausearch -ua 2010
Authentication Report
Lists all auth attempts and their result. This also includes logins by other means, eg. imap access
aureport -au
To list just logins
aureport -l
To list account modification attempts. This lists only successful attempts. For eg. it won't list an attempt made by an unprivileged user.
aureport -m
Rules for security standards
/usr/share/doc/audit-<version> has pre-defined rules for different standards
Controlled Access Protection Profile (CAPP) -
http://www.commoncriteriaportal.org/files/ppfiles/capp.pdf
Labeled Security Protection Profile (LSPP) -
http://www.commoncriteriaportal.org/files/ppfiles/lspp.pdf
National Industrial Security Program Operating Manual (NISPOM) -
http://www.fas.org/sgp/library/nispom.htm
Security Technical Implementation Guides -
http://iase.disa.mil/stigs/stig/index.html
Best Practices
- If you want everything audited you should add a boot param of audit=1 to Grub. Otherwise certain processes that start before audit loads will not be monitored.
- /var/log/audit should be on its own partition.
- Use Syscall rules with care. They will lead to performance degradation if overused
Commands Listing
Adding/Modifying Rules
auditctl -w /etc/yum.conf -p wa -k yum_watch
auditctl -w /usr/bin/nmap -p x -k nmap_watch
auditctl -w /etc/shadow -p rwa -k shadow_watch
- Remove a rule using auditctl
auditctl -W /etc/shadow -p rwa -k shadow_watch
- Watching for ptrace system call
auditctl -a entry,always -F arch=b64 -S ptrace -k info_scan
- Suppress 32bit clock_gettime & fstat64 system calls
auditctl -a entry,never -F arch=b32 -S clock_gettime -k clock_gettime -a entry,never -F arch=b32 -S fstat64 -k fstat64
- Audit files opened by a specific user
auditctl -a exit,always -S open -F auid=2010 auditctl -a exit,always -F arch=b64 -F auid=2010 -F uid=2010 -F path=/etc/hosts -S open
- Audit unsuccessful attempts for multiple system calls where user id is greater than or equal to 500
auditctl -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -F auid>=500
auditctl -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -F auid>=500
Reporting/Searching
auditctl -l
auditctl -s
- Report on watched files. Date format is local to the server's date format.
aureport -f aureport -f --start 02/18/10 17:42:00
aureport -f --start 02/18/10 17:00:00 --end 02/18/10 17:10:00 aureport -f -ts this-week aureport -f -ts today
ausearch -sc ptrace -i
- Search for user id or effective user id
ausearch -ui 2010 ausearch -ue 2010
- Lists all auth attempts and their result
aureport -au
aureport -l
- List account modification attempts.
aureport -m
- Search events where success value is no, User id is 500 and key is nmap_watch
ausearch -sv no -ua 500 -k nmap_watch
ausearch -x /usr/bin/nmap
ausearch -tm pts/0
- Search by daemon. Stuff like cron log terminal as the daemon name
ausearch -tm cron
Audit data visualisation
mkgraph & mkgraph are two scripts that make use of gnuplot to plot graphs using the data from aureport and ausearch.
More of this at
http://people.redhat.com/sgrubb/audit/visualize/index.html
