Load Average

Linux file descriptors and open modes

2012-01-18T09:39:00.000-08:00

Ever want to find out what modes a file was opened with originally ?

First find the file descriptor number


ls /proc//fd
Eg
ls -l /proc/32048/fd/30
l-wx------ 1 apache apache 64 Jan 18 07:15 30 -> /var/log/httpd/ntop-access_log

Then check fdinfo

cat /proc/32048/fdinfo/30
pos: 0
flags: 0102001

The flags are derived from the open system call http://linux.about.com/od/commands/l/blcmdl2_open.htm

To actually decipher the octal codes , look under /usr/include/bits/fcntl.h

If there are multiple codes, the codes are appended together.

Linux ACLs

2012-01-12T13:38:00.000-08:00

Filesystem options and commands

First check to make sure the file system is mounted with acl settings

cat /proc/mounts |grep acl

/dev/sda1 / ext3 rw,noatime,relatime,errors=remount-ro,acl,data=ordered 0 0

If not update /etc/fstab and add 'acl' to the options section and remount the file system

getfacl, setfacl, chacl are the two main commands. chacl is available for IRIX compatibility.

Use Cases

Grant 2 users permissions to the same directory and files under it

Let's say we want to grant user john and mary permissions to folder /var/www/mysite.com

We can start by creating the directory. At this point we can leave it owned by root as the ACLs will help here.

ls -ld /var/www/mysite.com/
drwxr-xr-x 2 root root 4096 Feb 11 15:21 /var/www/mysite.com/

The first 2 commands grant users john and mary permissions.

The second sets the default acl. This causes the acls to be applied with inhertiance set. So this makes good sense in a multi user multi edit environment. The next arguments between the : are the username and the permissions

setfacl -m john:rwx mysite.com 
setfacl -m mary:rwx mysite.com 

setfacl -m default:john:rwx mysite.com 
setfacl -m default:mary:rwx mysite.com

# file: mysite.com
# owner: root
# group: root
user::rwx
user:john:rwx
group::r-x
mask::rwx
other::r-x
default:user::rwx
default:user:john:rwx
default:user:mary:rwx
default:group::r-x
default:mask::rwx
default:other::r-x

Now create a file by logging in a user john.

john@slice01$ echo "john" > file1

john@slice01$ ls -l file1 
-rw-rw-r--+ 1 john john 4 Feb 11 15:31 file1

john@slice01$ getfacl file1 
# file: file1
# owner: john
# group: john
user::rw-
user:john:rwx   #effective:rw-
user:mary:rwx   #effective:rw-
group::r-x   #effective:r--
mask::rw-
other::r--

Then create a directory

john@slice01$ mkdir john

john@slice01$ getfacl john
# file: john
# owner: john
# group: john
user::rwx
user:john:rwx
user:mary:rwx
group::r-x
mask::rwx
other::r-x
default:user::rwx
default:user:john:rwx
default:user:mary:rwx
default:group::r-x
default:mask::rwx
default:other::r-x

As you can see mary is there in the ACLs also

You can test it by logging in as user mary & editing files created by john.

mary@slice01$ echo mary >> file1 

mary@slice01$ cat file1 
john
mary

mary@slice01$ cd john/

mary@slice01 $ echo mary > file2
mary@slice01 $ getfacl file2 
# file: file2
# owner: mary
# group: mary
user::rw-
user:john:rwx   #effective:rw-
user:mary:rwx   #effective:rw-
group::r-x   #effective:r--
mask::rw-
other::r--

Grant 2 users permissions to the same directory and files under it except to 2 individual directories

Lets say we want john and mary to have permissions under /var/www/mysite.com/ and all files but still have individual directories
/var/www/mysite.com/john & /var/www/mysite.com/mary

 getfacl mary
# file: mary
# owner: mary
# group: mary
user::rwx
user:john:rwx
user:mary:rwx
group::r-x
mask::rwx
other::r-x
default:user::rwx
default:user:john:rwx
default:user:mary:rwx
default:group::r-x
default:mask::rwx
default:other::r-x

The -k switch removes the default acls

setfacl -k mary

getfacl mary
# file: mary
# owner: mary
# group: mary
user::rwx
user:john:rwx
user:mary:rwx
group::r-x
mask::rwx
other::r-x

Then remove john from it also


setfacl -x john mary

getfacl mary
# file: mary
# owner: mary
# group: mary
user::rwx
user:mary:rwx
group::r-x
mask::rwx
other::r-x

Repeat the same with other folder

Redhat 6 - Part 1

2011-02-07T02:39:00.000-08:00

Here are some new stuff in RHEL 6

Software versions

PHP 5.3.1. It also ships with APC (Alternative PHP Cache).
Apache is 2.2.14
MySQL is 5.1.42
Tomcat is 6.0.20
PostgreSQL is version 8.4
Python is 2.6
Perl is 5.10.1
Gcc is 44.4

File systems

Ext4 support- Read more at
https://ext4.wiki.kernel.org/index.php/Ext4_Howto
XFS support
Technology preview of btrfs filesystem

Other Notable Changes

Default use of NFS v4
SysV init is gone in favour of upstart. Upstart comes with legacy support for traditional init scripts in /etc/init.d.
Support for Fibre Channel over Ethernet (FCoE)
iSCSI can now be used as root or boot devices
As expected, Xen has been dropped in favour or KVM

IPv6 Part 1

2011-01-17T14:12:00.000-08:00

Some trivia

Why 128 bit ? - It was a compromise between a fixed 64 bit and a variable 160 bits (google OSI NSAP for more)

How many IP addresses ? - 340,282,366,920,938,463,463,374,607,431,768,211,456

What about IPv5 ? - Used by ST - a stream protocol and not related to IP

Key Differences

More IPs
Different Header format
Extension support
Flow labeling
Authentication support

Header

Very simplified (See http://en.wikipedia.org/wiki/File:Ipv6_header.svg)
20 bytes + Options in IPv4 Vs 12 fields (40 bytes) in IPv6
Base header no longer contains fragmentation options
Header no longer contains any checksum
Time to live (TTL) is now called Hop Limit
Support for traffic classes
Extensibility in headers. Options are not limited to the 40 bytes

TCP timers and keepalives

2011-01-09T02:33:00.000-08:00

Netstat -o includes tcp timers which are useful for Apache keepalive analysis.

netstat -ntpo | grep ESTAB | egrep ":80|:443"

Output looks like this

<br />tcp  0  0  98.xxx.xxx.106:80  83.138.139.7:41089   ESTAB  25812/httpd  keepalive  (7196.71/0/0)<br />tcp  0  0  98.xxx.xxx.106:80  83.138.139.7:41088   ESTAB  19385/httpd  keepalive  (7196.71/0/0)<br />tcp  0  0  98.xxx.xxx.106:80  83.138.139.45:41090  ESTAB  26995/httpd  keepalive  (7196.71/0/0)<br />tcp  0  0  98.xxx.xxx.106:80  83.138.139.37:41087  ESTAB  25813/httpd  keepalive  (7196.70/0/0)<br />tcp  0  0  98.xxx.xxx.106:80  83.138.139.7:41088   ESTAB  19385/httpd  keepalive  (7196.71/0/0)<br />tcp  0  0  98.xxx.xxx.106:80  83.138.139.27:41090  ESTAB  26995/httpd  on         (7196.71/0/0)<br />tcp  0  0  98.xxx.xxx.106:80  83.138.139.17:41088  ESTABLISHED  19385/httpd  off        (7196.71/0/0)<br />tcp  0  0  98.xxx.xxx.106:80  83.138.139.17:41090  ESTABLISHED  26995/httpd  off        (7196.71/0/0)<br />

The last column denotes what the connection is doing.

'on' - Actively transfering data.

'off' - Currently disconnecting

'keepalive' - Connections are using TCP keepalives. The first number denotes the time in seconds from when the last data was transferred until when the next TCP keepalive probe will be sent. By default this starts at 7200s, and resets again every time more data is sent. If the value is low , for eg. 4000 seconds , it means some of the keep alive connections are hanging or doing nothing for a long period. Note, connections to internal proxy or other internal processes might hang longer but this should not happen to web based connection.

The defaults are dictated by the sysctl values :-

''net.ipv4.tcp_keepalive_probes'' - How many keepalive probes TCP sends out, until it decides that the connection is broken. Default value: 9.

''net.ipv4.tcp_keepalive_time'' - How often TCP sends out keepalive messages when keepalive is enabled. Default: 2hours (7200 seconds)

''net.ipv4.tcp_keepalive_intvl'' - How frequently the probes are send out. Multiplied by tcp_keepalive_probes it is time to kill not responding connection, after probes started. Default value: 75sec i.e. connection will be aborted after ~11 minutes of retries.

More information on the sysctly values can be found in the kernel documentation ''/usr/share/doc/kernel-doc-/Documentation/networking/ip-sysctl.txt''

Linux and Auditd

2010-12-30T02:40:00.000-08:00

Assumptions: Everything here is tested on Red Hat/Cent OS 5, and 2.6.24.XX kernel

auditd is the userspace side of kernel auditing functions. It can be used to watch file accesses, monitory system calls, log events etc.The closest to a home page it has http://people.redhat.com/sgrubb/audit/. It has some nice presentations done at different Red Hat conferences and some other articles.

Installation
Installing auditd is straightforward but to use it efficiently you need to tune it to your needs. To install use

yum install audit

To start the service do
/etc/init.d/auditd start

Configuration

Installation is a breeze but configuring to your needs requires some tweaking.

auditd.conf can be used to tweak auditd's behaviour. Common settings to review/tune are :-

log_file (default:/var/log/audit/audit.log) & log_format(default:raw)
Set the log file name and the format. The format can be either raw or nolog.

num_logs(default:0)
Auditd does its own rotation and the default is no rotation. This keyword specifies the number of log files to keep if rotate is given as the max_log_file_action. Note, there is a small catch to this setting. Turns out when Auditd is rotating logs it won't process any logs. So on a high volume server you might miss logs during rotation. To avoid this raise the kernel backlog buffers by adding -b <num> to /etc/audit/audit.rules. The default is 64 1K buffers. This cannot be greater than 99.

max_log_file (default:5MB) & max_log_file_action (default:ROTATE)
The first sets the maximum log size in megabytes. Action defines what to do when it reaches that state. Valid values are ignore, syslog, suspend, rotate and keep_logs. If set to ignore, the audit daemon does nothing. syslog means that it will issue a warning to syslog. suspend will cause the audit daemon to stop writing records to the disk. The daemon will still be alive. The rotate option will cause the audit daemon to rotate the logs.

space_left (default:75 MB) & space_left_action (default:SYSLOG)
admin_space_left (default:75 MB) & admin_space_left_action (default:SUSPEND)
disk_full_action (default: SUSPEND) & disk_error_action (default:SUSPEND)
The first 2 can be explained as a warning threshold + action and critical threshold + action.
The 3rd one defines what to do if the disk becomes full or disk errors occur when writing
The valid actions are
ignore - do nothing, syslog - log to syslog, email - send email to action_mail_acct,
exec - execute a script, suspend - stop audit logging, single - single user mode, and halt - halt system

action_mail_acct
Valid email address for email actions from above. This require /usr/lib/sendmail

flush & freqThe default is to use incremental and a count of 20 which means it will flush to disk after 20 events. Valid values are none, incremental, data, and sync. This is a trade off between disk I/O and how much log data you can afford to lose if the server loses power or abruptly reboots. The data forces data to be synced at all times & the sync forced both data and meta data to be synced at all times. The sync option will off course cause the most amount of I/O. If you can afford to lose some log data, then leaving it at default would be fine.

name_format & name
This inserts the node name/host name to the log line. Default is to insert nothing. It can be set to hostname, IP address or FQDN. This makes sense if you are passing logs from several machine to a centralized location.

tcp_listen_portYou can tell auditd to listen for events from other machines.

Auditd rules
The main use of auditd is derived from the rules. /etc/audit/audit.rules is used to set configuration like watch rules, buffers etc. auditctl can also be used to change rules on a live system.

General Settings

Backlog (default: 64 1K buffers)
Turns out when Auditd is rotating logs it won't write any logs to file. So on a high volume server you might miss logs during rotation. To avoid this raise the kernel backlog buffers by adding -b <num> to /etc/audit/audit.rules. The default is 64 1K buffers. If the limit is reached the failure flag is consulted

Rate
Messages/sec can be set by -r. if the rate is exceeded the failure flag is consulted.

Failure flag (default: 1=printk)
Determines how the kernel will handle errors. 0 - silent, 1 - printk , 2 - panic. Even though panic sounds like a terrible idea, it can be important in high security environments, especially, if someone is trying to circumvent the logging.

Locking rules
By adding -e 2 to end of the rules file, the configuration is locked and cannot be changed by the auditctl command. Attempts to do so will be logged and denied. Note, reverting this will require a reboot. This is another feature that is suited for very high security environments.

Listing current rules
auditctl -l

Watching Files

-w path
Wildcards are not supported and will generate a warning. The way that watches work is by tracking the inode internally. Unlike other syscall rules, watches do not impact performance. For more granular options like audit a specific user accessing a file, use the syscall auditing

-p r|w|x|a
Set permissions filter for a file system watch. r=read, w=write, x=execute, a=attribute change. They are not standard file permissions but the syscall the gets executed for them

-k <key>
The -k adds a key making it easy to search if there are several watches. One use for -k is to define different alert levels; eg. low, medium, high and then tag different rules with those levels.

eg.
Watch yum.conf for write and attribute changes
Watch /sbin/service for execution
Watch /etc/shadow for read, write and attribute changes

Note: The same can be added to /etc/audit/audit.rules without the auditctl command

auditctl -w /etc/yum.conf -p wa  -k yum_watch
auditctl -w /usr/bin/nmap -p x   -k nmap_watch
auditctl -w /etc/shadow   -p rwa -k shadow_watch

To report on watched files. Date format is local to the server's date format.
aureport -f

aureport -f --start 02/18/10 17:42:00
aureport -f --start 02/18/10 17:00:00 --end 02/18/10 17:10:00
aureport -f -ts this-week
aureport -f -ts today

Output will be similar to this
14742. 04/20/10 12:40:01 /etc/shadow 2 yes /usr/sbin/crond -1 977855

14743. 04/20/10 12:40:01 /etc/shadow 2 yes /usr/sbin/crond -1 977851
14744. 04/20/10 12:40:24 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977863
14745. 04/20/10 12:40:50 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977864
14746. 04/20/10 12:41:16 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977865
14747. 04/20/10 12:41:42 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977866

The 1st column is just an index

The 2nd column is the date

The 3nd column is the time

The 4rd column is the file name

The 5th column is the system call number. To convert this number into name use aureport -f -i

The 6th column is the result of the system call. Success of failure. You can use aureport --failed to list only failed

The 7th column is the process accessing it

The 8th column is the Actual/Audit UID (AUID). If you login as user joe(uid 500) and su to root (uid=0). The AUID is 500.

The 9th column is the event number. You can use ausearch -a <event no> to look up further details

Note : -i is useful as it converts to more human readable value. However the conversion is done at the time of viewing. The log will contain numeric values only. So if uid 500 belongs to john at the time of logging and at a later date, uid 500 is assigned to user joe, there will be a discrepancy.

The above produces more of a summary. You can use 'ausearch' to get more details.

Let's take a closer look.

nmap is set to 0700 permissions.

[root@slice01 ~]# ls -l /usr/bin/nmap
-rwx------ 1 root root 3580248 Jul  6  2009 /usr/bin/nmap

Access by root yields the following log

# ausearch -i -k nmap_watch 

time->Fri Apr 16 16:10:18 2010
node=slice01 type=PATH msg=audit(04/16/10 16:10:18.044:954405) : item=0 name=/usr/bin/nmap inode=721629 dev=08:01 mode=file,700 
ouid=root ogid=root rdev=00:00 
node=slice01 type=CWD msg=audit(04/16/10 16:10:18.044:954405) :  cwd=/root 
node=slice01 type=EXECVE msg=audit(04/16/10 16:10:18.044:954405) : a0=nmap 
node=slice01 type=SYSCALL msg=audit(04/16/10 16:10:18.044:954405) : arch=x86_64 
syscall=execve per=400000 success=yes exit=0 a0=6e2bd0 a1=6e5160 a2=7ae510 a3=0 items=1 
ppid=7318 pid=14772 auid=sri uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 
comm=nmap exe=/usr/bin/nmap key=nmap_watch

Access by user with uid 500 who has been granted access results in this

----
time->Fri Apr 16 16:11:44 2010
node=slice01 type=PATH msg=audit(04/16/10 16:11:44.470:954410) : item=0 name=/usr/bin/nmap inode=721629 dev=08:01 mode=file,700 
ouid=root ogid=root rdev=00:00 
node=slice01 type=CWD msg=audit(04/16/10 16:11:44.470:954410) :  cwd=/root 
node=slice01 type=SYSCALL msg=audit(04/16/10 16:11:44.470:954410) : arch=x86_64 
syscall=execve success=no exit=-13(Permission denied) a0=6f26e0 a1=6f2260 a2=6cf8e0 a3=0 items=1 
ppid=13286 pid=15089 auid=sri uid=sri gid=sri euid=sri suid=sri fsuid=sri egid=sri sgid=sri fsgid=sri tty=pts1 
comm=bash exe=/bin/bash key=nmap_watch

success=<yes|no> shows if the system call was successful

Again the -i lists output in more human readable format and converts syscall numbers and user ids into names.

To remove a rule using auditctl you can use the -W switch

auditctl -W /etc/shadow -p rwa -k shadow_watch

Another way to watch files is via system calls. See below.

Watching system calls
You can use this to watch any system call. There are various options available here that can be combined to audit different types of events.

Required switches with syscall monitoring are the -a list,action -A list,action

-a appends to the end of the list

-A appends to the start of the list

The corresponding switch to the -a is the -d switch which can delete rules of a certain type

The list values can be

entry/exit - This determines when to log with respect to the system call invocation

exclude - This can be used to filter events

The action values can be :-

never - No audit records will be generated. The order of rules in the audit.rules file matters. Normally you would put suppressions at the top

always - Always write out a record

The -S switch defines the system call to watch. It can be name or the system call number

The next switch is the -F which builds a rule field. Several of these can be grouped together to filter various stuff.

Common things to use with this are :-

arch - cpu architecture

auid/uid - The original ID the user logged in with & the user id

euid/egid - Effective user and group ID

path/dir - Full path of the file or directory to watch. Directory watches are recursive

filetype - Type of the file. file, dir, socket, symlink, char, block, or fifo

perm - permission filter for file operation

pid/ppid - process id and parent process id

Watching for ptrace system call. Utilities like strace use it

auditctl -a entry,always -F arch=b64 -S ptrace -k info_scan

Suppressing 32bit clock_gettime & fstat64 system calls

-a entry,never -F arch=b32 -S clock_gettime -k clock_gettime
-a entry,never -F arch=b32 -S fstat64 -k fstat64

Audit files opened by a specific user. The first rule will audit all open files. It is a good idea to watch for both auid and uid.
Also the architecture is required with system calls like these

auditctl -a exit,always -S open -F auid=2010
auditctl -a exit,always -F arch=b64 -F auid=2010  -F uid=2010 -F path=/etc/hosts -S open

To search you can use the key or use the -sc switch

ausearch -k info_scan -i
ausearch -sc ptrace -i

To search by user id

ausearch -ua 2010

Authentication Report

Lists all auth attempts and their result. This also includes logins by other means, eg. imap access

aureport -au

To list just logins

aureport -l

To list account modification attempts. This lists only successful attempts. For eg. it won't list an attempt made by an unprivileged user.

aureport -m

Rules for security standards

/usr/share/doc/audit-<version> has pre-defined rules for different standards

Controlled Access Protection Profile (CAPP) - http://www.commoncriteriaportal.org/files/ppfiles/capp.pdf

Labeled Security Protection Profile (LSPP) - http://www.commoncriteriaportal.org/files/ppfiles/lspp.pdf

National Industrial Security Program Operating Manual (NISPOM) - http://www.fas.org/sgp/library/nispom.htm

Security Technical Implementation Guides - http://iase.disa.mil/stigs/stig/index.html

Best Practices

If you want everything audited you should add a boot param of audit=1 to Grub. Otherwise certain processes that start before audit loads will not be monitored.

/var/log/audit should be on its own partition.

Use Syscall rules with care. They will lead to performance degradation if overused

Commands Listing
Adding/Modifying Rules

Watch for files

auditctl -w /etc/yum.conf -p wa -k yum_watch
auditctl -w /usr/bin/nmap -p x -k nmap_watch

auditctl -w /etc/shadow -p rwa -k shadow_watch

Remove a rule using auditctl

auditctl -W /etc/shadow -p rwa -k shadow_watch

Watching for ptrace system call

auditctl -a entry,always -F arch=b64 -S ptrace -k info_scan

Suppress 32bit clock_gettime & fstat64 system calls

auditctl -a entry,never -F arch=b32 -S clock_gettime -k clock_gettime -a entry,never -F arch=b32 -S fstat64 -k fstat64

Audit files opened by a specific user

auditctl -a exit,always -S open -F auid=2010 auditctl -a exit,always -F arch=b64 -F auid=2010 -F uid=2010 -F path=/etc/hosts -S open

Audit unsuccessful attempts for multiple system calls where user id is greater than or equal to 500

auditctl -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -F auid>=500
auditctl -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -F auid>=500

Reporting/Searching

List all rules

auditctl -l

List status

auditctl -s

Report on watched files. Date format is local to the server's date format.

aureport -f aureport -f --start 02/18/10 17:42:00
aureport -f --start 02/18/10 17:00:00 --end 02/18/10 17:10:00

aureport -f -ts this-week aureport -f -ts today

Search by system call

ausearch -sc ptrace -i

Search for user id or effective user id

ausearch -ui 2010 ausearch -ue 2010

Lists all auth attempts and their result

aureport -au

List just logins

aureport -l

List account modification attempts.

aureport -m

Search events where success value is no, User id is 500 and key is nmap_watch

ausearch -sv no -ua 500 -k nmap_watch

Search by executable

ausearch -x /usr/bin/nmap

Search by terminal

ausearch -tm pts/0

Search by daemon. Stuff like cron log terminal as the daemon name

ausearch -tm cron

Audit data visualisation

mkgraph & mkgraph are two scripts that make use of gnuplot to plot graphs using the data from aureport and ausearch.

More of this at http://people.redhat.com/sgrubb/audit/visualize/index.html