Sunday, 21 December 2014

Docker, Rackspace On Metal and Core OS


This post will cover Rackspace On Metal, an intro to Docker, review of CoreOS & Fleet and demonstrate how one could use all of them to build a multi-tier application.

The associated presentation on this can be found at http://bit.ly/rs-onmetal-docker & the code at https://github.com/srirajan/onmetal-docker

Before you start

You will need the following
  •  A Rackspace cloud account. Get a free tier Rackspace developer account - https://developer.rackspace.com/signup/
  •  If you don't have an account, you can still follow this and do it on your own servers.  Some of the examples are specific to Rackspace cloud servers but the ones around Docker, CoreOS and Fleet can be done on any server.
  •  Ensure you have novaclient installed. Refer to http://www.rackspace.com/knowledge_center/article/installing-python-novaclient-on-linux-and-mac-os for more details.

Rackspace On Metal


Onmetal provides single-tenant bare metal servers and you read more about it here
http://www.rackspace.com/cloud/servers/onmetal

As of Dec, 2014, On Metal is only available in US region of IAD. So you need a Rackspace US cloud account.

We will have two groups of servers.
  • First we will build a single Ubuntu 14.04 LTS (Trusty Tahr) On Metal server
key=sri-mb

nova boot --flavor onmetal-compute1 \

--image 6cbfd76c-644c-4e28-b3bf-5a6c2a879a4a \

--key-name $key \

--poll play01
  • Now prepare CoreOS cluster. First get a discovery URL for your cluster. More on this at https://coreos.com/docs/cluster-management/setup/cluster-discovery/
curl -w "\n" https://discovery.etcd.io/new
  • Edit the cloud init file named coreos-cluster/cloudinit.yaml. Replace the token url from above.
  • If you are doing on On Metal use coreos-cluster/cloudinit-onmetal.yaml as a workaround for https://github.com/coreos/coreos-cloudinit/issues/195. Eventually the above should work on both flavors.
  • Decide which flavor you are using
#On metal
flavor=onmetal-compute1
image=75a86b9d-e016-4cb7-8532-9e9b9b5fc58b
key=sri-mb
cloudinit=cloudinit-onmetal.yaml


# Performance
flavor=performance1-1
image=749dc22a-9563-4628-b0d1-f84ced8c7b7a
key=sri-mb
cloudinit=cloudinit.yaml


  • Boot 4 servers for the cluster
nova boot --flavor $flavor --image $image  --key-name $key \
--config-drive true --user-data $cloudinit core01

nova boot --flavor $flavor --image $image  --key-name $key \
--config-drive true --user-data $cloudinit core02

nova boot --flavor $flavor --image $image  --key-name $key \
--config-drive true --user-data $cloudinit core03

nova boot --flavor $flavor --image $image  --key-name $key \
--config-drive true --user-data $cloudinit core04


Docker

Now let's play a litte with Docker.

  • Install docker on Ubuntu (play01 above)
apt-get update
apt-get install -y docker.io screen git vim
update-rc.d docker.io  defaults

  • Pull our first image. Review the listing & you should have 3 centos images for each of the versions.
docker pull centos

docker images
  •  Run it. '-i' is for interactive & '-t' allocates a pseudo-tty.
docker run -i -t centos /bin/bash


  • This runs the default image for CentOS. Review the inside to see how this looks inside docker.
cat /etc/redhat-release

ps

ls

whoami

cat /etc/hosts

exit


  •  List Docker processes
docker ps -a


  •  Run a different release
docker run -i -t centos:centos6 /bin/bash

cat /etc/redhat-release

exit


  •  On the host (play01) with the networking. Docker uses a combination of Linux bridges and iptables to build managed networking on the container, communication with other containers and communication from the outside.
ifconfig docker0

iptables -nvL

iptables -nvL -t nat


  •  Let's run something more than bash.

docker run -d  centos python -m SimpleHTTPServer 8888


  •  And check a few things.
docker ps

docker top <container UID>

docker inspect <container UID> |less

curl  http://<container IP>:8888


  •  Let's do some more. Clone the repo from git
apt-get install -y git

git clone https://github.com/srirajan/onmetal-docker


  •  Review the Dockerfile. This image file installs Nginx and PHP & loads a sample php file
cd /root/onmetal-docker/ubuntu_phpapp

cat Dockerfile

docker build -t="srirajan/ubuntu_phpapp" .

docker images


  •  Run the container and map the port 80 on the container to port 8082 on the host. You can curl the URL to see the site.
docker run -d -p 8082:80 "srirajan/ubuntu_phpapp"

docker top <container UID>

docker logs <container UID>

docker inspect <container UID>



#curl the container IP

curl http://<container IP>/home.php



#curl the public IP and port

curl http://<IP>:8082/home.php


  •  Docker diff
docker diff <container UID>


  •  Now let's look at linking containers. Start a mysql container and map the mysql port on the container to the host port 3306
docker run --name db -e MYSQL_ROOT_PASSWORD=dh47dk504dk44dd -d -p 3306:3306 mysql

docker logs db


  •  Build the helper container
cd /root/onmetal-docker/dbhelper

docker build -t="srirajan/dbhelper" .


  •  Start a helper container as a linked container and check it's environment variables. Linking allows information to be shared across containers. You can read more about linking here https://docs.docker.com/userguide/dockerlinks/
docker run --name dbhelper --link db:db srirajan/dbhelper env


  •  Now if you run the actual container as it is, it will login to the mysql instance on the db container and install the world database.
docker rm dbhelper

docker run -d --name dbhelper --link db:db srirajan/dbhelper  /usr/local/bin/configuredb.sh

docker logs dbhelper


CoreOS, Fleet & Docker

  • In the above steps, we should have created 4 core os machines with cloudinit. Now, lets play with CoreOS, etcd and Fleet. This should list all 4 machines in the cluster. Fleet is a distributed cluster management tool. It relies on etcd, which is a distributed key value store for operation. It also works with systemd files and behaves like a distributed systemd in a multi-node setup.
fleetctl list-machines


  •  Pull our repo on one of the nodes.
git clone https://github.com/srirajan/onmetal-docker


  •  Review and load all the services. We will go into details of each service in following steps.
cd /home/core/onmetal-docker/fleet-services

fleetctl submit *.service


fleetctl list-unit-files

UNIT       HASH DSTATE  STATE  TARGET

db.service      fbf415a launched launched -

dbhelper.service 747c778 inactive inactive -

lbhelper.service 0592528 inactive inactive -

mondb.service  a4f50cc inactive inactive -

monweb@.service  d5ed242 inactive inactive -

web@.service  0ac8be5 inactive inactive -



  •  Run the db service.
fleetctl start db.service

Unit db.service launched on 2aa4e35a.../10.208.201.253



fleetctl list-units

UNIT  MACHINE    ACTIVE SUB

db.service 2aa4e35a.../10.208.201.253 active running



  • You can also review the systemd service file. This one is fairly simple service and runs a mysql container on one of the hosts. Wait for this service to start before proceeding. Also note, that Fleet decides which host to run the container on.
cat db.service

[Unit]

Description=DB service

Requires=etcd.service



[Service]

EnvironmentFile=/etc/environment

TimeoutStartSec=0

ExecStartPre=/usr/bin/docker pull mysql

ExecStart=/usr/bin/docker run --rm --name db -e MYSQL_ROOT_PASSWORD=dh47dk504dk44dd -p ${COREOS_PRIVATE_IPV4}:3306:3306 mysql

ExecStop=/usr/bin/docker stop db

Restart=always


  •  One oddity with fleet is that to query the status, you have to run the command on the host running the container.
fleetctl status db.service

db.service - DB service

   Loaded: loaded (/run/fleet/units/db.service; linked-runtime)

   Active: active (running) since Thu 2014-11-13 14:11:06 UTC; 8min ago

  Process: 22050 ExecStartPre=/usr/bin/docker pull mysql (code=exited, status=0/SUCCESS)

 Main PID: 22068 (docker)

   CGroup: /system.slice/db.service

           └─22068 /usr/bin/docker run --rm --name db -e MYSQL_ROOT_PASSWORD=dh47dk504dk44dd -p 10.208.201.253:3306:3306 mysql



Nov 13 14:11:12 core04 docker[22068]: 2014-11-13 14:11:12 1 [Warning] No existing UUID has been found, so we assume that this is the first time that this server has been started. Generating a new UUID: ec53a4d8-6b3e-11e4-8386-0a1bbbebe238.

Nov 13 14:11:12 core04 docker[22068]: 2014-11-13 14:11:12 1 [Note] Server hostname (bind-address): '*'; port: 3306

Nov 13 14:11:12 core04 docker[22068]: 2014-11-13 14:11:12 1 [Note] IPv6 is available.

Nov 13 14:11:12 core04 docker[22068]: 2014-11-13 14:11:12 1 [Note]   - '::' resolves to '::';

Nov 13 14:11:12 core04 docker[22068]: 2014-11-13 14:11:12 1 [Note] Server socket created on IP: '::'.

Nov 13 14:11:12 core04 docker[22068]: 2014-11-13 14:11:12 1 [Note] Event Scheduler: Loaded 0 events

Nov 13 14:11:12 core04 docker[22068]: 2014-11-13 14:11:12 1 [Note] Execution of init_file '/tmp/mysql-first-time.sql' started.

Nov 13 14:11:12 core04 docker[22068]: 2014-11-13 14:11:12 1 [Note] Execution of init_file '/tmp/mysql-first-time.sql' ended.

Nov 13 14:11:12 core04 docker[22068]: 2014-11-13 14:11:12 1 [Note] mysqld: ready for connections.

Nov 13 14:11:12 core04 docker[22068]: Version: '5.6.21'  socket: '/tmp/mysql.sock'  port: 3306  MySQL Community Server (GPL)


  •  dbhelper.service uses container linking to install the mysql world database and configure some users for our application. The systemd configuration tells fleet to run on the same host as the db.service. mondb.service is not a container but uses systemd to run a script that updates etcd with the information about the db service. In this case we are just pushing private IPs to etcd but this can be leveraged to do other things as well.
fleetctl start dbhelper.service

Unit dbhelper.service launched on 2aa4e35a.../10.208.201.253



fleetctl start mondb.service

Unit mondb.service launched on 2aa4e35a.../10.208.201.253


  •  Run fleetctl again to see where our containers are deployed. Because of our systemd definition file, fleet will ensure they run on the same host.
fleetctl list-units

UNIT   MACHINE    ACTIVE SUB

db.service  2aa4e35a.../10.208.201.253 active running

dbhelper.service 2aa4e35a.../10.208.201.253 active running

mondb.service  2aa4e35a.../10.208.201.253 active running



  •  You can also login to the host running the dbhelper service and review the journal (logs) for the service.
fleetctl journal dbhelper

-- Logs begin at Mon 2014-11-10 21:00:41 UTC, end at Thu 2014-11-13 14:23:43 UTC. --

Nov 11 05:22:51 core04.novalocal systemd[1]: Stopped DB Helperservice.

Nov 11 05:22:51 core04.novalocal systemd[1]: Unit dbhelper.service entered failed state.

-- Reboot --

Nov 13 14:22:43 core04 systemd[1]: Starting DB Helperservice...

Nov 13 14:22:43 core04 docker[22239]: Pulling repository srirajan/dbhelper

Nov 13 14:23:00 core04 systemd[1]: Started DB Helperservice.

Nov 13 14:23:06 core04 docker[22272]: Creating the world database

Nov 13 14:23:13 core04 docker[22272]: Creating application user

Nov 13 14:23:13 core04 docker[22272]: Counting rows in world.city

Nov 13 14:23:13 core04 docker[22272]: COUNT(*)

Nov 13 14:23:13 core04 docker[22272]: 4079


  •  Now let's move on the web containers. Start the one container from the web service. In systemd a service with @ is generic service and you can append values to start as many of them. The first container will take a little bit of time as it is downloading the container.
fleetctl start web@01.service

Unit web@01.service launched on 6847f4f7.../10.208.201.226



fleetctl list-units

UNIT   MACHINE    ACTIVE  SUB

db.service  2aa4e35a.../10.208.201.253 active  running

dbhelper.service 2aa4e35a.../10.208.201.253 active  running

mondb.service  2aa4e35a.../10.208.201.253 active  running

web@01.service  6847f4f7.../10.208.201.226 active running


  •  Start 9 more web containers. Fleet will disribute them across the different hosts.
fleetctl start web@{02..10}.service

Unit web@04.service launched on ee5398cf.../10.208.201.250

Unit web@10.service launched on 2aa4e35a.../10.208.201.253

Unit web@07.service launched on 6847f4f7.../10.208.201.226

Unit web@09.service launched on ee5398cf.../10.208.201.250

Unit web@03.service launched on 6847f4f7.../10.208.201.226

Unit web@05.service launched on c3f52cb3.../10.208.201.234

Unit web@02.service launched on ee5398cf.../10.208.201.250

Unit web@06.service launched on c3f52cb3.../10.208.201.234

Unit web@08.service launched on c3f52cb3.../10.208.201.234



fleetctl list-units

UNIT   MACHINE    ACTIVE SUB

db.service  2aa4e35a.../10.208.201.253 active running

dbhelper.service 2aa4e35a.../10.208.201.253 active running

mondb.service  2aa4e35a.../10.208.201.253 active running

web@01.service  6847f4f7.../10.208.201.226 active running

web@02.service  ee5398cf.../10.208.201.250 active running

web@03.service  6847f4f7.../10.208.201.226 active running

web@04.service  ee5398cf.../10.208.201.250 active running

web@05.service  c3f52cb3.../10.208.201.234 active running

web@06.service  c3f52cb3.../10.208.201.234 active running

web@07.service  6847f4f7.../10.208.201.226 active running

web@08.service  c3f52cb3.../10.208.201.234 active running

web@09.service  ee5398cf.../10.208.201.250 active running

web@10.service  2aa4e35a.../10.208.201.253 active running



  •  Start the monweb services. These are similar to the mondb.service and update etcd with different values from the running containers.
fleetctl start monweb@{01..10}.service

Unit monweb@04.service launched on ee5398cf.../10.208.201.250

Unit monweb@02.service launched on ee5398cf.../10.208.201.250

Unit monweb@01.service launched on 6847f4f7.../10.208.201.226

Unit monweb@05.service launched on c3f52cb3.../10.208.201.234

Unit monweb@03.service launched on 6847f4f7.../10.208.201.226

Unit monweb@09.service launched on ee5398cf.../10.208.201.250

Unit monweb@07.service launched on 6847f4f7.../10.208.201.226

Unit monweb@10.service launched on 2aa4e35a.../10.208.201.253

Unit monweb@08.service launched on c3f52cb3.../10.208.201.234

Unit monweb@06.service launched on c3f52cb3.../10.208.201.234



fleetctl list-units

UNIT   MACHINE    ACTIVE SUB

db.service  2aa4e35a.../10.208.201.253 active running

dbhelper.service 2aa4e35a.../10.208.201.253 active running

mondb.service  2aa4e35a.../10.208.201.253 active running

monweb@01.service 6847f4f7.../10.208.201.226 active running

monweb@02.service ee5398cf.../10.208.201.250 active running

monweb@03.service 6847f4f7.../10.208.201.226 active running

monweb@04.service ee5398cf.../10.208.201.250 active running

monweb@05.service c3f52cb3.../10.208.201.234 active running

monweb@06.service c3f52cb3.../10.208.201.234 active running

monweb@07.service 6847f4f7.../10.208.201.226 active running

monweb@08.service c3f52cb3.../10.208.201.234 active running

monweb@09.service ee5398cf.../10.208.201.250 active running

monweb@10.service 2aa4e35a.../10.208.201.253 active running

web@01.service  6847f4f7.../10.208.201.226 active running

web@02.service  ee5398cf.../10.208.201.250 active running

web@03.service  6847f4f7.../10.208.201.226 active running

web@04.service  ee5398cf.../10.208.201.250 active running

web@05.service  c3f52cb3.../10.208.201.234 active running

web@06.service  c3f52cb3.../10.208.201.234 active running

web@07.service  6847f4f7.../10.208.201.226 active running

web@08.service  c3f52cb3.../10.208.201.234 active running

web@09.service  ee5398cf.../10.208.201.250 active running

web@10.service  2aa4e35a.../10.208.201.253 active running



  •  Query etcd for values. This will return the IP addresses and ports of the web containers.
for i in {01..10}; do  etcdctl get /services/web/web$i/unit; etcdctl get /services/web/web$i/host; etcdctl get /services/web/web$i/public_ipv4_addr; etcdctl get /services/web/web$i/port; echo "-----" ; done

monweb@01.service

core01

162.242.254.113

18001

-----

monweb@02.service

core03

162.242.255.71

18002

-----

monweb@03.service

core01

162.242.254.113

18003

-----

monweb@04.service

core03

162.242.255.71

18004

-----

monweb@05.service

core02

162.242.254.215

18005

-----

monweb@06.service

core02

162.242.254.215

18006

-----

monweb@07.service

core01

162.242.254.113

18007

-----

monweb@08.service

core02

162.242.254.215

18008

-----

monweb@09.service

core03

162.242.255.71

18009

-----

monweb@10.service

core04

162.242.255.73

18010

-----

  •  Test the site on one of the container.

curl http://162.242.255.73:18010/home.php

<!DOCTYPE html>

<html>

<body>



<strong>There is no place like 127.0.0.1</strong><br/>Date & Time: 2014-11-13 14:29:18<br/>Container name: dca363b9af75<hr/>



</body>

</html>



  •  At this point, we have database container running and a bunch of web containers running on different hosts. The communication between them has been established as well.
  • Optionally, we can run the lbhelper service that updates the cloud load balancer. This requires a Rackspace cloud load balancer pre-configured and you need to set the values in etcd
etcdctl set /services/rscloud/OS_USERNAME <cloud username>

etcdctl set /services/rscloud/OS_REGION <cloud region>

etcdctl set /services/rscloud/OS_PASSWORD <cloud api key>

etcdctl set /services/rscloud/OS_TENANT_NAME <cloud account no>

etcdctl set /services/rscloud/LB_NAME <cloud lb name>

etcdctl set /services/rscloud/SERVER_HEALTH_URL health.php

etcdctl set /services/rscloud/SERVER_HEALTH_DIGEST dbe72348d4e3aa87958f421e4a9592a82839f3d8

</ODE>

  • Now run the lbhelper service.
fleetctl start lbhelper.service

Unit lbhelper.service launched on 2aa4e35a.../10.208.201.253



fleetctl list-units

UNIT   MACHINE    ACTIVE SUB

db.service  2aa4e35a.../10.208.201.253 active running

dbhelper.service 2aa4e35a.../10.208.201.253 active running

lbhelper.service 2aa4e35a.../10.208.201.253 active running

mondb.service  2aa4e35a.../10.208.201.253 active running

monweb@01.service 6847f4f7.../10.208.201.226 active running

monweb@02.service ee5398cf.../10.208.201.250 active running

monweb@03.service 6847f4f7.../10.208.201.226 active running

monweb@04.service ee5398cf.../10.208.201.250 active running

monweb@05.service c3f52cb3.../10.208.201.234 active running

monweb@06.service c3f52cb3.../10.208.201.234 active running

monweb@07.service 6847f4f7.../10.208.201.226 active running

monweb@08.service c3f52cb3.../10.208.201.234 active running

monweb@09.service ee5398cf.../10.208.201.250 active running

monweb@10.service 2aa4e35a.../10.208.201.253 active running

web@01.service  6847f4f7.../10.208.201.226 active running

web@02.service  ee5398cf.../10.208.201.250 active running

web@03.service  6847f4f7.../10.208.201.226 active running

web@04.service  ee5398cf.../10.208.201.250 active running

web@05.service  c3f52cb3.../10.208.201.234 active running

web@06.service  c3f52cb3.../10.208.201.234 active running

web@07.service  6847f4f7.../10.208.201.226 active running

web@08.service  c3f52cb3.../10.208.201.234 active running

web@09.service  ee5398cf.../10.208.201.250 active running

web@10.service  2aa4e35a.../10.208.201.253 active running


  •  You can look at the logs from it. This container runs a python script that will populate the load balancer with the IP addresses and port numbers from the web containers. The script queries etcd for the information and also does a health check to determine the status of the container.
docker logs lbhelper

[11/13/14 14:35:48][INFO]:Authenticated using rtsdemo10

[11/13/14 14:36:32][INFO]:web 01 Found. Processing server

[11/13/14 14:36:32][INFO]:Health test passed. Adding to load balancer

[11/13/14 14:36:32][INFO]:Load balancer pool myworld found

[11/13/14 14:36:32][INFO]:Adding server to load balancer

[11/13/14 14:36:33][INFO]:web 02 Found. Processing server

[11/13/14 14:36:33][INFO]:Health test passed. Adding to load balancer

[11/13/14 14:36:33][INFO]:Load balancer pool myworld found

[11/13/14 14:36:33][INFO]:Adding server to load balancer

[11/13/14 14:36:44][INFO]:web 03 Found. Processing server

[11/13/14 14:36:44][INFO]:Health test passed. Adding to load balancer

[11/13/14 14:36:44][INFO]:Load balancer pool myworld found

[11/13/14 14:36:44][INFO]:Adding server to load balancer

[11/13/14 14:36:55][INFO]:web 04 Found. Processing server

[11/13/14 14:36:55][INFO]:Health test passed. Adding to load balancer

[11/13/14 14:36:55][INFO]:Load balancer pool myworld found

[11/13/14 14:36:55][INFO]:Adding server to load balancer

[11/13/14 14:37:06][INFO]:web 05 Found. Processing server

[11/13/14 14:37:06][INFO]:Health test passed. Adding to load balancer

[11/13/14 14:37:06][INFO]:Load balancer pool myworld found

[11/13/14 14:37:06][INFO]:Adding server to load balancer

[11/13/14 14:37:12][INFO]:web 06 Found. Processing server

[11/13/14 14:37:12][INFO]:Health test passed. Adding to load balancer

[11/13/14 14:37:12][INFO]:Load balancer pool myworld found

[11/13/14 14:37:13][INFO]:Adding server to load balancer

[11/13/14 14:37:23][INFO]:web 07 Found. Processing server

[11/13/14 14:37:23][INFO]:Health test passed. Adding to load balancer

[11/13/14 14:37:23][INFO]:Load balancer pool myworld found

[11/13/14 14:37:24][INFO]:Adding server to load balancer

[11/13/14 14:37:34][INFO]:web 08 Found. Processing server

[11/13/14 14:37:34][INFO]:Health test passed. Adding to load balancer

[11/13/14 14:37:34][INFO]:Load balancer pool myworld found

[11/13/14 14:37:35][INFO]:Adding server to load balancer

[11/13/14 14:37:45][INFO]:web 09 Found. Processing server

[11/13/14 14:37:45][INFO]:Health test passed. Adding to load balancer

[11/13/14 14:37:46][INFO]:Load balancer pool myworld found

[11/13/14 14:37:46][INFO]:Adding server to load balancer

[11/13/14 14:37:57][INFO]:web 10 Found. Processing server

[11/13/14 14:37:57][INFO]:Health test passed. Adding to load balancer

[11/13/14 14:37:57][INFO]:Load balancer pool myworld found

[11/13/14 14:37:57][INFO]:Adding server to load balancer

[11/13/14 14:38:08][INFO]:web 11 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 12 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 13 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 14 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 28 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 29 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 30 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 31 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 32 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 33 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 60 does not exist.Skipping...

<truncated>

[11/13/14 14:38:08][INFO]:web 97 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 98 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:web 99 does not exist.Skipping...

[11/13/14 14:38:08][INFO]:Printing summary

[11/13/14 14:38:08][INFO]:Load balancer pool myworld found

[11/13/14 14:38:08][INFO]:Nodes: 1.1.1.1 80

[11/13/14 14:38:08][INFO]:Nodes: 162.242.254.113 18001

[11/13/14 14:38:08][INFO]:Nodes: 162.242.255.71 18002

[11/13/14 14:38:08][INFO]:Nodes: 162.242.254.113 18003

[11/13/14 14:38:08][INFO]:Nodes: 162.242.255.71 18004

[11/13/14 14:38:08][INFO]:Nodes: 162.242.254.215 18005

[11/13/14 14:38:08][INFO]:Nodes: 162.242.254.215 18006

[11/13/14 14:38:08][INFO]:Nodes: 162.242.254.113 18007

[11/13/14 14:38:08][INFO]:Nodes: 162.242.254.215 18008

[11/13/14 14:38:08][INFO]:Nodes: 162.242.255.71 18009

[11/13/14 14:38:08][INFO]:Nodes: 162.242.255.73 18010

[11/13/14 14:38:08][INFO]:Sleeping 10 seconds...

  • This covers our initial exploration of Docker, CoreOS and Fleet.  There is more these tools can do to help with tighter integration but overall this combination is a good way to manage docker containers and run serious workloads on it.

Misc Commands

A collection of random snippets that are useful.
  • Build without cache. This burnt me the first time. Ubuntu removes old package versions from their repos and if a cached image has that version, apt-get install will try to pull that and fail. Needless to say --no-cache will takelonger to build.
docker build --no-cache
  •  Review logs of etcd and fleet
journalctl -u etcd

journalctl -u fleet



  •  Delete all containers
docker stop $(docker ps -a -q)

sleep 2

docker rm $(docker ps -a -q)



  •  Delete all images
docker rmi $(docker images -q)



  • Cleanup fleet
fleetctl destroy $(fleetctl list-units -fields=unit -no-legend)

fleetctl destroy $(fleetctl list-unit-files -fields=unit -no-legend)

sleep 5

fleetctl list-unit-files

fleetctl list-units

  • Restart Fleet
sudo systemctl restart fleet.service

Resources

  • Free Rackspace developer account - https://developer.rackspace.com/signup/
  • Core OS - https://coreos.com/docs/
  • A good overview on why docker was created. Done by dotCloud founder and CTO Solomon Hykes - https://www.youtube.com/watch?v=Q5POuMHxW-0  -
  • Getting Started with systemd - https://coreos.com/docs/launching-containers/launching/getting-started-with-systemd/

Thursday, 8 May 2014

Chromecast your life


Chromecast is fundamentally simple and powerful.  The price is amazingly low but that does not mean it lacks functionality. To me, it simplifies the smart TV model by putting the "smart" elsewhere. In this case it is your device which could be a laptop, phone or tablet.  Because of Google's weight behind it, it already has ton of apps who support it.

So how am I using ? Simply put I'm using it in two cases.

TV
This is straightforward and this works fine out of the box. Connect your Chromecast and follow instructions on the screen.

Audio
This needed some additional hardware. Buy this Audio Extractor and it will extract audio from the HDMI. It does not require any separate power source.  Any now you can attach this to any speaker.

Now I have 4 such setups on different Chromecast devices. My speaker collection is a mix of cheap devices and some better speakers. For couple of my rooms like the kitchen, I am using a cheap Logitech Z120 Stereo Speakers which works fine simple audio. This does not so advanced things like the same audio in each room, although given that the API is open, you can write an app to do that.

My App Ecosystem
Youtube
Plex -
BeyondPod
Google Play
Netflix
AllCast



Links and Further Reading




Tuesday, 18 March 2014

Chef Backups

There are few ways to backup a Chef server. Opscode has some documentation on their wiki https://wiki.opscode.com/display/chef/Backing+Up+Chef+Server

Some of this is outdated now because chef no longer uses Couch DB under the hood. However there is this little gem (pun intended) called knife-backup. To put this to test, install it first
    gem install knife-backup

More details are here https://github.com/mdxp/knife-backup An execution looks like this
    knife backup export -D ./backups
    Backing up clients
    Backing up clients chef-validator
    Backing up clients chef-webui
    Backing up clients test01.example.com
    Backing up clients web01.example.com
    Backing up nodes
    ...Output Truncated...

This nicely exports all your settings such as nodes, clients, roles, environments and cookbooks into the backups directory. Typically your cookbooks would be version controlled via Git or some other revision control system and you can restore it from there as well. This method allows you to completely mirror cookbooks from one chef server to another along with the other stuff in couple of simple commands.
    ls backups
    clients      cookbooks    data_bags    environments nodes        roles
To test your backup spin up a new server and install chef per your OS http://www.opscode.com/chef/install If you want to try this on a existing server, you can use the following

  ** This will erase all your chef server data **
    sudo chef-server-ctl cleanse
    sudo chef-server-ctl reconfigure

Copy _/etc/chef-server/admin.pem_ from the new server to your local workstation. You will use this user to perform the restore. Once you have restored you can use other clients/users that you were using with the original server.
    knife backup restore -D ./backups -u admin -k <path to admin key> -s  <new server url>
This will restore to the new server with the exception of few things. This is because knife restore does not overwrite existing clients.

 1. The _admin_ user and it's credentials.

 2. The _chef-webui_. This is used by the web-interface and so it makes sense to leave it.

 3. The _chef-validator_ client. Now this has some implications. chef-validator's key is used on a node when it runs the chef-client for the first time in order to get an API client identity. Since this is now different from your original server, and if you are using knife to bootstrap nodes, you will need to re-copy this to your knife workstation setup. Existing nodes don't need this as they are already registered.

All said this is a handy tool and with a little bit of scripting you can run these backups hourly/daily and use time stamped directories.

Monday, 29 April 2013

Chef Experiments - Create Users



The objective here is to create  a users cookbook with data bags

Create the data bag

knife data bag create user_config

Create the user json file

data_bags/users/usr_sri.json 

"id": "sri",
{

    "comment": "Sriram Rajan",

    "uid": 2000,

    "gid": 0,

    "home":"/home/sri",

    "shell":"/bin/bash",

    "pubkey":"<replace with the SSH public key"

}


Import the file

knife data bag from file users_config usr_sri.json


Create a key for the encrypted data bag

openssl rand -base64 512 > data_bags/users/enckey


Create the encrypted data bag

knife data bag create --secret-file  data_bags/users/enckey password_config pwdlist



Edit the data bag

knife data bag edit --secret-file  data_bags/users/enckey password_config pwdlist


"id": "pwdlist",
{
"sri": "Replace with SHA password string"
}


At this point you should have a data bag with users and encrypted data bag with passwords. Now we move to the cookbook

Create the cookbook

knife cookbook create user_config

Recipe looks like this. We add the user and also ensure the .ssh directory is created and populated with the public keys. The password will be pulled from the encrypted bag.


decrypted = Chef::EncryptedDataBagItem.load("password_config", "pwdlist")
search(:user_config, "*:*").each do |user_data|
    user user_data['id'] do
        comment user_data['comment']
        uid user_data['uid']
        gid user_data['gid']
        home user_data['home']
        shell user_data['shell']
        manage_home true
        password decrypted[user_data['id']
        action:create
    end
  
    ssh_dir = user_data['home'] + "/.ssh"
    directory ssh_dir do
        owner user_data['uid']
        group user_data['gid']
        mode "0700"
    end

    template "#{ssh_dir}/authorized_keys" do
        owner user_data['uid']
        group user_data['gid']
        mode "0600"
        variables(
             :ssh_keys => user_data['pubkey']
             )
        source "authorized_keys.erb"
    end
end

The template file

base_users/templates/default/authorized_keys.erb 

<% Array(@ssh_keys).each do |key| %>

<%= key %>

<% end %>



Finishing up
knife cookbook upload user_config

Ensure the secret key for the encrypted data bag is also sent to the node and stored under  /etc/chef/encrypted_data_bag_secret. You can bootstrap this file into the node build. See http://docs.opscode.com/essentials_data_bags_encrypt.html

Then add the recipe to a role or node run list and run the chef-client to test.

Designing in the cloud


Service based model
This is not a very new concept (http://en.wikipedia.org/wiki/Service-oriented_architecture)  but the cloud model makes this very important.  Build your business model  such that it can be consumed as a service. This would also force you to modularize parts and all this would ensure you have a high degree of portability.


Build for failure
Cloud is multi-tenant in most cases and with it comes challenges like noisy neighbours or failure of individual components.  Build for these scenarios.  Symian army (http://techblog.netflix.com/2011/07/netflix-simian-army.html) talks a lot about this and is an interesting read.  Importantly plan for "What happens when"

In building for failure you are also creating a good recovery model.  One the benefits of running everything as code means that you can recover faster and this would translate into better uptime.

Cloud is all about the API and pluggabiliity. Think about  building a top level API for your business model. Then use vendor APIs and plug them into your API.  Wherever possible, loosely couple your application interaction. For e.g., instead of direct database calls use an API


Monitoring
Monitoring becomes more than just making sure your applications are working fine. If you leverage multiple cloud providers, you can use it to make operational decisions.  You can use it to go with the best cloud provider and save costs.  You can use it to find low performing instances within the same provider.  One important point here is to make sure your monitoring is vendor agnostic and wherever possible not a tool provided by the vendor. Frameworks like Sensu  (http://www.sonian.com/cloud-monitoring-sensu/)  or tools like Riemann(http://riemann.io) can help


Automation
Cloud will force automation to a large extent and you need to embrace it.  Automation also allows you to build across different vendors  When using multiple vendors, us a model that works on all platforms. There are open source libraries like libcloud which provide vendor agnostic ways.   Be careful with using automation providers as you can get vendor lock-in in a different way.  While building your own autoscale model is complex in the long term, there is a lot more to gain as it will fit your business model.


Think about data
Cloud provides commodity based services for things like compute, storage etc but your data is not commodity. So think about distributing this over different vendors or build that into your recovery model.


Think about security
Security in the cloud is a hot topic and it is safe to say that this is still evolving.  This is also something that is overlooked while you plug in other nuts and bolts.  Make sure things like identity management, access control models are at the heart of your cloud strategy.  Even if security is not an immediate requirement, you can build them as services which can be implemented at a later stage.


Wednesday, 13 March 2013

Chef Experiments - Create SSH config


The objective here is to create  a simple sshd cookbook for Red Hat/CentOS configuration.

Create the cookbook

knife cookbook create sshd


Create the default recipe.
Options like sshd port , banner etc will be pulled from a data bag called base_config.   The template file for SSHD configuration would be sshd.erb.


File : cookbooks/sshd/recipes/default.rb
sshd_config = data_bag_item('base_config', 'sshd')
template "/etc/ssh/sshd_config" do
    source "sshd.erb"
    mode "0644"
    variables(
    :sshd_port => sshd_config['port'],
    :x11_forwarding => sshd_config['x11_forwarding'],
    :banner => sshd_config['banner'],
    :permit_root => sshd_config['permit_root']
)
end

template "/etc/issue.net" do
    source "issue.net.erb"
end

service "sshd" do
    action [ :restart ]
end
Templates sshd.erb looks like this
File : cookbooks/sshd/template/sshd.erb
Port <%= @sshd_port %>

Protocol 2

#other SSHD config has been omitted for the sake of the blog post

PermitRootLogin <%= @permit_root %>

X11Forwarding <%= @x11_forwarding %>

Banner <%= @banner %>

#other SSHD config has been omitted for the sake of the blog post

In issue.net.erb we are are reading from motd and adding a little blurb after that
File : cookbooks/sshd/template/issue.net.erb
<%= File.read("/etc/motd") %>

*****************************
Use of the Site by unauthorized users is prohibited and 
unauthorized users will be prosecuted to the fullest 
extent of the law.
*****************************
Data bag
Create the data bag
File :data_bags/base)config/config.json 
{  
  "id": "sshd",  
  "port": "2222",  
  "x11_forwarding": "no",  
  "banner":"/etc/issue.net",  
  "permit_root": "no"  
}  
Finishing up
knife data bag create base_config
knife data bag from file base_config data_bags/base_config/config.json 
knife cookbook upload sshd

Then add the recipe to a role or node run list and run the chef-client

Wednesday, 6 March 2013

Chef Experiments - Create host files

I am in the process of experimenting with Chef and here's one of them

knife create cookbook host_file_update


Then create the recipe

recipes/default.rb

hosts = search(:node, "*:*")
template "/etc/hosts" do
  source "hosts.erb"
  owner "root"
  group "root"
  mode 0644
  variables(
    :hosts => hosts,
    :hostname => node[:hostname],
    :fqdn => node[:fqdn]
  )
end
And then the template file hosts.erb referenced above

templates/default/hosts.erb 
127.0.0.1   localhost

<% @hosts.each do |node| %>
<%= node['ipaddress'] %> <%= node['hostname'] %> <%= node['fqdn'] %>
<% end %>

Pretty useful, if you want to populate this automatically as and when you add servers.  One of the next things to try is see if we can make Chef pick the additional IPs (e.g service net in Rackspace cloud) and create separate entries for it



Sunday, 3 March 2013

Mysql Information Schema

Re-publishing from my wiki.
  • Show tables that use Barracuda disk format
select * from INFORMATION_SCHEMA.TABLES where TABLE_SCHEMA NOT IN ('mysql', 'INFORMATION_SCHEMA', 'performance_schema') AND ( ROW_FORMAT='Compressed' OR ROW_FORMAT='Dynamic'); 

  • Show me all tables that are InnoDB 
SELECT `table_schema`, `table_name` FROM `information_schema`.`TABLES` WHERE `Engine`='Innodb' AND `TABLE_SCHEMA` !='information_schema' AND `TABLE_SCHEMA` !='mysql'; 

  • Show me all tables that are MyISAM 
SELECT `table_schema`, `table_name` FROM `information_schema`.`TABLES` WHERE `Engine`='MyISAM' AND `TABLE_SCHEMA` !='information_schema' AND `TABLE_SCHEMA` !='mysql'; 

  • Print Queries to aid in conversion FROM MyISAM to InnoDB 
 use `information_schema`; SELECT CONCAT("ALTER TABLE `" , `TABLE_SCHEMA`, "`.`", `table_name`, "` Engine=Innodb;") AS "" FROM `information_schema`.`TABLES` WHERE `Engine`='MyISAM' AND `TABLE_SCHEMA` !='information_schema' AND `TABLE_SCHEMA` !='mysql'; 

You can save the above in a file and run this
mysql --batch < input.sql > out.sql 

  • Show me a count of tables grouped by engine type 
SELECT `Engine`, count(*) as Total FROM `information_schema`.`TABLES` WHERE `TABLE_SCHEMA` !='information_schema' AND `TABLE_SCHEMA` !='mysql' GROUP BY `Engine`;

  • Show me the datasize and index size of all tables grouped by engine type
SELECT `Engine`, COUNT(ENGINE), sum(data_length)/(1024*1024*1024) as 'Datasize-GB', sum(index_length)/(1024*1024*1024) as 'Indexsize-GB' FROM `information_schema`.`TABLES` GROUP BY `Engine`; 

  • Show me the top 10 tables by size outside of information_schema and mysql 
SELECT TABLE_SCHEMA, TABLE_NAME,data_length/1024*1024 FROM `information_schema`.`TABLES` WHERE `TABLE_SCHEMA` !='information_schema' AND `TABLE_SCHEMA` !='mysql' ORDER BY `data_length` DESC LIMIT 10; 

  • Tables without indexes
USE `information_schema`; SELECT CONCAT(TABLES.table_schema,".",TABLES.table_name) as name, `TABLES`.`TABLE_TYPE`,`TABLE_ROWS` FROM `TABLES` LEFT JOIN `TABLE_CONSTRAINTS` ON `TABLES`.`table_schema` = `TABLE_CONSTRAINTS`.`table_schema` AND `TABLES`.`table_name` = `TABLE_CONSTRAINTS`.`table_name` AND `TABLE_CONSTRAINTS`.`constraint_type` = 'PRIMARY KEY' WHERE `TABLE_CONSTRAINTS`.`constraint_name` IS NULL; Check for redundant indexes SELECT * FROM ( SELECT `TABLE_SCHEMA`, `TABLE_NAME`, `INDEX_NAME`, GROUP_CONCAT(`COLUMN_NAME` ORDER BY `SEQ_IN_INDEX`) AS columns FROM `information_schema`.`STATISTICS` WHERE `TABLE_SCHEMA` NOT IN ('mysql', 'INFORMATION_SCHEMA') AND NON_UNIQUE = 1 AND INDEX_TYPE='BTREE' GROUP BY `TABLE_SCHEMA`, `TABLE_NAME`, `INDEX_NAME` ) AS i1 INNER JOIN ( SELECT `TABLE_SCHEMA`, `TABLE_NAME`, `INDEX_NAME`, GROUP_CONCAT(`COLUMN_NAME` ORDER BY `SEQ_IN_INDEX`) AS columns FROM `information_schema`.`STATISTICS` WHERE INDEX_TYPE='BTREE' GROUP BY `TABLE_SCHEMA`, `TABLE_NAME`, `INDEX_NAME` ) AS i2 USING (`TABLE_SCHEMA`, `TABLE_NAME`) WHERE i1.columns != i2.columns AND LOCATE(CONCAT(i1.columns, ','), i2.columns) = 1 

  • List character sets 
 SELECT `TABLE_SCHEMA`, `TABLE_NAME`, `CHARACTER_SET_NAME`, `TABLE_COLLATION` FROM `INFORMATION_SCHEMA`.`TABLES` INNER JOIN `INFORMATION_SCHEMA`.`COLLATION_CHARACTER_SET_APPLICABILITY` ON (`TABLES`.`TABLE_COLLATION` = `COLLATION_CHARACTER_SET_APPLICABILITY`.`COLLATION_NAME`) WHERE `TABLES`.`TABLE_SCHEMA` !='information_schema' AND `TABLES`.`TABLE_SCHEMA` !='mysql' ;

  • List average row length and index length 
SELECT CONCAT (`TABLE_SCHEMA`, "." , `TABLE_NAME`) as name , `AVG_ROW_LENGTH`, `DATA_LENGTH`, `INDEX_LENGTH` FROM `TABLES` ORDER BY `AVG_ROW_LENGTH` DESC LIMIT 15; 

  • Oldest tables with respect to update times 
SELECT CONCAT (`TABLE_SCHEMA`, "." , `TABLE_NAME`) as name , `UPDATE_TIME` FROM `TABLES` WHERE `UPDATE_TIME` IS NOT NULL ORDER BY `UPDATE_TIME` LIMIT 10; 

  • Tables with foreign keys 
SELECT * FROM `table_constraints` WHERE `constraint_type` = 'FOREIGN KEY' ; 

  • List of indexes and their total count 
SELECT `INDEX_TYPE`, count(*) as NUM FROM `STATISTICS` group by `INFORMATION_SCHEMA`.`INDEX_TYPE`; 

  • Get a summary of privileges 
SELECT * from `INFORMATION_SCHEMA`.`USER_PRIVILEGES`;

Wednesday, 20 February 2013

Rackspace cloud files - symlinks / aliases



The requirement is to have multiple names to the same object. For starters, there is no inbuilt way to do aliases or multiple names to the same object. However, after some documentation trolling there is a way to achieve it, although it is not straightforward.

Cloud files offers large file (over 5G) support by allowing multiple segments to be uploaded and then a manifest that links the segments.

http://www.rackspace.com/blog/rackspace-cloud-files-now-supporting-extremely-large-file-sizes/

You can use this feature to sort of achieve symlinking/aliasing

Here's a small example

curl -D - \

     -H "X-Auth-Key: AUTH key" \

     -H "X-Auth-User: user" \

     https://lon.identity.api.rackspacecloud.com/v1.0



curl -X PUT -H 'X-Auth-Token: AUTH TOKEN' \

https://storage101.lon3.clouddrive.com/v1/<URL>/stream/imagedata1/1 --data-binary 'Image1'



curl -X PUT -H 'X-Auth-Token: AUTH TOKEN' \

-H 'X-Object-Manifest: sriramrajan.com/imagedata1/' \

https://storage101.lon3.clouddrive.com/v1/<URL>/stream/image1.txt --data-binary ''



curl -X PUT -H 'X-Auth-Token: AUTH TOKEN' \

-H 'X-Object-Manifest: sriramrajan.com/imagedata1/' \

https://storage101.lon3.clouddrive.com/v1/<URL>/stream/image2.txt --data-binary ''



This can be on CDN enabled containers as well and so once you do this, the following will be technically pointing to the same object.
<CDN URL>/image1.txt

<CDN URL>/image2.txt


Friday, 21 September 2012

MySQL Versions - A survey


Different versions

MySQL versions make interesting and at times confusing reading. Here's a list of the version soup.

  • Version 5.2 was re-branded as version 6.0.
  • Version 6.0 was then cancelled.
  • Version 5.4 then replaced version 6.0.
  • Version 5.4 was then re-branded as version 5.5.
  • MySQL NDB cluster comes with it's own versioning system. As of this writing 7.2 is the latest release.
  • Innodb, the most used engine in MySQL now has it's own versioning. MySQL 5.0 had Innodb version 1.0.xx and MySQL 5.5 has innodb version 1.1.xx.
  • MySQL 5.6 will be the next release from MySQL(Oracle). For a full feature list see http://dev.mysql.com/tech-resources/articles/whats-new-in-mysql-5.6.html

Forks and Patches

On top of this there are quite a few forks and patches.

  • Drizzle was a fork of MySQL 6.0. Drizzle is probably the only true fork, has fully re-factored code and, is in active development.
  • MariaDB is a release driven by Michael "Monty" Widenius, the original author of MySQL. MySQL 5.1 was the basis for MariaDB and as of this writing the latest version is Maria DB 5.5.
  • Percona has several patches for both MySQL 5.1 and MySQL 5.5. Percona patches are a re-base of the main release and XtraDB is a re-base of InnoDB. Both Percona Server and XtraDB is also not a true forks of MySQL or Innodb. They have some very good performance & monitoring enhancements.
  • Facebook also releases patches for MySQL.  These are done for specific Facebook requirements but as is often the case some of these patches eventually find their way into main releases.  These can be found at https://www.facebook.com/MySQLatFacebook
  • OurDelta is another set of patches released by former MySQL employee Arjen Lentz.  This is now more aligned with MariaDB. These patches are similar to the Percona ones in terms of adding extra functionality to existing releases.

3rd Party Storage Engines

Outside of InnoDB and MyISAM, there are few 3rd party plugins that suit specific needs as well.


Selecting the right one

  • For most part, staying with the Oracle releases will suit most application needs. These releases are in active development, generally stable and have binary versions for most systems.
  • MariaDB is binary compatible with MySQL and may suit certain needs. It also includes XtraDB from Percona. For a good comparison and the various incompatibilities, refer to http://kb.askmonty.org/en/mariadb-versus-mysql/
  • Going with a fork like Drizzle will likely require some application change. Drizzle does not maintain server level compatibility but does talk the MySQL protocol. For a full list of differences refer to http://docs.drizzle.org/mysql_differences.html
  • The Percona patches and versions provide a more seamless migration as they are generally fully compatible with the MySQL releases. If you are concerned about vendor support , Percona also provides that. Percona toolkit (formerly MaatKit) is a good addition as it provides some nice tools to automate a variety of tasks. This toolkit can be used with any MySQL release and so you don't need to run Percona server.
  • The 3rd party engines come into a play only if are looking for specific features that come with the engines.

Links & further reading

URL fun


Some interesting ways, URLs work


http://sriramrajan.com - The conventional one

http://46.38.167.114 - The IP addresss

http://774285170 - The decimal converted

http://0x2e.0x26.0xa7.0x72 - The hexadecimal converted

http://0x2e26a772 -  Another hexadecimal variant



Wednesday, 6 June 2012

IPv6 Tunnel

Get a free IPv6 IP from a broker like http://tunnelbroker.net/

Configure the tunnel
modprobe ipv6
ip tunnel add he-ipv6 mode sit remote  local  ttl 255
ip link set he-ipv6 up
ip addr add  dev he-ipv6
ip route add ::/0 dev he-ipv6
ip -f inet6 addr


Check with ifconfig
#ifconfig he-ipv6
he-ipv6   Link encap:IPv6-in-IPv4  
          inet6 addr: fe80::2e26:a772/128 Scope:Link
          inet6 addr: 2001:470:1f08:1a7e::2/64 Scope:Global
          UP POINTOPOINT RUNNING NOARP  MTU:1480  Metric:1
          RX packets:411 errors:0 dropped:0 overruns:0 frame:0
          TX packets:251 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0 
          RX bytes:488490 (477.0 KiB)  TX bytes:22616 (22.0 KiB)

Test ping
# ping6 -c2 ipv6.google.com
PING ipv6.google.com(par03s02-in-x11.1e100.net) 56 data bytes
64 bytes from par03s02-in-x11.1e100.net: icmp_seq=0 ttl=57 time=10.9 ms
64 bytes from par03s02-in-x11.1e100.net: icmp_seq=1 ttl=57 time=10.0 ms
--- ipv6.google.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1002ms
rtt min/avg/max/mdev = 10.070/10.512/10.955/0.454 ms, pipe 2

Test website
# curl -6  -v http://sriramrajan.com/status.php
* About to connect() to sriramrajan.com port 80
*   Trying 2001:470:1f08:1a7e::2... connected
* Connected to sriramrajan.com (2001:470:1f08:1a7e::2) port 80
> GET /status.php HTTP/1.1
> User-Agent: curl/7.15.5 (x86_64-redhat-linux-gnu) libcurl/7.15.5 OpenSSL/0.9.8b zlib/1.2.3 libidn/0.6.5
> Host: sriramrajan.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Wed, 06 Jun 2012 20:34:45 GMT
< Server: Apache
< Vary: Accept-Encoding
< P3P: policyref="http://www.sriramrajan.com/p3p.xml",CP= "NON DSP COR CURa TIA"
< Content-Length: 7
< Content-Type: text/html; charset=UTF-8
ALL OK
* Connection #0 to host sriramrajan.com left intact
* Closing connection #0

#curl -I  -v  -g -k  http://[2001:470:1f08:1a7e::2]/status.php
* About to connect() to 2001:470:1f08:1a7e::2 port 80 (#0)
*   Trying 2001:470:1f08:1a7e::2... connected
* Connected to 2001:470:1f08:1a7e::2 (2001:470:1f08:1a7e::2) port 80 (#0)
> HEAD /status.php HTTP/1.1
> User-Agent: curl/7.21.7 (i386-redhat-linux-gnu) libcurl/7.21.7 NSS/3.13.3.0 zlib/1.2.5 libidn/1.22 libssh2/1.2.7
> Host: [2001:470:1f08:1a7e::2]
> Accept: */*
> 
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Date: Sat, 09 Jun 2012 10:35:09 GMT
Date: Sat, 09 Jun 2012 10:35:09 GMT
< Server: Apache
Server: Apache


Check SSH
# ssh -6 -p 2222 2001:470:1f08:1a7e::2
root@2001:470:1f08:1a7e::2's password: 

Firewall
Don't forget the firewall. If you use iptables for a firewall, make sure, you have equivalent setup for IPv6. You can check using
# ip6tables -nvL

Wednesday, 30 May 2012

Mysql 5.6 new features

Sunday, 20 May 2012

Cisco Anyconnect Errors


Anyconnect client gives this error


AnyConnect cannot confirm it is connected to your secure gateway.  The local network may not be trustworthy.  Please try another network.

After fighting it for a while, I found an answer in the release notes

http://www.cisco.com/en/US/docs/security/vpn_client/anyconnect/anyconnect25/release/notes/anyconnect25rn.html

Firefox 2.0 or later with libnss3.so installed in /usr/local/lib, /usr/local/firefox/lib, or /usr/lib. Firefox must be installed in /usr/lib or /usr/local, or there must be a symbolic link in /usr/lib or /usr/local called firefox that points to the Firefox installation directory.



So the following fixes it


mkdir /usr/local/firefox

cd /usr/local/firefox

ln -s /usr/lib64/libnss3.so 

ln -s /lib64/libplc4.so 

ln -s /lib64/libnspr4.so 

ln -s /usr/lib64/libsmime3.so 


Wednesday, 18 January 2012

Linux file descriptors and open modes

Ever want to find out what modes a file was opened with originally ?

First find the file descriptor number
ls /proc//fd
Eg
ls -l /proc/32048/fd/30
l-wx------ 1 apache apache 64 Jan 18 07:15 30 -> /var/log/httpd/ntop-access_log


Then check fdinfo

cat /proc/32048/fdinfo/30
pos: 0
flags: 0102001

The flags are derived from the open system call http://linux.about.com/od/commands/l/blcmdl2_open.htm

To actually decipher the octal codes , look under /usr/include/bits/fcntl.h

If there are multiple codes, the codes are appended together.

Thursday, 12 January 2012

Linux ACLs

Filesystem options and commands


First check to make sure the file system is mounted with acl settings

cat /proc/mounts |grep acl

/dev/sda1 / ext3 rw,noatime,relatime,errors=remount-ro,acl,data=ordered 0 0

If not update /etc/fstab and add 'acl' to the options section and remount the file system

getfacl, setfacl, chacl are the two main commands. chacl is available for IRIX compatibility.


Use Cases


Grant 2 users permissions to the same directory and files under it


Let's say we want to grant user john and mary permissions to folder /var/www/mysite.com

We can start by creating the directory. At this point we can leave it owned by root as the ACLs will help here.

ls -ld /var/www/mysite.com/
drwxr-xr-x 2 root root 4096 Feb 11 15:21 /var/www/mysite.com/

The first 2 commands grant users john and mary permissions.

The second sets the default acl. This causes the acls to be applied with inhertiance set. So this makes good sense in a multi user multi edit environment. The next arguments between the : are the username and the permissions

setfacl -m john:rwx mysite.com 
setfacl -m mary:rwx mysite.com 

setfacl -m default:john:rwx mysite.com 
setfacl -m default:mary:rwx mysite.com

# file: mysite.com
# owner: root
# group: root
user::rwx
user:john:rwx
group::r-x
mask::rwx
other::r-x
default:user::rwx
default:user:john:rwx
default:user:mary:rwx
default:group::r-x
default:mask::rwx
default:other::r-x

Now create a file by logging in a user john.

john@slice01$ echo "john" > file1

john@slice01$ ls -l file1 
-rw-rw-r--+ 1 john john 4 Feb 11 15:31 file1

john@slice01$ getfacl file1 
# file: file1
# owner: john
# group: john
user::rw-
user:john:rwx   #effective:rw-
user:mary:rwx   #effective:rw-
group::r-x   #effective:r--
mask::rw-
other::r--

Then create a directory

john@slice01$ mkdir john

john@slice01$ getfacl john
# file: john
# owner: john
# group: john
user::rwx
user:john:rwx
user:mary:rwx
group::r-x
mask::rwx
other::r-x
default:user::rwx
default:user:john:rwx
default:user:mary:rwx
default:group::r-x
default:mask::rwx
default:other::r-x

As you can see mary is there in the ACLs also

You can test it by logging in as user mary & editing files created by john.

mary@slice01$ echo mary >> file1 

mary@slice01$ cat file1 
john
mary

mary@slice01$ cd john/

mary@slice01 $ echo mary > file2
mary@slice01 $ getfacl file2 
# file: file2
# owner: mary
# group: mary
user::rw-
user:john:rwx   #effective:rw-
user:mary:rwx   #effective:rw-
group::r-x   #effective:r--
mask::rw-
other::r--



Grant 2 users permissions to the same directory and files under it except to 2 individual directories

Lets say we want john and mary to have permissions under /var/www/mysite.com/ and all files but still have individual directories
/var/www/mysite.com/john & /var/www/mysite.com/mary

 getfacl mary
# file: mary
# owner: mary
# group: mary
user::rwx
user:john:rwx
user:mary:rwx
group::r-x
mask::rwx
other::r-x
default:user::rwx
default:user:john:rwx
default:user:mary:rwx
default:group::r-x
default:mask::rwx
default:other::r-x


The -k switch removes the default acls

setfacl -k mary

getfacl mary
# file: mary
# owner: mary
# group: mary
user::rwx
user:john:rwx
user:mary:rwx
group::r-x
mask::rwx
other::r-x


Then remove john from it also
setfacl -x john mary

getfacl mary
# file: mary
# owner: mary
# group: mary
user::rwx
user:mary:rwx
group::r-x
mask::rwx
other::r-x

Repeat the same with other folder

Monday, 7 February 2011

Redhat 6 - Part 1

Here are some new stuff in RHEL 6

Software versions
  • PHP 5.3.1. It also ships with APC (Alternative PHP Cache).
  • Apache is 2.2.14
  • MySQL is 5.1.42
  • Tomcat is 6.0.20
  • PostgreSQL is version 8.4
  • Python is 2.6
  • Perl is 5.10.1
  • Gcc is 44.4
File systems

Other Notable Changes
  • Default use of NFS v4
  • SysV init is gone in favour of upstart. Upstart comes with legacy support for traditional init scripts in /etc/init.d.
  • Support for Fibre Channel over Ethernet (FCoE)
  • iSCSI  can now be used as root or boot devices
  • As expected, Xen has been dropped in favour or KVM

Monday, 17 January 2011

IPv6 Part 1

Some trivia


Why 128 bit ?  - It was a compromise between a fixed 64 bit and a variable 160 bits (google OSI NSAP for more)

How many IP addresses ? - 340,282,366,920,938,463,463,374,607,431,768,211,456

What about IPv5 ? -  Used by ST - a stream protocol and not related to IP


Key Differences
  • More IPs
  • Different Header format
  • Extension support
  • Flow labeling
  • Authentication support

Header

  • Very simplified (See http://en.wikipedia.org/wiki/File:Ipv6_header.svg)
  • 20 bytes + Options in IPv4 Vs  12 fields (40 bytes) in IPv6
  • Base header no longer contains fragmentation options
  • Header no longer contains any checksum
  • Time to live (TTL) is now called Hop Limit
  • Support for traffic classes
  • Extensibility in headers. Options are not limited to the 40 bytes

Sunday, 9 January 2011

TCP timers and keepalives

Netstat -o includes tcp timers which are useful for Apache keepalive analysis.

netstat -ntpo | grep ESTAB | egrep ":80|:443"

Output looks like this


The last column denotes what the connection is doing.


  • 'on' - Actively transfering data.




  • 'off' - Currently disconnecting




  • 'keepalive' - Connections are using TCP keepalives. The first number denotes the time in seconds from when the last data was transferred until when the next TCP keepalive probe will be sent. By default this starts at 7200s, and resets again every time more data is sent. If the value is low , for eg. 4000 seconds , it means some of the keep alive connections are hanging or doing nothing for a long period. Note, connections to internal proxy or other internal processes might hang longer but this should not happen to web based connection.




  • The defaults are dictated by the sysctl values :-


  • ''net.ipv4.tcp_keepalive_probes'' - How many keepalive probes TCP sends out, until it decides that the connection is broken. Default value: 9.




  • ''net.ipv4.tcp_keepalive_time'' - How often TCP sends out keepalive messages when keepalive is enabled. Default: 2hours (7200 seconds)




  • ''net.ipv4.tcp_keepalive_intvl'' - How frequently the probes are send out. Multiplied by tcp_keepalive_probes it is time to kill not responding connection, after probes started. Default value: 75sec i.e. connection will be aborted after ~11 minutes of retries.




  • More information on the sysctly values can be found in the kernel documentation ''/usr/share/doc/kernel-doc-/Documentation/networking/ip-sysctl.txt''

    Thursday, 30 December 2010

    Linux and Auditd

    Assumptions: Everything here is tested on Red Hat/Cent OS 5, and 2.6.24.XX kernel

    auditd is the userspace side of kernel auditing functions. It can be used to watch file accesses, monitory system calls, log events etc.The closest to a home page it has http://people.redhat.com/sgrubb/audit/. It has some nice presentations done at different Red Hat conferences and some other articles.

    Installation
    Installing auditd is straightforward but to use it efficiently you need to tune it to your needs. To install use

    yum install audit
    
    To start the service do
    /etc/init.d/auditd start

    Configuration

    Installation is a breeze but configuring to your needs requires some tweaking.

    auditd.conf can be used to tweak auditd's behaviour. Common settings to review/tune are :-
    • log_file (default:/var/log/audit/audit.log) & log_format(default:raw)
      Set the log file name and the format. The format can be either raw or nolog.
    • num_logs(default:0)
      Auditd does its own rotation and the default is no rotation. This keyword specifies the number of log files to keep if rotate is given as the max_log_file_action. Note, there is a small catch to this setting. Turns out when Auditd is rotating logs it won't process any logs. So on a high volume server you might miss logs during rotation. To avoid this raise the kernel backlog buffers by adding -b <num> to /etc/audit/audit.rules. The default is 64 1K buffers. This cannot be greater than 99.
    • max_log_file (default:5MB) & max_log_file_action (default:ROTATE)
      The first sets the maximum log size in megabytes. Action defines what to do when it reaches that state. Valid values are ignore, syslog, suspend, rotate and keep_logs. If set to ignore, the audit daemon does nothing. syslog means that it will issue a warning to syslog. suspend will cause the audit daemon to stop writing records to the disk. The daemon will still be alive. The rotate option will cause the audit daemon to rotate the logs.
    • space_left (default:75 MB) & space_left_action (default:SYSLOG)

    • admin_space_left (default:75 MB) & admin_space_left_action (default:SUSPEND)

    • disk_full_action (default: SUSPEND) & disk_error_action (default:SUSPEND)
      The first 2 can be explained as a warning threshold + action and critical threshold + action.
      The 3rd one defines what to do if the disk becomes full or disk errors occur when writing
      The valid actions are
      ignore - do nothing, syslog - log to syslog, email - send email to action_mail_acct,
      exec - execute a script, suspend - stop audit logging, single - single user mode, and halt - halt system
    • action_mail_acct
      Valid email address for email actions from above. This require /usr/lib/sendmail
    • flush & freqThe default is to use incremental and a count of 20 which means it will flush to disk after 20 events. Valid values are none, incremental, data, and sync. This is a trade off between disk I/O and how much log data you can afford to lose if the server loses power or abruptly reboots. The data forces data to be synced at all times & the sync forced both data and meta data to be synced at all times. The sync option will off course cause the most amount of I/O. If you can afford to lose some log data, then leaving it at default would be fine.
    • name_format & name
      This inserts the node name/host name to the log line. Default is to insert nothing. It can be set to hostname, IP address or FQDN. This makes sense if you are passing logs from several machine to a centralized location.
    • tcp_listen_portYou can tell auditd to listen for events from other machines.
    Auditd rules
    The main use of auditd is derived from the rules. /etc/audit/audit.rules is used to set configuration like watch rules, buffers etc. auditctl can also be used to change rules on a live system.

    General Settings

    • Backlog (default: 64 1K buffers)
      Turns out when Auditd is rotating logs it won't write any logs to file. So on a high volume server you might miss logs during rotation. To avoid this raise the kernel backlog buffers by adding -b <num> to /etc/audit/audit.rules. The default is 64 1K buffers. If the limit is reached the failure flag is consulted
    • Rate
      Messages/sec can be set by -r. if the rate is exceeded the failure flag is consulted.
    • Failure flag (default: 1=printk)
      Determines how the kernel will handle errors. 0 - silent, 1 - printk , 2 - panic. Even though panic sounds like a terrible idea, it can be important in high security environments, especially, if someone is trying to circumvent the logging.
    • Locking rules
      By adding -e 2 to end of the rules file, the configuration is locked and cannot be changed by the auditctl command. Attempts to do so will be logged and denied. Note, reverting this will require a reboot. This is another feature that is suited for very high security environments.
    • Listing current rules
      auditctl -l

    Watching Files

    -w path
    Wildcards are not supported and will generate a warning. The way that watches work is by tracking the inode internally. Unlike other syscall rules, watches do not impact performance. For more granular options like audit a specific user accessing a file, use the syscall auditing

    -p r|w|x|a
    Set permissions filter for a file system watch. r=read, w=write, x=execute, a=attribute change. They are not standard file permissions but the syscall the gets executed for them

    -k <key>
    The -k adds a key making it easy to search if there are several watches. One use for -k is to define different alert levels; eg. low, medium, high and then tag different rules with those levels.

    eg.
    Watch yum.conf for write and attribute changes
    Watch /sbin/service for execution
    Watch /etc/shadow for read, write and attribute changes

    Note: The same can be added to /etc/audit/audit.rules without the auditctl command

    auditctl -w /etc/yum.conf -p wa  -k yum_watch
    auditctl -w /usr/bin/nmap -p x   -k nmap_watch
    auditctl -w /etc/shadow   -p rwa -k shadow_watch
    

    To report on watched files. Date format is local to the server's date format.
    aureport -f
    aureport -f --start 02/18/10 17:42:00
    aureport -f --start 02/18/10 17:00:00 --end 02/18/10 17:10:00
    aureport -f -ts this-week
    aureport -f -ts today
    

    Output will be similar to this
    14742. 04/20/10 12:40:01 /etc/shadow 2 yes /usr/sbin/crond -1 977855
    14743. 04/20/10 12:40:01 /etc/shadow 2 yes /usr/sbin/crond -1 977851
    14744. 04/20/10 12:40:24 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977863
    14745. 04/20/10 12:40:50 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977864
    14746. 04/20/10 12:41:16 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977865
    14747. 04/20/10 12:41:42 /etc/shadow 89 no /opt/splunk/bin/splunkd 500 977866
    

    The 1st column is just an index

    The 2nd column is the date

    The 3nd column is the time

    The 4rd column is the file name

    The 5th column is the system call number. To convert this number into name use aureport -f -i

    The 6th column is the result of the system call. Success of failure. You can use aureport --failed to list only failed

    The 7th column is the process accessing it

    The 8th column is the Actual/Audit UID (AUID). If you login as user joe(uid 500) and su to root (uid=0). The AUID is 500.

    The 9th column is the event number. You can use ausearch -a <event no> to look up further details

    Note : -i is useful as it converts to more human readable value. However the conversion is done at the time of viewing. The log will contain numeric values only. So if uid 500 belongs to john at the time of logging and at a later date, uid 500 is assigned to user joe, there will be a discrepancy.

    The above produces more of a summary. You can use 'ausearch' to get more details.

    Let's take a closer look.

    nmap is set to 0700 permissions.

    [root@slice01 ~]# ls -l /usr/bin/nmap
    -rwx------ 1 root root 3580248 Jul  6  2009 /usr/bin/nmap
    

    Access by root yields the following log

    # ausearch -i -k nmap_watch 
    
    time->Fri Apr 16 16:10:18 2010
    node=slice01 type=PATH msg=audit(04/16/10 16:10:18.044:954405) : item=0 name=/usr/bin/nmap inode=721629 dev=08:01 mode=file,700 
    ouid=root ogid=root rdev=00:00 
    node=slice01 type=CWD msg=audit(04/16/10 16:10:18.044:954405) :  cwd=/root 
    node=slice01 type=EXECVE msg=audit(04/16/10 16:10:18.044:954405) : a0=nmap 
    node=slice01 type=SYSCALL msg=audit(04/16/10 16:10:18.044:954405) : arch=x86_64 
    syscall=execve per=400000 success=yes exit=0 a0=6e2bd0 a1=6e5160 a2=7ae510 a3=0 items=1 
    ppid=7318 pid=14772 auid=sri uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 
    comm=nmap exe=/usr/bin/nmap key=nmap_watch 
    
    


    Access by user with uid 500 who has been granted access results in this

    ----
    time->Fri Apr 16 16:11:44 2010
    node=slice01 type=PATH msg=audit(04/16/10 16:11:44.470:954410) : item=0 name=/usr/bin/nmap inode=721629 dev=08:01 mode=file,700 
    ouid=root ogid=root rdev=00:00 
    node=slice01 type=CWD msg=audit(04/16/10 16:11:44.470:954410) :  cwd=/root 
    node=slice01 type=SYSCALL msg=audit(04/16/10 16:11:44.470:954410) : arch=x86_64 
    syscall=execve success=no exit=-13(Permission denied) a0=6f26e0 a1=6f2260 a2=6cf8e0 a3=0 items=1 
    ppid=13286 pid=15089 auid=sri uid=sri gid=sri euid=sri suid=sri fsuid=sri egid=sri sgid=sri fsgid=sri tty=pts1 
    comm=bash exe=/bin/bash key=nmap_watch 
    

    success=<yes|no> shows if the system call was successful

    Again the -i lists output in more human readable format and converts syscall numbers and user ids into names.


    To remove a rule using auditctl you can use the -W switch

    auditctl -W /etc/shadow -p rwa -k shadow_watch
    
    Another way to watch files is via system calls. See below.

    Watching system calls
    You can use this to watch any system call. There are various options available here that can be combined to audit different types of events.

    Required switches with syscall monitoring are the -a list,action -A list,action

    -a
    appends to the end of the list

    -A appends to the start of the list

    The corresponding switch to the -a is the -d switch which can delete rules of a certain type

    The list values can be

    entry/exit - This determines when to log with respect to the system call invocation

    exclude - This can be used to filter events


    The action values can be :-

    never - No audit records will be generated. The order of rules in the audit.rules file matters. Normally you would put suppressions at the top

    always - Always write out a record

    The -S switch defines the system call to watch. It can be name or the system call number

    The next switch is the -F which builds a rule field. Several of these can be grouped together to filter various stuff.

    Common things to use with this are :-

    arch - cpu architecture

    auid/uid - The original ID the user logged in with & the user id

    euid/egid - Effective user and group ID

    path/dir - Full path of the file or directory to watch. Directory watches are recursive

    filetype - Type of the file. file, dir, socket, symlink, char, block, or fifo

    perm - permission filter for file operation

    pid/ppid - process id and parent process id


    Watching for ptrace system call. Utilities like strace use it

    auditctl -a entry,always -F arch=b64 -S ptrace -k info_scan
    


    Suppressing 32bit clock_gettime & fstat64 system calls

    -a entry,never -F arch=b32 -S clock_gettime -k clock_gettime
    -a entry,never -F arch=b32 -S fstat64 -k fstat64
    


    Audit files opened by a specific user. The first rule will audit all open files. It is a good idea to watch for both auid and uid.
    Also the architecture is required with system calls like these

    auditctl -a exit,always -S open -F auid=2010
    auditctl -a exit,always -F arch=b64 -F auid=2010  -F uid=2010 -F path=/etc/hosts -S open
    

    To search you can use the key or use the -sc switch

    ausearch -k info_scan -i
    ausearch -sc ptrace -i
    
    To search by user id

    ausearch -ua 2010
    


    Authentication Report

    Lists all auth attempts and their result. This also includes logins by other means, eg. imap access

    aureport -au
    
    To list just logins

    aureport -l
    
    To list account modification attempts. This lists only successful attempts. For eg. it won't list an attempt made by an unprivileged user.

    aureport -m
    

    Rules for security standards

    /usr/share/doc/audit-<version> has pre-defined rules for different standards

    Controlled Access Protection Profile (CAPP) - http://www.commoncriteriaportal.org/files/ppfiles/capp.pdf

    Labeled Security Protection Profile (LSPP) - http://www.commoncriteriaportal.org/files/ppfiles/lspp.pdf

    National Industrial Security Program Operating Manual (NISPOM) - http://www.fas.org/sgp/library/nispom.htm

    Security Technical Implementation Guides - http://iase.disa.mil/stigs/stig/index.html

    Best Practices
    • If you want everything audited you should add a boot param of audit=1 to Grub. Otherwise certain processes that start before audit loads will not be monitored.
    • /var/log/audit should be on its own partition.
    • Use Syscall rules with care. They will lead to performance degradation if overused
    Commands Listing
    Adding/Modifying Rules
    • Watch for files
    auditctl -w /etc/yum.conf -p wa -k yum_watch  
    auditctl -w /usr/bin/nmap -p x -k nmap_watch  
    auditctl -w /etc/shadow -p rwa -k shadow_watch
      • Remove a rule using auditctl
      auditctl -W /etc/shadow -p rwa -k shadow_watch
        • Watching for ptrace system call
        auditctl -a entry,always -F arch=b64 -S ptrace -k info_scan
          • Suppress 32bit clock_gettime & fstat64 system calls
          auditctl -a entry,never -F arch=b32 -S clock_gettime -k clock_gettime -a entry,never -F arch=b32 -S fstat64 -k fstat64
            • Audit files opened by a specific user
            auditctl -a exit,always -S open -F auid=2010 auditctl -a exit,always -F arch=b64 -F auid=2010 -F uid=2010 -F path=/etc/hosts -S open
              • Audit unsuccessful attempts for multiple system calls where user id is greater than or equal to 500
              auditctl -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EACCES -F auid>=500 
              auditctl -a always,exit -F arch=b32 -S creat -S open -S openat -S truncate -S ftruncate -F exit=-EPERM -F auid>=500
                Reporting/Searching
                • List all rules
                auditctl -l
                  • List status
                  auditctl -s
                    • Report on watched files. Date format is local to the server's date format.
                    aureport -f aureport -f --start 02/18/10 17:42:00 
                    aureport -f --start 02/18/10 17:00:00 --end 02/18/10 17:10:00 
                    aureport -f -ts this-week aureport -f -ts today
                      • Search by system call
                      ausearch -sc ptrace -i
                        • Search for user id or effective user id
                        ausearch -ui 2010 ausearch -ue 2010
                          • Lists all auth attempts and their result
                          aureport -au
                            • List just logins
                            aureport -l
                              • List account modification attempts.
                              aureport -m
                                • Search events where success value is no, User id is 500 and key is nmap_watch
                                ausearch -sv no -ua 500 -k nmap_watch
                                  • Search by executable
                                  ausearch -x /usr/bin/nmap
                                    • Search by terminal
                                    ausearch -tm pts/0
                                      • Search by daemon. Stuff like cron log terminal as the daemon name
                                      ausearch -tm cron

                                        Audit data visualisation

                                        mkgraph & mkgraph are two scripts that make use of gnuplot to plot graphs using the data from aureport and ausearch.

                                        More of this at http://people.redhat.com/sgrubb/audit/visualize/index.html