Chef Backups

There are few ways to backup a Chef server. Opscode has some documentation on their wiki https://wiki.opscode.com/display/chef/Backing+Up+Chef+Server

Some of this is outdated now because chef no longer uses Couch DB under the hood. However there is this little gem (pun intended) called knife-backup. To put this to test, install it first

    gem install knife-backup

More details are here https://github.com/mdxp/knife-backup An execution looks like this

    knife backup export -D ./backups
    Backing up clients
    Backing up clients chef-validator
    Backing up clients chef-webui
    Backing up clients test01.example.com
    Backing up clients web01.example.com
    Backing up nodes
    ...Output Truncated...

This nicely exports all your settings such as nodes, clients, roles, environments and cookbooks into the backups directory. Typically your cookbooks would be version controlled via Git or some other revision control system and you can restore it from there as well. This method allows you to completely mirror cookbooks from one chef server to another along with the other stuff in couple of simple commands.

    ls backups
    clients      cookbooks    data_bags    environments nodes        roles

To test your backup spin up a new server and install chef per your OS http://www.opscode.com/chef/install If you want to try this on a existing server, you can use the following

  ** This will erase all your chef server data **

    sudo chef-server-ctl cleanse
    sudo chef-server-ctl reconfigure

Copy _/etc/chef-server/admin.pem_ from the new server to your local workstation. You will use this user to perform the restore. Once you have restored you can use other clients/users that you were using with the original server.

    knife backup restore -D ./backups -u admin -k <path to admin key> -s  <new server url>

This will restore to the new server with the exception of few things. This is because knife restore does not overwrite existing clients.

 1. The _admin_ user and it's credentials.

 2. The _chef-webui_. This is used by the web-interface and so it makes sense to leave it.

 3. The _chef-validator_ client. Now this has some implications. chef-validator's key is used on a node when it runs the chef-client for the first time in order to get an API client identity. Since this is now different from your original server, and if you are using knife to bootstrap nodes, you will need to re-copy this to your knife workstation setup. Existing nodes don't need this as they are already registered.

All said this is a handy tool and with a little bit of scripting you can run these backups hourly/daily and use time stamped directories.

Chef Experiments - Create Users

The objective here is to create  a users cookbook with data bags

Create the data bag

knife data bag create user_config

Create the user json file

data_bags/users/usr_sri.json 

"id": "sri",
{

    "comment": "Sriram Rajan",

    "uid": 2000,

    "gid": 0,

    "home":"/home/sri",

    "shell":"/bin/bash",

    "pubkey":"<replace with the SSH public key"

}

Import the file

knife data bag from file users_config usr_sri.json


Create a key for the encrypted data bag

openssl rand -base64 512 > data_bags/users/enckey


Create the encrypted data bag

knife data bag create --secret-file  data_bags/users/enckey password_config pwdlist



Edit the data bag

knife data bag edit --secret-file  data_bags/users/enckey password_config pwdlist

"id": "pwdlist",
{
"sri": "Replace with SHA password string"
}

At this point you should have a data bag with users and encrypted data bag with passwords. Now we move to the cookbook

Create the cookbook

knife cookbook create user_config

Recipe looks like this. We add the user and also ensure the .ssh directory is created and populated with the public keys. The password will be pulled from the encrypted bag.

decrypted = Chef::EncryptedDataBagItem.load("password_config", "pwdlist")
search(:user_config, "*:*").each do |user_data|
    user user_data['id'] do
        comment user_data['comment']
        uid user_data['uid']
        gid user_data['gid']
        home user_data['home']
        shell user_data['shell']
        manage_home true
        password decrypted[user_data['id']
        action:create
    end
  
    ssh_dir = user_data['home'] + "/.ssh"
    directory ssh_dir do
        owner user_data['uid']
        group user_data['gid']
        mode "0700"
    end

    template "#{ssh_dir}/authorized_keys" do
        owner user_data['uid']
        group user_data['gid']
        mode "0600"
        variables(
             :ssh_keys => user_data['pubkey']
             )
        source "authorized_keys.erb"
    end
end

The template file

base_users/templates/default/authorized_keys.erb 

<% Array(@ssh_keys).each do |key| %>

<%= key %>

<% end %>

Finishing up

knife cookbook upload user_config

Ensure the secret key for the encrypted data bag is also sent to the node and stored under  /etc/chef/encrypted_data_bag_secret. You can bootstrap this file into the node build. See http://docs.opscode.com/essentials_data_bags_encrypt.html

Then add the recipe to a role or node run list and run the chef-client to test.

Designing in the cloud

Service based model
This is not a very new concept (http://en.wikipedia.org/wiki/Service-oriented_architecture)  but the cloud model makes this very important.  Build your business model  such that it can be consumed as a service. This would also force you to modularize parts and all this would ensure you have a high degree of portability.


Build for failure
Cloud is multi-tenant in most cases and with it comes challenges like noisy neighbours or failure of individual components.  Build for these scenarios.  Symian army (http://techblog.netflix.com/2011/07/netflix-simian-army.html) talks a lot about this and is an interesting read.  Importantly plan for "What happens when"

In building for failure you are also creating a good recovery model.  One the benefits of running everything as code means that you can recover faster and this would translate into better uptime.

Cloud is all about the API and pluggabiliity. Think about  building a top level API for your business model. Then use vendor APIs and plug them into your API.  Wherever possible, loosely couple your application interaction. For e.g., instead of direct database calls use an API

Monitoring
Monitoring becomes more than just making sure your applications are working fine. If you leverage multiple cloud providers, you can use it to make operational decisions.  You can use it to go with the best cloud provider and save costs.  You can use it to find low performing instances within the same provider.  One important point here is to make sure your monitoring is vendor agnostic and wherever possible not a tool provided by the vendor. Frameworks like Sensu  (http://www.sonian.com/cloud-monitoring-sensu/)  or tools like Riemann(http://riemann.io) can help


Automation
Cloud will force automation to a large extent and you need to embrace it.  Automation also allows you to build across different vendors  When using multiple vendors, us a model that works on all platforms. There are open source libraries like libcloud which provide vendor agnostic ways.   Be careful with using automation providers as you can get vendor lock-in in a different way.  While building your own autoscale model is complex in the long term, there is a lot more to gain as it will fit your business model.


Think about data
Cloud provides commodity based services for things like compute, storage etc but your data is not commodity. So think about distributing this over different vendors or build that into your recovery model.


Think about security
Security in the cloud is a hot topic and it is safe to say that this is still evolving.  This is also something that is overlooked while you plug in other nuts and bolts.  Make sure things like identity management, access control models are at the heart of your cloud strategy.  Even if security is not an immediate requirement, you can build them as services which can be implemented at a later stage.

Chef Experiments - Create SSH config

The objective here is to create  a simple sshd cookbook for Red Hat/CentOS configuration.

Create the cookbook

knife cookbook create sshd


Create the default recipe.
Options like sshd port , banner etc will be pulled from a data bag called base_config.   The template file for SSHD configuration would be sshd.erb.

File : cookbooks/sshd/recipes/default.rb

sshd_config = data_bag_item('base_config', 'sshd')
template "/etc/ssh/sshd_config" do
    source "sshd.erb"
    mode "0644"
    variables(
    :sshd_port => sshd_config['port'],
    :x11_forwarding => sshd_config['x11_forwarding'],
    :banner => sshd_config['banner'],
    :permit_root => sshd_config['permit_root']
)
end

template "/etc/issue.net" do
    source "issue.net.erb"
end

service "sshd" do
    action [ :restart ]
end

Templates sshd.erb looks like this

File : cookbooks/sshd/template/sshd.erb

Port <%= @sshd_port %>

Protocol 2

#other SSHD config has been omitted for the sake of the blog post

PermitRootLogin <%= @permit_root %>

X11Forwarding <%= @x11_forwarding %>

Banner <%= @banner %>

#other SSHD config has been omitted for the sake of the blog post

In issue.net.erb we are are reading from motd and adding a little blurb after that

File : cookbooks/sshd/template/issue.net.erb

<%= File.read("/etc/motd") %>

*****************************
Use of the Site by unauthorized users is prohibited and 
unauthorized users will be prosecuted to the fullest 
extent of the law.
*****************************

Data bag

Create the data bag

File :data_bags/base)config/config.json 

{  
  "id": "sshd",  
  "port": "2222",  
  "x11_forwarding": "no",  
  "banner":"/etc/issue.net",  
  "permit_root": "no"  
}  

Finishing up

knife data bag create base_config

knife data bag from file base_config data_bags/base_config/config.json 

knife cookbook upload sshd

Then add the recipe to a role or node run list and run the chef-client

Chef Experiments - Create host files

I am in the process of experimenting with Chef and here's one of them

knife create cookbook host_file_update


Then create the recipe

recipes/default.rb

hosts = search(:node, "*:*")
template "/etc/hosts" do
  source "hosts.erb"
  owner "root"
  group "root"
  mode 0644
  variables(
    :hosts => hosts,
    :hostname => node[:hostname],
    :fqdn => node[:fqdn]
  )
end

And then the template file hosts.erb referenced above

templates/default/hosts.erb 

127.0.0.1   localhost

<% @hosts.each do |node| %>
<%= node['ipaddress'] %> <%= node['hostname'] %> <%= node['fqdn'] %>
<% end %>

Pretty useful, if you want to populate this automatically as and when you add servers.  One of the next things to try is see if we can make Chef pick the additional IPs (e.g service net in Rackspace cloud) and create separate entries for it