Nagios Server Installtion

1) Install the prerequisites

yum install httpd php gcc glibc glibc-common gd gd-devel libssl-dev openssl*

2) Create a new nagios user account and give it a password.

/usr/sbin/useradd -m nagios
passwd nagios

3) Create a new nagcmd group for allowing external commands to be submitted through the web interface. 

Add both the nagios user and the apache user to the group.

/usr/sbin/groupadd nagcmd
/usr/sbin/usermod -a -G nagcmd nagios
/usr/sbin/usermod -a -G nagcmd apache

4) Now go to http://www.nagios.org and download the files..

wget http://downloads.sourceforge.net/project/nagios/nagios-3.x/nagios-3.4.3/nagios-3.4.3.tar.gz
wget http://downloads.sourceforge.net/project/nagiosplug/nagiosplug/1.4.16/nagios-plugins-1.4.16.tar.gz

5) Compile and Install Nagios

tar zxvf nagios-3.4.3.tar.gz
cd nagios
./configure --with-command-group=nagcmd
make all
make install; make install-init; make install-config; make install-commandmode;

6) Customize Configuration

Edit the /usr/local/nagios/etc/objects/contacts.cfg config file with your favorite editor and change the email address associated with the nagiosadmin contact definition to the address you’d like to use for receiving alerts.

vim /usr/local/nagios/etc/objects/contacts.cfg
-----------------------------------------------

define contact{
        contact_name                    nagiosadmin             ; Short name of user
        use                             generic-contact         ; Inherit default values from generic-contact template (defined above)
        alias                           Nagios Admin            ; Full name of user

        email                           sankar.k@gmail.com ; <<***** CHANGE THIS TO YOUR EMAIL ADDRESS ******
        }


7) Configure the Web Interface

Install the Nagios web config file in the Apache conf.d directory.

make install-webconf

Create a nagiosadmin account for logging into the Nagios web interface. Remember the password you assign to this account – you’ll need it later.

htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

Restart Apache to make the new settings take effect.

service httpd restart

8) Compile and Install the Nagios Plugins

tar zxvf nagios-plugins-1.4.16.tar.gz
nagios-plugins-1.4.16
./configure --with-nagios-user=nagios --with-nagios-group=nagios
make
make install

9) Start Nagios

chkconfig --add nagios
chkconfig nagios on

Verify the sample Nagios configuration files.

/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg

If there are no errors, start Nagios.

service nagios start

10) Login to the Web Interface using username (nagiosadmin) and password you specified earlier
From your internet browser navigate to the following URL:

http://<your server name or IP>/nagios


Monitor Remote Linux Host using Nagios.
===================================
Follow below steps to monitor a remote Linux host and the various services running on the remote host

 6 steps to install Nagios plugin and NRPE on remote host.

   1) Download Nagios Plugins and NRPE Add-on
   2) Create nagios account
   3) Install Nagios Plugins
   4) Install NRPE
   5) Setup NRPE to run as daemon
   6) Modify the /usr/local/nagios/etc/nrpe.cfg

 4 Configuration steps on the Nagios monitoring server to monitor remote host:

   1) Download NRPE Add-on
   2) Install check_nrpe
   3) Create host and service definition for remote host
   4) Restart the nagios service


Overview

  a) Nagios will execute check_nrpe command on nagios-server and request it to monitor disk usage on remote host using check_disk command.
  b) The check_nrpe on the nagios-server will contact the NRPE daemon on remote host and request it to execute the check_disk on remote host.
  c) The results of the check_disk command will be returned back by NRPE daemon to the check_nrpe on nagios-server.


Following flow summarizes the above explanation:

  Nagios Server (check_nrpe) —–> Remote host (NRPE deamon) —–> check_disk

  Nagios Server (check_nrpe) <—– Remote host (NRPE deamon) <—– check_disk (returns disk space usage)

Steps to install Nagios Plugins and NRPE on the remote host

1. Download Nagios Plugins and NRPE Add-on.

wget http://downloads.sourceforge.net/project/nagiosplug/nagiosplug/1.4.16/nagios-plugins-1.4.16.tar.gz
wget http://downloads.sourceforge.net/project/nagios/nrpe-2.x/nrpe-2.13/nrpe-2.13.tar.gz

2. Create nagios account

useradd nagios
passwd nagios

3. Install nagios-plugin

tar zxvf nagios-plugins-1.4.16.tar.gz
cd nagios-plugins-1.4.16
export LDFLAGS=-ldl
./configure --with-nagios-user=nagios --with-nagios-group=nagios --enable-redhat-pthread-workaround
make
make install
chown nagios.nagios /usr/local/nagios
chown -R nagios.nagios /usr/local/nagios/libexec/

4. Install NRPE

tar zxvf nrpe-2.13.tar.gz
cd nrpe-2.13
./configure
make all
make install-plugin
make install-daemon
make install-daemon-config
make install-xinetd

5. Setup NRPE to run as daemon (i.e as part of xinetd):

 ==> Modify the /etc/xinetd.d/nrpe to add the ip-address of the Nagios monitoring server to the only_from directive. Note that there is a space after the 127.0.0.1 and the nagios monitoring server ip-address (in this example, nagios monitoring server ip-address is: 192.168.80.70)


vim /etc/xinetd.d/nrpe
---------------------

only_from       = 127.0.0.1 192.168.80.70


 ==> Modify the /etc/services and add the following at the end of the file.

vim /etc/services
-----------------

nrpe         5666/tcp             # NRPE

 ==> Start the service

service xinetd restart

 ==> Verify whether NRPE is listening

netstat -at | grep nrpe
       tcp 0      0 *:nrpe *:*                         LISTEN


 ==> Verify to make sure the NRPE is functioning properly

[remotehost]# /usr/local/nagios/libexec/check_nrpe -H localhost
NRPE v2.13

6. Modify the /usr/local/nagios/etc/nrpe.cfg

The nrpe.cfg file located on the remote host contains the commands that are needed to check the services on the remote host. By default the nrpe.cfg comes with few standard check commands as samples. check_users and check_load are shown below as an example.

command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20


In all the check commands, the “-w” stands for “Warning” and “-c” stands for “Critical”. for e.g. in the check_disk command below, if the available disk space gets to 20% of less, nagios will send warning message. If it gets to 10% or less, nagios will send critical message. Change the value of “-c” and “-w” parameter below depending on your environment.

command[check_disk]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1


Note: You can execute any of the commands shown in the nrpe.cfg on the command line on remote host and see the results for yourself. For e.g. When I executed the check_disk command on the command line, it displayed the following:

[remotehost]#/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /dev/hda1
DISK CRITICAL - free space: / 6420 MB (10% inode=98%);| /=55032MB;51792;58266;0;64741


In the above example, since the free disk space on /dev/hda1 is only 10% , it is displaying the CRITICAL message, which will be returned to nagios server.


Configuration steps on the Nagios monitoring server to monitor remote host

1. Download NRPE Add-on

wget http://downloads.sourceforge.net/project/nagios/nrpe-2.x/nrpe-2.13/nrpe-2.13.tar.gz

2. Install check_nrpe on the nagios monitoring server

tar zxvf nrpe-2.13.tar.gz
cd nrpe-2.13
./configure
make all
make install-plugin

 ==> Verify whether nagios monitoring server can talk to the remotehost.

/usr/local/nagios/libexec/check_nrpe -H 192.168.80.129
NRPE v2.13

Note: 192.168.80.70 in the ip-address of the remotehost where the NRPE and nagios plugin was installed as explained in Section II above.


3. Create host and service definition for remotehost

Create a new configuration file /usr/local/nagios/etc/objects/remotehost.cfg to define the host and service definition for this particular remotehost. It is good to take the localhost.cfg and copy it as remotehost.cfg and start modifying it according to your needs.

Ex :

# vim /etc/hosts
   --------------
  192.168.80.129    slave.sankar.com slave

# vim /usr/local/nagios/etc/objects/commands.cfg
  ------------------------------------------------
 ### Add this below lines in end of the file
 # check_nrpe command definition
 define command{
 command_name check_nrpe
 command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -t 30 -c $ARG1$
 }

# cp localhost.cfg slave.cfg

# vim slave.cfg
  --------------
  define host{
        use                     linux-server            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               slave.sankar.com
        alias                   slave.sankar.com
        address                 192.168.80.129
        }

# Define a service to "ping" the local machine

define service{
        use                             local-service         ; Name of service template to use
        host_name                       slave.sankar.com
        service_description             PING
        check_command                   check_ping!100.0,20%!500.0,60%
        }


:%s/localhost/slave.sankar.com/g

# vim /usr/local/nagios/etc/nagios.cfg
  ---------------------------------------

# Definitions for monitoring the local (Linux) host
cfg_file=/usr/local/nagios/etc/objects/localhost.cfg
cfg_file=/usr/local/nagios/etc/objects/slave.cfg


4. Restart the nagios service

Restart the nagios as shown below and login to the nagios web (http://nagios-server/nagios/) to verify the status of the remotehost linux sever that was added to nagios for monitoring.

# service nagios reload

Enjoy............

Compiling nagios-plugins-1.4.16 throws an error

While compiling nagios plugins, you can get an error given below.

==========================
 check_http.c:312:9: error: ‘ssl_version’ undeclared (first use in this function)
....
make[2]: *** [check_http.o] Error 1
make[2]: Leaving directory `/usr/local/src/nagios-plugins-1.4.16/plugins'
make[1]: *** [all-recursive] Error 1
make[1]: Leaving directory `/usr/local/src/nagios-plugins-1.4.16'
make: *** [all] Error 2
========================

Fix :

yum install libssl-dev
yum install openssl*

Command for linux background processing

Perfect for long running batch jobs on a remote server over unreliable connections or if you want to bring your laptop home (instead of keeping that terminal running).

1) screen
   ------------
Log in and run

screen -t title_of_your_choice

Do the same thing again if you want to create another window.
All the following screen commands are preceeded by Ctrl-a (i.e. first press ctrl-a then the shortcut below)

• 0-9 – switch to window by id
• Ctrl-n – next window
• Ctrl-a – previous window
• d – quit screen (leaving it running)
• k – kill window

The next day, log in as usual and attach to the screen session using

screen -x

2) nohup
    ----------

 nohup utility which allows to run command./process or shell script that can continue running in the background after you log out from a shell:

Log in and run

 nohup command-name &

Where,

• command-name : is name of shell script or command name. You can pass argument to command or a shell script.
• & : nohup does not automatically put the command it runs in the background; you must do that explicitly, by ending the command line with an & symbol.

Linux Patch Management with SpaceWalk

Prerequisites


# hostname rhn.sankar.com
# vim /etc/sysconfig/network
==========
HOSTNAME=rhn.sankar.com

install spacewalk-repo package with commands below:

# rpm -Uvh http://yum.spacewalkproject.org/1.8/RHEL/5/x86_64/spacewalk-repo-1.8-4.el5.noarch.rpm
# rpm -Uvh http://yum.pgrpms.org/reporpms/8.4/pgdg-redhat-8.4-2.noarch.rpm

If you want to use the nightly builds, install the spacewalk-repo package based on your operating system (see above) and then enable the nightly repository:
 
# sed -i 's/enabled=0/enabled=1/' /etc/yum.repos.d/spacewalk-nightly.repo
# sed -i 's/enabled=1/enabled=0/' /etc/yum.repos.d/spacewalk.repo

NOTE: Nigthly repo contains developers snapshot and it is not suitable for production environment. Especially beware that you might not be able to upgrade from the nightly installation to the next release, especially with respect to the database schema.

Spacewalk requires a Java Virtual Machine with version 1.6.0 or greater. ​EPEL - Extra Packages for Enterprise Linux contains a version of the openjdk that works with Spacewalk. Other dependencies can get installed from EPEL as well. To get packages from EPEL just install this RPM:

# rpm -Uvh http://dl.fedoraproject.org/pub/epel/5/i386/epel-release-5-4.noarch.rpm

Setup of the PostgreSQL database
You should have PostgreSQL server running somewhere. Let's assume you will run the server on the same machine as Spacewalk itself:
# yum install -y 'postgresql-server > 8.4'

# chkconfig postgresql on

# /etc/init.d/postgresql initdb

# /etc/init.d/postgresql start

Create database, user, and plpgsql language there:

# su - postgres -c 'PGPASSWORD=spacepw; createdb spaceschema ; createlang plpgsql spaceschema ; yes $PGPASSWORD | createuser -P -sDR spaceuser'

Configure the user to use md5 password to connect to that database. Put the lines like following to /var/lib/pgsql/data/pg_hba.conf. Avoid the common pitfall: Make sure you put them *before* those existing lines that are for all..
# vim /var/lib/pgsql/data/pg_hba.conf
===================================
local spaceschema spaceuser md5
host  spaceschema spaceuser 127.0.0.1/8 md5
host  spaceschema spaceuser ::1/128 md5
local spaceschema postgres  ident


Then reload PostgreSQL:

# service postgresql reload

and test the connection:

# PGPASSWORD=spacepw psql -a -U spaceuser spaceschema
# PGPASSWORD=spacepw psql -h localhost -a -U spaceuser spaceschema

Tune up PostgreSQL's performance by running pgtune:
# yum install pgtune
# pgtune --type=web -c 600 -i /var/lib/pgsql/data/postgresql.conf >/tmp/pgtune.conf

 Review the changes by

# diff -u /var/lib/pgsql/data/postgresql.conf /tmp/pgtune.conf
# cp /var/lib/pgsql/data/postgresql.conf /var/lib/pgsql/data/postgresql.conf.bak
# cp /tmp/pgtune.conf /var/lib/pgsql/data/postgresql.conf
# service postgresql restart

or at least increase maximal number of connections to 600:# echo max_connections = 600 >>/var/lib/pgsql/data/postgresql.conf

Install the spacewalk-postgresql and configure it

When installing Spacewalk, you install spacewalk-postgresql which should give you correct backend and dependencies.

# wget http://pkgs.repoforge.org/python-simplejson/python-simplejson-2.0.5-1.el5.rf.i386.rpm

# rpm -ivh python-simplejson-2.0.5-1.el5.rf.i386.rpm

# yum install spacewalk-postgresql

Then, when you run spacewalk-setup, you'll be asked for connection information:

# spacewalk-setup --disconnected --external-db
** Database: Setting up database connection for PostgreSQL backend.
Hostname (leave empty for local)?
Database? spaceschema
Username? spaceuser
Password? spacepw
** Database: Populating database.

Managing Spacewalk
Spacewalk consists of several services. Each of them has its own init.d script to stop/start/restart. If you want manage all spacewalk services at once use

/usr/sbin/spacewalk-service [stop|start|restart].

Once Spacewalk installation is completed we can access spacewalk admin control panel using below URL

http://rhn.sankar.com

This is time to create admin user id and password.

Creating Channels

1. Create a base channel within Spacewalk.

Channels > Manage Software Channels > Create New Channel

2. Fill up all the required fields such as Channel Name, Channel Label, and Channel Summary

3. Select the Parent (its depends upon your channel)

4. Select the channel architecture from the drop down list

5. Select the Checksum type

6. Write a description about your channel

7. Fill the Contact support information, Channel access control and security GPG

8. Now click the “Create Channel” button.

The channel with the specified name has been created.

Adding packages to repository
There are two ways to add packages to the spacewalk server. We can either add using spacewalk-repo-sync or rhnpush command.

Spacewalk-repo-sync

The  spacewalk-repo-sync tool is used to sync packages from external or local yum repositories. All the packages within the specified repository will be added to the channel.Any url supported by yum is supported by this utility, including mirror lists.  If the url is not supplied, the tool will look to see what  repositories are associated with the specified channel and use those.

Example:

spacewalk-repo-sync --channel=repo1 --url=http://example.com/yum-repo/
spacewalk-repo-sync --channel=repo2 --url=file:///var/share/localrepo/
spacewalk-repo-sync --channel=repom --url=http://example.com/mirrorlist.xml/

You can also use WebGUI and this is the easiest way to create repositories






    Goto Channels -> Manage Software Channels -> Manage Repositories -> create new repository

After creating the repository, you need to link it to one or more Software Channels.

    Goto: Channels -> Manage Software Channels -> Choose the channel to be linked -> Repositories -> Select the repositories to be linked to the channel -> Update Repositories.

Now you can sync the repository by clicking on the sync tab.

Click on sync now or schedule a sync.

Alternatively you can start a sync of a yum repository defined in the web ui by command line:

spacewalk-repo-sync --channel CHANNEL_LABEL


If, when doing a spacewalk-repo-sync, you get a "yum.Errors.NoMoreMirrorsRepoError?" error then you need to install python-hashlib.

The logs are stored in /var/log/rhn/reposync/



RHNpush :

The  RHN  Satellite Package Pusher (rhnpush) pushes RPMs into locally managed channels on an RHN Satellite Server. Rhnpush has three configuration files called /etc/sysconfig/rhn/rhnpushrc, ~/.rhnpushrc, and ./.rhnpushrc.

/etc/sysconfig/rhn/rhnpushrc is the system-wide default settings for rhnpush.
~/.rhnpushrc is the user-specific settings that override the system-wide settings.
./.rhnpushrc controls the directory specific settings that override the user-specific and system-specific settings.

/etc/sysconfig/rhn/rhnpushrc must be present for rhnpush to function correctly. If it is missing, rhnpush will attempt to use a series of  default  settings  stored  internally as a replacement. ~/.rhnpushrc and ./.rhnpushrc are not required to be present, but will be used if they are present. They are not created automatically by rhnpush.

Rhnpush uses a cache, stored at ~/.rhnpushcache, to temporarily hold the username and password for a user. If the cache is missing, it will be created by rhnpush.
If the cache is present and not too old, the usename-password combo will be used as a convenience for the user. The amount of time a cache lasts is configurable in any of the three configuration files. If your username/password combination gets messed up you have two options. One, you can wait until the cache expires, which takes  minutes  by  default. Two, you can use the –new_cache option to force rhnpush to let you reenter your username/password.

Using  the  –stdin  and –dir options at the same time works as follows: rhnpush will let you type in rpm names, each rpm name on a separate line. When       you have finished entering in rpm names, hit Ctrl-D. Rhnpush will then grab the files from directory you specified with –dir, put them in a  list  with       the rpms you listed through standard input, and send them to the channel that was listed on the command-line or in the configuration files.

Note : Make sure /var/satellite exists on the Spacewalk server and has owner:group apache before pushing.

[root@sathishhost ~]# chgrp apache /var/satellite/ -R

[root@sathishhost ~]# ls -l /var | grep satellite

drwxr-xr-x.  3 apache apache 4096 Mar 20 10:06 satellite

Example

rhnpush --server localhost -u <username> -p <password> --channel <channel-name> /usr/local/src/additional/*.rpm

rhnpush --server localhost -u sathish -p redhat --channel spacewalk-nightly-rhel-6-x86_64 /usr/local/src/additional/*.rpm

rhnpush -v --channel=<channel-name> --server=http://localhost/APP --dir=<package-dir>

rhnpush -v --channel=spacewalk-nightly-rhel-6-x86_64 --server=http://localhost/APP --dir=/usr/local/src/additional

Creating activation key

Activation keys are used to register a system to spacewalk server. System registered with an activation key will inherit the characteristics defined by that key.

1. To create an activation key

Systems > Activation keys > Create new key

2. Enter the description of the activation key

3. If you have a specific key value, type it in the key textbox else leave it as blank. Spacewalk will generate a key after clicking create key button.

4. Enter a numeric value for the limited usage of the key. In case if you want to use the key unlimited times leave the usage textbox as blank.

5. Select the base channel from the drop down list box or choose “Spacewalk Default” to allow systems to register to the default Red Hat provided channel that corresponds to their installed version of Red Hat Enterprise Linux.

6. Enable the universal default check box and click “Create Activation Key”



Registering Clients
Install the client-tools in the client

For RHEL 5 

# rpm -Uvh http://spacewalk.redhat.com/yum/1.7/RHEL/5/i386/spacewalk-client-repo-1.7-5.el5.noarch.rpm

Now install the client packages

# yum install rhn-client-tools rhn-check rhn-setup rhnsd m2crypto yum-rhn-plugin

Register your CentOS or Red Hat Enterprise Linux system to Spacewalk using the activation key you created earlier

# rhnreg_ks --serverUrl=http://YourSpacewalk.example.org/XMLRPC --activationkey=<key-with-rhel-custom-channel>

rhnreg_ks is used for registration of clients to Spacewalk. If you need to re-register a client to your Spacewalk server or change registration from one environment or server to another Spacewalk server then use the “–force” flag with rhnreg_ks, otherwise there is no need to use “–force”.