Basic HAProxy Installation and configurations

What is HAProxy : 

HAProxy is a very fast and reliable solution for high availability, load balancing. It supports TCP and HTTP-based applications. Now a days all website need 99.9% of up-time required for their site. It's not possible with single server setup. So we need high availability environment which can easily manage single server failure. 



Installation of HAProxy

Installation of HAProxy is very easy follow the simple step to install

yum install haproxy -y

that's all only you need internet access. After that we have configure not configure just add some lines to the haproxy configuration file and need to start the service.

Configurations of HAProxy

Main configuration file is in /etc/haproxy/haproxy.cfg, in that we need to add some details that we need to access. Follow the steps below.

add the below mentioned line in EOD 

vi /etc/haproxy/haproxy.cfg

add below lines

######### Configure frondend Server ################

frontend webapp
        bind *:80

        default_backend webserver

######### Configure Backend Server #################

backend webserver
        balance roundrobin
        server  web1 10.0.0.11:80 check
        server  web1 10.0.0.12:80 check
        server  web1 10.0.0.13:80 check
        server  web1 10.0.0.14:80 check

Save & Exit

start the haproxy service and enable it for start when booting

In RHEL 6

service haproxy start
chkconfig haproxy on

In RHEL 7

systemctl start haproxy
systemctl enable haproxy

Note :

if your http service is running in 80 change it to other port. It will cause starting issue in haproxy.


Now try in web browser it will work.

Nagios Installations and configurations

Nagios Installations and configurations

What is Nagios :

Nagios is known as Nagios Core and it's a open source computer software application that monitor systems, network and infrastructure. It's also have alerting system for servers, switch, applications and services.

Server Monitoring :

Nagios is the one of the best monitoring software for servers. It made easy to monitor with flexibility of both agent-based and agent-less monitoring. It has over 5000 different kind of addons available to monitor servers.

Network Monitoring :

Nagios monitors the network for the problems caused by overloaded data links or network connections, same as switches, routers and more. It's easy to monitor availability up-time and response time of every node on the network.

Application Monitoring :

Implementing effective application monitoring with nagios allows our organization to quickly detect applications, services, and process problems. It alert when app got problems to the user that control nagios. 

Installation :

Before installation we need to have root access for that server will run Nagios ideally. 

Installation of Nagios 4 :

Step 1 

Install dependencies of below mentioned

yum install gcc glibc glibc-common gd gd-devel make net-snmp open-ssl xineted unzip -y

Step 2

Create nagios user and group for running Nagios.

useradd nagios

groupadd nagcmd

usermod -a -G nagcmd nagios

Step 3

Download the Ngaios stable release and place it where required that. I always use /opt dir. 

We can download nagios core and other open source in followed in here.

goto /opt

cd /opt

curl -L -o https://assets.nagios.com/downloads/nagioscore/releases/nagios-4.1.1.tar.gz

step 4

Extract the package in same location

tar -xvzf nagios-4.1.1.tar.gz

change the name as nagios-4.1.1 to nagios

mv nagios-4.1.1 nagios

Step 5

now let install 

cd nagios

./configure --with-command-group=nagcmd

after that

make all

now we need to install init script with following commands

make install
make install-commandmode
make install-init
make install-config
make install-webconf

We also need the apache to access our nagios so we need to add the apache user to nagcmd group.

usermod -G nagcmd apache

now Nagios installed. We need to install plugins for the nagios.

Installation of Nagios Plugin :

Step 1

got /opt directory.

cd /opt

curl -L -o http://nagios-plugins.org/download/nagios-plugins-2.1.1.tar.gz

Step 2

Extract it on same location.

tar -xvzf nagios-plugins-2.1.1.tar.gz

cd nagios-plugin-2.1.1

Step 3

Configuring nagios plugin with following command

./configure --with-nagios-user=nagios --with-nagios-group=nagcmd --with-openssl

Now compile the nagios plugin with following command

make

now install with following command

make install

now nagios plugin is completed

Installation of NRPE :

We can find the latest NRPE file following link in here.

Step 1

Download the NRPE and extract them.

cd /opt

curl -L -o http://downloads.sourceforge.net/project/nagios/nrpe-2.x/nrpe-2.15/nrpe-2.15.tar.gz

tar -xvzf nrpe-2.15.tar.gz

Step 2

Move to the directory and configure nrpe with following command

./configure --enable-command-args --with-nagios-user=nagios --with-nagios-group=nagios --with-ssl=/usr/bin/openssl --with-ssl-lib=/usr/lib/x86_64-linux-gnu

now we ready to install the NRP and init script with following command

make all
make install
make install-xinetd
make install-daemon-config

Step 3

Now open the startup xinetd script in your favorite editor

vi /etc/xinetd.d/nrpe

modify the only from line by adding your private IP address.

only_from = 127.0.0.1 192.168.0.0

save and exit.

Now the Nagios server will be communicate with NRPE.

Now restart the xinetd service to start the NRPE service.

Service xinetd restart.

Step 4

Configure the new command to our nagios configuration. For this we need to add some lines in commands.cfg file. Followed

vi /usr/local/nagios/etc/objects/commands.cfg

in that add the details below to the end of the file

define command{
          command_name check_nrpe
          command_line $USER1$/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

save & exit

This will allow us to use the check_nrpe command in nagios service definitions.

Configuration of Nagios :

Now we are going to make some changes in nagios default configuration.

open the nagios.cfg file and remove the # symbol from below mentioned line

vi /usr/local/nagios/etc/nagios.cfg

find the line of cfg_dir=/usr/local/nagios/etc/servers

#cfg_dir=/usr/local/nagios/etc/server

remove # 

cfg_dir=/usr/local/nagios/etc/server

save & exit

now create the directory that will store the nagios configuration file that each you monitor.

mkdir /usr/local/nagios/etc/servers

Configuration of Notification :

In this we can add our mail id for the alert notifications. All you need to configure the details in contacts.cfg

vi /usr/local/nagios/etc/objects/contacts.cfg

In that find the email directive and replace your mail id. 

email            yourmailid       ;<<***** CHANGE THIS TO YOUR EMAIL ADDRESS *****

save & exit

configurations of Apache :

Her you can set the password for nagiosadmin web user admin password. For this we use htpasswd in apache.

htpasswd -c /usr/local/nagios/etc/htpasswd.users nagiosadmin

New password:

Re-type password: 

Adding password for nagiosadmin

if you don't want to use admin username as nagiosadmin the you have to edit /usr/local/nagios/etc/cgi.cfg file and change the name as you wished.

Now restart the nagios and httpd service 

service nagios restart
service httpd restart

make sure you enable the nagios on startup

chkconfig nagios on

Configurations of Restricted Access by IP :

If you want to access the nagios web interface only by your system then you have to edit the nagios apache configuration file.

vi /etc/httpd/conf.d/nagios.conf

and find the two line that mentioned below and add # in-front of it.

Order allow,deny
Allow from all

then remove the # from below mentioned lines and add your ip in Allow from line.

# Order deny,allow
# Deny from all
# Allow from 127.0.0.1 <- you can add your ip here 

As these line appear multiple the you have do the same in other.

save & exit

now restart nagios and httpd

service nagios restart
service httpd restart

now we have completed the configuration now open the nagios in web browser.

http://nagios_server_ip/nagios

it will ask username password

After that the page will appear like below image

That's all for nagios installation. Next I will come up with NRPE client configuration and adding host in nagios.

Elasticsearch password authentication

Elasticsearch password authentication & SSL Auth

What is elasticsearch:

Elasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in java and released as opensource under the terms of the Apache license.

Configuring user authentication in elasticsearch

It's very simple to configure elastic search basic authentication. All you need to download the shield package and license in elastic home website and installed it. 

Below I have mentioned how to install shield license and plugins.

Download the package and placed it in where you want to I placed it in /opt location.

then go to the elasticsearch location I have portable elasticsearch with me in /opt. So I move to that directory and install shield plugin.

cd /opt/elasticsearch

ls

It will show us like this. Then

bin/plugins install file:///opt/license.xx.xx.zip

bin/plugins install file:///opt/shield.xx.xx.zip

now we have completed the installation shield plugin. now we all have to do simple step for authentication.

Role Based Access Control

We need to move to shield configuration location to configuring the users and their roles with which we'll be able to execute various of APIs on Elasticsearch

Shield directory location will be /opt/elasticsearch/config/shield.

Go to that directory and check the permissions that which user have which permissions. 

cd /opt/elasticsearch/config.shield

vi roles.yml

Admin role:

The admin role enables full access to the cluster and all it's indices.

Power_user role:

The power_user role enables monitoring only access on the cluster and full access on all it's indices.

User role:

The user role has no cluster wide permissions and only has read access to the data on all it's indices.

For the basic authentication we don't need to change anything from roles.yml file that comes from shield. It's already defines with required default roles listed in above.

User creation per permissions

We already have know the user permissions that have in shield. Now we need to create the user and password and required permissions. 

Now i'm going to create one user with admin role and another with user role. 

For that shield have internal user like root shield have esusers it comes with shield.

cd /opt/elasticsearch

bin/shield/esusers useradd rain -p R@!N@1234 -r admin

in above command 

-p - refereed as password

-r - refereed as roles

admin - refereed as admin role

bin/shield/esusers useradd pearl -p PE@R!@1234 -r user 

now we have completed the user creation and password authentication. 

Verify the shield installation

Once elasticsearch started we can use curl command to check the shield installed correctly or not.

curl --user rain:R@!N@1234 'localhost:9200/_shield'

{
  "status" : "enabled",
  "name" : "rain",
  "cluster_name" : "Test.elasticseach.com",
  "version" : {
    "number" : "2.3.3",
    "build_hash" : "8a3e9756c34e3110c49fd3a81a5ac90a83dc2eaa",
    "build_timestamp" : "2016-05-17T16:35:12Z",
    "build_snapshot" : false
  },
  "tagline" : "You Know, for Security"

}

If result appear like above then your configuration completed successfully.

SSL Auth:

We need to add the ssl file in elastic source place and mention that in the elasticsearch.yml. Follow my lead.

goto elastcisearch directory

cd /opt/elasticsearch

place the ssl file or copy the ssl file

make sure the permission for this in ssl appropriate user

chown test:test test.jks

now wee need to add the location this jks file in elasticsearch.yml.

vi conf/elasticsearc.yml

goto the last line and add the following line

shield.ssl.keystore.path:          /opt/elasticsearch/test.jks
shield.ssl.keystore.password:      test123
shield.transport.ssl: true
shield.http.ssl: true 

now restart the elasticsearch and check it will work perfectly.

hung_task_timeout_secs and blocked for more than 120 seconds problem

Linux Kernel panic issue: How to fix hung_task_timeout_secs and blocked for more than 120 seconds problem

A panic may occur as a result of a hardware failure or a software bug in the operating system. In many cases, the operating system is capable of continued operation after an error has occurred. However, the system is in an unstable state and rather than risking security breaches and data corruption, the operating system stops to prevent further damage and facilitate diagnosis of the error and, in usual cases, restart. After recompiling a kernel binary image from source code, a kernel panic during booting the resulting kernel is a common problem if the kernel was not correctly configured, compiled or installed.  Add-on hardware or malfunctioning RAM could also be sources of fatal kernel errors during start up, due to incompatibility with the OS or a missing device driver.  A kernel may also go into panic() if it is unable to locate a root file system. During the final stages of kernel userspace initialization, a panic is typically triggered if the spawning of init fails, as the system would then be unusable.

Background

When I face this issue i have checked the log in /var/log/messages. In that i have got below mentioned log

INFO: task jbd2/vda3-8:250 blocked for more than 120 seconds.
 Not tainted 2.6.32-431.11.2.el6.x86_64 #1
 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Step by step troubleshooting data and logs


Following command will show the server memory usage



sar -r



we can check the memory status that server consumes.



then we have to check the cpu usage with following command



sar -u



Solution for hung_task_timeout_secs

Explanation

By default Linux uses up to 40% of the available memory for file system 
caching. After this mark has been reached the file system flushes all 
outstanding data to disk causing all following IOs going synchronous. 
For flushing out this data to disk this there is a time limit of 120 
seconds by default. In the case here the IO subsystem is not fast enough
 to flush the data withing 120 seconds. As IO subsystem responds slowly 
and more requests are served, System Memory gets filled up resulting in 
the above error, thus serving HTTP requests.

Testing

I tested this theory with the following:

Change vm.dirty_ratio and vm.dirty_backgroud_ratio

sysctl -w vm.dirty_ratio=10

sysctl -w vm.dirty_background_ratio=5

commit the change

sysctl -p

Make it permanent

When the server seemed more stable and no Kernel/Swap/Memory Panic for a week, 
I edited /etc/sysctl.conf file to make these permanent after reboot.


vi /etc/sysctl.conf


add the 2 lines at the bottom


vm.dirty_background_ratio = 5
vm.dirty_ratio = 10


save and exit


then reboot the server.

Ldap Installation and configuration in rhel 6

Ldap Installation & Configurations

Step 1 - Ldap installation:

Run the following command to install ldap

yum install *openldap* -y

or

yum install openldap openldap-clients openldap-servers -y


Step 2 - Assign ldap admin password

Run the following command to assign password for admin ldap

slappasswd

New password : password
re-enter new password : password

{SSHA}50UEVw19zeh7LT53hQH69znzj4OuSrHd


Step 3 - Add root user password hash in configuration file

Root user only have access to add user, groups, OU etc... Do the following in the olcDatabase={2}bdb.ldif file

cd /etc/openldap//slapd.d/cn\=config

vi olcDatabase\=\{2\}bdb.ldif

if olcRootPW doesn't exist create it add the password like below

olcRootPW: {SSHA}50UEVw19zeh7LT53hQH69znzj4OuSrHd

after that we need to made some changes in that file that mentioned below

change distinguished name (DN) of the olcSuffix to something appropriate.

olcSuffix: dc=test,dc=com
olcRootDN: cn:ldap,dc=test,dc=com



Step 4 - Modify RootDN in olcDatabase={2}


follow the steps below to add root dn

vi olcDatabase\=\{1\}monitor.ldif

modify the olcAccess line so the dn.base matches with olcRootDN from olcDtabase={2}

olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" read by dn.base="cn=ldap,dc=test,dc=com" read by * none

Step 5 - Hide the HASH password from other users

edit the oclDatabase\=\{2\}bdb.ldif file

vi olcDatabase\=\{2\}bdb.ldif

and add the following details to last line

olcAccess: {0}to attrs=userPassword by self write by dn.base="cn=ldap,dc=test,dc=com" write by anonymous auth by * none 

olcAccess: {1}to * by dn.base="cn=ldap,dc=test,dc=com" write by self write by * read

These line only allow the users to read or write users own passwords.

Step 5 - Start the ldap service

service slapd start

May be you will get error like below

581cb279 ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config/olcDatabase={1}monitor.ldif"

581cb279 ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif"

Just ignore for this time.

make sure to turn on it automatically at restart

chkconfig slapd on

Step 6 Creating DN entry in ldap tree by manually

An LDAP directory is analogous to a tree. Nodes in this tree are
called LDAP "entries" and may represent users, groups, organizational
units, domain controllers, or other objects. The attributes in each
entry are determined by the LDAP schema. In this tutorial we will build entries based on the InetOrgPerson schema (which ships with OpenLDAP by default).In order to build our LDAP tree we must first create the root entry. 

Root entries are usually a special type of entrycalled a domain controller (DC).  

Because we are assuming that the organization is called something, and that the domain is "test.com," we will create a domain controller LDAP entry called dc=test,dc=com. Again, you will need to replace "test" with your organization's domain name.  

Also note that dc=test,dc=com is what is called an LDAP distinguished name (DN). An LDAP distinguished name uniquely identifies an LDAP entry.

create a file test.ldif in /tmp and add the below mentioned line.

cd /tmp

vi test.ldif

dn: dc=test,dc=com
objectClass: dcObject
objectClass: organization
dc: test
o : test



Step 7 Add the content to LDAP

Run the following commands to add that we create ldif file to LDAP tree
ldapadd -f test.ldif -D cn=ldap,dc=test,dc=com -w password
now the content will added to the ldap. Make sure that it's entered correctly or not with following command
ldapsearch -x -LLL -b dc-test,dc=com

it will show the result like below

dn: dc=test,dc=com
objectClass: dcObject
objectClass: organization
dc: test
o : test



Step 8 Allow or disable the firewall

by default rhel have enable the iptables you can allow the ldap port to access or you can just disable to move on

for disabling run the folowing command.

service iptables stop
chkconfig iptables off

if you want to then do the following.

iptables -A INPUT -p tcp -m tcp --dport 389 -j ACCEPT

service iptables save



Step 9 - Adding OU in ldap tree

add temprory file in tmp and you can delete once it added to ldap tree 

cd /tmp
vi users/ldif

and add the following line

dn: ou=Users,dc=test,dc=com
objectClass: organizationalUnit
ou: Users

Now add the content to ldap
ldapadd -f users.ldif -D cn=ldap,dc=test,dc=com -w password



Step 10 - Add users to ldap

Same as step 9

cd /tmp
vi testuser.ldif

add the following line

dn: cn=test user,ou=Users,dc=test,dc=com
cn: test
sn: user
objectClass: inetOrgPerson
userPassword: test123
uid: testuser

now add the content to ldap tree

ldapadd -f testuser.ldif -D cn=ldap,dc=tes,dc=com -w password



Step 11 - Add group to ldap

same as step 9 and 10

cd /tmp
vi testgroup.ldif

now add the following lines

dn: cn=testgroup,ou=Users,dc=test,dc=com
cn: testgroup
objectClass: groupOfNames
member: cn=test user,ou=Users,dc=test,dc=com

now add to ldap

ldapadd -f testgroup.ldif -D cn=ldap,dc=tes,dc=com -w password



Step 12 - Add user to group in ldap

same as above 

cd /tmp
vi addusertogroup.ldif

add the following lines

dn: cn=testgroup,ou=Users,dc=test,dc=com
changetype: modify
add: member
member: cn=test user,ou=Users,dc=test,dc=com

now add to ldap

ldapadd -f addusertogroup.ldif -D cn=ldap,dc=tes,dc=com -w password

that's all to configure basic openldap ....