Elasticsearch password authentication

Elasticsearch password authentication & SSL Auth

What is elasticsearch:

Elasticsearch is a search engine based on Lucene. It provides a distributed, multitenant-capable full-text search engine with an HTTP web interface and schema-free JSON documents. Elasticsearch is developed in java and released as opensource under the terms of the Apache license.

Configuring user authentication in elasticsearch

It's very simple to configure elastic search basic authentication. All you need to download the shield package and license in elastic home website and installed it. 

Below I have mentioned how to install shield license and plugins.

Download the package and placed it in where you want to I placed it in /opt location.

then go to the elasticsearch location I have portable elasticsearch with me in /opt. So I move to that directory and install shield plugin.

cd /opt/elasticsearch

ls

It will show us like this. Then

bin/plugins install file:///opt/license.xx.xx.zip

bin/plugins install file:///opt/shield.xx.xx.zip

now we have completed the installation shield plugin. now we all have to do simple step for authentication.

Role Based Access Control

We need to move to shield configuration location to configuring the users and their roles with which we'll be able to execute various of APIs on Elasticsearch

Shield directory location will be /opt/elasticsearch/config/shield.

Go to that directory and check the permissions that which user have which permissions. 

cd /opt/elasticsearch/config.shield

vi roles.yml

Admin role:

The admin role enables full access to the cluster and all it's indices.

Power_user role:

The power_user role enables monitoring only access on the cluster and full access on all it's indices.

User role:

The user role has no cluster wide permissions and only has read access to the data on all it's indices.

For the basic authentication we don't need to change anything from roles.yml file that comes from shield. It's already defines with required default roles listed in above.

User creation per permissions

We already have know the user permissions that have in shield. Now we need to create the user and password and required permissions. 

Now i'm going to create one user with admin role and another with user role. 

For that shield have internal user like root shield have esusers it comes with shield.

cd /opt/elasticsearch

bin/shield/esusers useradd rain -p R@!N@1234 -r admin

in above command 

-p - refereed as password

-r - refereed as roles

admin - refereed as admin role

bin/shield/esusers useradd pearl -p PE@R!@1234 -r user 

now we have completed the user creation and password authentication. 

Verify the shield installation

Once elasticsearch started we can use curl command to check the shield installed correctly or not.

curl --user rain:R@!N@1234 'localhost:9200/_shield'

{
  "status" : "enabled",
  "name" : "rain",
  "cluster_name" : "Test.elasticseach.com",
  "version" : {
    "number" : "2.3.3",
    "build_hash" : "8a3e9756c34e3110c49fd3a81a5ac90a83dc2eaa",
    "build_timestamp" : "2016-05-17T16:35:12Z",
    "build_snapshot" : false
  },
  "tagline" : "You Know, for Security"

}

If result appear like above then your configuration completed successfully.

SSL Auth:

We need to add the ssl file in elastic source place and mention that in the elasticsearch.yml. Follow my lead.

goto elastcisearch directory

cd /opt/elasticsearch

place the ssl file or copy the ssl file

make sure the permission for this in ssl appropriate user

chown test:test test.jks

now wee need to add the location this jks file in elasticsearch.yml.

vi conf/elasticsearc.yml

goto the last line and add the following line

shield.ssl.keystore.path:          /opt/elasticsearch/test.jks
shield.ssl.keystore.password:      test123
shield.transport.ssl: true
shield.http.ssl: true 

now restart the elasticsearch and check it will work perfectly.

hung_task_timeout_secs and blocked for more than 120 seconds problem

Linux Kernel panic issue: How to fix hung_task_timeout_secs and blocked for more than 120 seconds problem

A panic may occur as a result of a hardware failure or a software bug in the operating system. In many cases, the operating system is capable of continued operation after an error has occurred. However, the system is in an unstable state and rather than risking security breaches and data corruption, the operating system stops to prevent further damage and facilitate diagnosis of the error and, in usual cases, restart. After recompiling a kernel binary image from source code, a kernel panic during booting the resulting kernel is a common problem if the kernel was not correctly configured, compiled or installed.  Add-on hardware or malfunctioning RAM could also be sources of fatal kernel errors during start up, due to incompatibility with the OS or a missing device driver.  A kernel may also go into panic() if it is unable to locate a root file system. During the final stages of kernel userspace initialization, a panic is typically triggered if the spawning of init fails, as the system would then be unusable.

Background

When I face this issue i have checked the log in /var/log/messages. In that i have got below mentioned log

INFO: task jbd2/vda3-8:250 blocked for more than 120 seconds.
 Not tainted 2.6.32-431.11.2.el6.x86_64 #1
 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.

Step by step troubleshooting data and logs


Following command will show the server memory usage



sar -r



we can check the memory status that server consumes.



then we have to check the cpu usage with following command



sar -u



Solution for hung_task_timeout_secs

Explanation

By default Linux uses up to 40% of the available memory for file system 
caching. After this mark has been reached the file system flushes all 
outstanding data to disk causing all following IOs going synchronous. 
For flushing out this data to disk this there is a time limit of 120 
seconds by default. In the case here the IO subsystem is not fast enough
 to flush the data withing 120 seconds. As IO subsystem responds slowly 
and more requests are served, System Memory gets filled up resulting in 
the above error, thus serving HTTP requests.

Testing

I tested this theory with the following:

Change vm.dirty_ratio and vm.dirty_backgroud_ratio

sysctl -w vm.dirty_ratio=10

sysctl -w vm.dirty_background_ratio=5

commit the change

sysctl -p

Make it permanent

When the server seemed more stable and no Kernel/Swap/Memory Panic for a week, 
I edited /etc/sysctl.conf file to make these permanent after reboot.


vi /etc/sysctl.conf


add the 2 lines at the bottom


vm.dirty_background_ratio = 5
vm.dirty_ratio = 10


save and exit


then reboot the server.

Ldap Installation and configuration in rhel 6

Ldap Installation & Configurations

Step 1 - Ldap installation:

Run the following command to install ldap

yum install *openldap* -y

or

yum install openldap openldap-clients openldap-servers -y


Step 2 - Assign ldap admin password

Run the following command to assign password for admin ldap

slappasswd

New password : password
re-enter new password : password

{SSHA}50UEVw19zeh7LT53hQH69znzj4OuSrHd


Step 3 - Add root user password hash in configuration file

Root user only have access to add user, groups, OU etc... Do the following in the olcDatabase={2}bdb.ldif file

cd /etc/openldap//slapd.d/cn\=config

vi olcDatabase\=\{2\}bdb.ldif

if olcRootPW doesn't exist create it add the password like below

olcRootPW: {SSHA}50UEVw19zeh7LT53hQH69znzj4OuSrHd

after that we need to made some changes in that file that mentioned below

change distinguished name (DN) of the olcSuffix to something appropriate.

olcSuffix: dc=test,dc=com
olcRootDN: cn:ldap,dc=test,dc=com



Step 4 - Modify RootDN in olcDatabase={2}


follow the steps below to add root dn

vi olcDatabase\=\{1\}monitor.ldif

modify the olcAccess line so the dn.base matches with olcRootDN from olcDtabase={2}

olcAccess: {0}to * by dn.base="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" read by dn.base="cn=ldap,dc=test,dc=com" read by * none

Step 5 - Hide the HASH password from other users

edit the oclDatabase\=\{2\}bdb.ldif file

vi olcDatabase\=\{2\}bdb.ldif

and add the following details to last line

olcAccess: {0}to attrs=userPassword by self write by dn.base="cn=ldap,dc=test,dc=com" write by anonymous auth by * none 

olcAccess: {1}to * by dn.base="cn=ldap,dc=test,dc=com" write by self write by * read

These line only allow the users to read or write users own passwords.

Step 5 - Start the ldap service

service slapd start

May be you will get error like below

581cb279 ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config/olcDatabase={1}monitor.ldif"

581cb279 ldif_read_file: checksum error on "/etc/openldap/slapd.d/cn=config/olcDatabase={2}bdb.ldif"

Just ignore for this time.

make sure to turn on it automatically at restart

chkconfig slapd on

Step 6 Creating DN entry in ldap tree by manually

An LDAP directory is analogous to a tree. Nodes in this tree are
called LDAP "entries" and may represent users, groups, organizational
units, domain controllers, or other objects. The attributes in each
entry are determined by the LDAP schema. In this tutorial we will build entries based on the InetOrgPerson schema (which ships with OpenLDAP by default).In order to build our LDAP tree we must first create the root entry. 

Root entries are usually a special type of entrycalled a domain controller (DC).  

Because we are assuming that the organization is called something, and that the domain is "test.com," we will create a domain controller LDAP entry called dc=test,dc=com. Again, you will need to replace "test" with your organization's domain name.  

Also note that dc=test,dc=com is what is called an LDAP distinguished name (DN). An LDAP distinguished name uniquely identifies an LDAP entry.

create a file test.ldif in /tmp and add the below mentioned line.

cd /tmp

vi test.ldif

dn: dc=test,dc=com
objectClass: dcObject
objectClass: organization
dc: test
o : test



Step 7 Add the content to LDAP

Run the following commands to add that we create ldif file to LDAP tree
ldapadd -f test.ldif -D cn=ldap,dc=test,dc=com -w password
now the content will added to the ldap. Make sure that it's entered correctly or not with following command
ldapsearch -x -LLL -b dc-test,dc=com

it will show the result like below

dn: dc=test,dc=com
objectClass: dcObject
objectClass: organization
dc: test
o : test



Step 8 Allow or disable the firewall

by default rhel have enable the iptables you can allow the ldap port to access or you can just disable to move on

for disabling run the folowing command.

service iptables stop
chkconfig iptables off

if you want to then do the following.

iptables -A INPUT -p tcp -m tcp --dport 389 -j ACCEPT

service iptables save



Step 9 - Adding OU in ldap tree

add temprory file in tmp and you can delete once it added to ldap tree 

cd /tmp
vi users/ldif

and add the following line

dn: ou=Users,dc=test,dc=com
objectClass: organizationalUnit
ou: Users

Now add the content to ldap
ldapadd -f users.ldif -D cn=ldap,dc=test,dc=com -w password



Step 10 - Add users to ldap

Same as step 9

cd /tmp
vi testuser.ldif

add the following line

dn: cn=test user,ou=Users,dc=test,dc=com
cn: test
sn: user
objectClass: inetOrgPerson
userPassword: test123
uid: testuser

now add the content to ldap tree

ldapadd -f testuser.ldif -D cn=ldap,dc=tes,dc=com -w password



Step 11 - Add group to ldap

same as step 9 and 10

cd /tmp
vi testgroup.ldif

now add the following lines

dn: cn=testgroup,ou=Users,dc=test,dc=com
cn: testgroup
objectClass: groupOfNames
member: cn=test user,ou=Users,dc=test,dc=com

now add to ldap

ldapadd -f testgroup.ldif -D cn=ldap,dc=tes,dc=com -w password



Step 12 - Add user to group in ldap

same as above 

cd /tmp
vi addusertogroup.ldif

add the following lines

dn: cn=testgroup,ou=Users,dc=test,dc=com
changetype: modify
add: member
member: cn=test user,ou=Users,dc=test,dc=com

now add to ldap

ldapadd -f addusertogroup.ldif -D cn=ldap,dc=tes,dc=com -w password

that's all to configure basic openldap ....

Hadoop Multinode Installation in RHEL 6

Step 1

Assign static ip to all the required servers.

Ex.

192.168.0.90 - master
192.168.0.91 - slave1
192.168.0.92 - slave 2

Step 2

Make sure all machines have host entries of all servers.

vi /etc/hosts

192.168.0.90  master
192.168.0.91  slave 1
192.168.0.92  slave 2

Step 3

Create hadoop user in all the servers

useradd hadoop

passwd hadoop

Step 4

Setup ssh passwordless authentication for all the server for inter communication.

su hadoop

ssh-keygen -t rsa

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

chmod 0600 ~/.ssh/authorized_keys

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@slave1

ssh-copy-id -i ~/.ssh/id_rsa.pub hadoop@slave2

Same need to do all the slave servers.

Step 5

Disable the ipv6 in master and slave servers hadoop will not support. insert the below details in /etc/sysctl.conf in last line

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

Step 6

Set environment variables uses by hadoop by editing /etc/profile file 

and append the following values at the end of the file by excuting below commands

echo "" >> /etc/profile

echo "### HADOOP Variables ###" >> /etc/profile

echo "export HADOOP_HOME=/opt/hadoop" >> /etc/profile

echo "export HADOOP_INSTALL=\$HADOOP_HOME" >> /etc/profile

echo "export HADOOP_MAPRED_HOME=\$HADOOP_HOME" >> /etc/profile

echo "export HADOOP_COMMON_HOME=\$HADOOP_HOME" >> /etc/profile

echo "export HADOOP_HDFS_HOME=\$HADOOP_HOME" >> /etc/profile

echo "export YARN_HOME=\$HADOOP_HOME" >> /etc/profile

echo "export HADOOP_COMMON_LIB_NATIVE_DIR=\$HADOOP_HOME/lib/native" >> /etc/profile

echo "export PATH=\$PATH:\$HADOOP_HOME/sbin:\$HADOOP_HOME/bin" >> /etc/profile

Reload Configuration using below command.

source /etc/profile

Step 7

Create Hadoop data directories in all nodes (root can only have the access to create)

mkdir -p /data/hadoop-data/nn 

mkdir -p /data/hadoop-data/snn

mkdir -p /data/hadoop-data/dn

mkdir -p /data/hadoop-data/mapred/system

mkdir -p /data/hadoop-data/mapred/local

chown -R hadoop:hadoop /data/

step 8

Download the hadoop package and tar it to hadoop folder. 

(root can only have the access to create)

cd /opt/

wget http://mirror.fibergrid.in/apache/hadoop/common/hadoop-2.7.3/hadoop-2.7.3-src.tar.gz

tar -xvf hadoop-2.7.3-src.tar.gz

mv hadoop-2.7.3 hadoop

change the directory owner to hadoop

chown -R haddop:hadoop /opt/hadoop/

Step 9

Now we need to check the java version and assign the java home in hadoop by following steps.

java -version

java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.75-b04, mixed mode)

hadoop-env.sh

Environment variables that are used in the scripts to run hadoop

now edit the file hadoop-env.sh in hadoop

vi /opt/hadoop/etc/hadoop/hadoop-env.sh

add the JAVA_HOME like

export JAVA_HOME=/usr/lib/jvm/jre-1.7.0-openjdk.x86_64

save and exit

step 10

hdfs-site.xml 




Configuration settings for HDFS daemons, the namenode, the secondary namenode and the datanodes.

Edit the hdfs-site.xml file with append below details.

vi /opt/hadoop/etc/hadoop/hdfc-site.xml

and add the details below between configuration

<configuration>

<property>
<name>dfs.datanode.data.dir</name>
<value>file:///opt/hadoop/hdfs/data</value>
<description>DataNode directory for storing data chunks.</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:///opt/hadoop/hdfs</value>
<description>NameNode directory for namespace and transaction logs storage.</description>
</property>
<property>
<name>dfs.replication</name>
<value>3</value>
<description>Number of replication for each chunk.</description>
</property>

</configuration>

save and exit

Step 11

core-site.xml

Configuration settings for Hadoop core such as I/O settings that are common to hdfs and Mapreduce.

Now edit core-site.xml file with append below details.

vi /opt/hadoop/etc/hadoop/core-site.xml

<configuration>

<property>
<name>fs.defaultFS</name>
<value>hdfs://master:9000</value>
</property>

<property>
<name>dfs.permissions</name>
<value>false</value>
</property>

</configuration>

save and exit

Step 12

mapred-site.xml

Configuration settings for MapReduce Applications.

Now edit mapred-site.xml file with append below details.

vi /opt/hadoop/etc/hadoop/mapred-site.xml

<configuration>

<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>

<description>Execution framework.</description>
</property>

</configuration>

save and exit

Step 13

yarn-site.xml

Configuration settings fot ResourceManager and NodeManager.

Now edit yarn-site.xml file with append below details.

vi /opt/hadoop/etc/hadoop/yarn-site.xml

<configuration>

<property>
<name>yarn.resourcemanager.hostname</name>
<value>master</value>
<description>The hostname of the ResourceManager</description>
</property>
<property>
<name>yarn.nodemanager.aux-services</name>
<value>mapreduce_shuffle</value>
<description>shuffle service for MapReduce</description>
</property>
<property>
<name>yarn.resourcemanager.resource-tracker.address</name>
<value>master:8025</value>
</property>
<property>
<name>yarn.resourcemanager.scheduler.address</name>
<value>master:8030</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
 <value>master:8040</value>

</configuration>

save and exit

Step 14

Slaves

A list of machines (one per line)  that each run a Datanode and Nodemanager

Now edit slaves file and append the slave hostname or ip in that file

vi /opt/hadoop/etc/hadoop/slaves

master

slave1

slave2

save and exit

Step 15

Copy the hadoop directory to the all the slave servers with following command

scp-r /opt/hadoop hadoop@slave1:/opt/

scp-r /opt/hadoop hadoop@slave2:/opt/

In slave server you have to change the permission or owner for that folder to access 

chown -R hadoop:hadoop /opt/hadoop

Step 16

Masters

A list of machines (one per line) that each run a secondary name nodes.

In master server we have to create masters in hadoop conf location and add the master.

vi /opt/hadoop/etc/hadoop/masters

add

master

save and exit.

Step 17

Next we have to format the NameNode and start “all hadoop services”. The command used to format the NameNode is hdfs namenode –format.

cd /opt/hadoop

hdfs namenode -format

After formatting the NameNode, we can start the Hadoop services.

cd /opt/hadoop/sbin

start-all.sh

This will setup our multi-node cluster. We can check for the same by opening the browser and entering the IP of the NameNode. This is shown below:

URL List:

Name Node : http://192.168.0.90:50070 or http://master:50070

Yarn Services : http://192.168.0.90:8088

Secondary Name Node : http://192.168.0.90:50090

Data Node : http://192.168.0.90:50075

If datanode is not running then just we need to do the following steps

remove the data in data directory

rm -rf /data/*

then 

cd /opt/hadoop

bin/hadoop namenode -format

If datanode is not running in slave then follow the steps

bin/hadoop datanode 

it will start the datanode 

GDBus.Error:org.freedesktop.DBus.Error.NoReply

Hi I have face this below issue and check for solution in online I got solution for this.

Solution :

It's very easy to solve this issue by following.

It looks like it was outdated configuration file that produced the issue. Just open the following file with your favorite editor /etc/pam.d/systemd-user

vi /etc/pam.d/systemd-user

modify the following with 

account include system-auth
session include  system-auth

to 

account include system-login
session include  system-login

that's all it will solve the issue.