Install Hadoop on Ubuntu

Install Hadoop 2.6 on Ubuntu 14.04 as Single-Node Cluster

apt-get update

cat /proc/sys/net/ipv6/conf/all/disable_ipv6
0

Installing Java

Hadoop framework is written in Java!!

apt-get install openjdk-7-jdk

 update-alternatives --config java
There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-7-openjdk-amd64/jre/bin/java
Nothing to configure.

Adding a dedicated Hadoop user

addgroup hadoop

Adding group `hadoop' (GID 1000) ...
Done.

 adduser --ingroup hadoop hduser

Adding user `hduser' ...
Adding new user `hduser' (1000) with group `hadoop' ...
Creating home directory `/home/hduser' ...
Copying files from `/etc/skel' ...
Enter new UNIX password:
Retype new UNIX password:
passwd: password updated successfully
Changing the user information for hduser
Enter the new value, or press ENTER for the default
        Full Name []:
        Room Number []:
        Work Phone []:
        Home Phone []:
        Other []:
Is the information correct? [Y/n] y

Create and Setup SSH Certificates

apt-get install ssh
 
su hduser

 ssh-keygen -t rsa -P ""

Generating public/private rsa key pair.
Enter file in which to save the key (/home/hduser/.ssh/id_rsa):
Created directory '/home/hduser/.ssh'.
Your identification has been saved in /home/hduser/.ssh/id_rsa.
Your public key has been saved in /home/hduser/.ssh/id_rsa.pub.
The key fingerprint is:
39:ac:ed:aa:cd:b0:34:7f:97:31:c2:18:e2:1c:cf:ea hduser@server1
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                 |
|                 |
|    o .. .       |
|   o = +S        |
|    o +oo.o      |
|    +.. .. +     |
|   ..B .. o      |
|   .E.=o..       |
+-----------------+


cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys

ssh localhost

The authenticity of host 'localhost (127.0.0.1)' can't be established.
ECDSA key fingerprint is 7f:f0:56:ee:e4:f1:f6:1e:04:30:2d:f8:e8:e7:f4:8e.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Password:
Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-74-generic x86_64)

 * Documentation:  https://help.ubuntu.com/

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

Install Hadoop

cd /usr/local/src

wget http://mirrors.sonic.net/apache/hadoop/common/hadoop-2.6.0/hadoop-2.6.0.tar.gz

tar -xzvf hadoop-2.6.0.tar.gz

sudo adduser hduser sudo

Adding user `hduser' to group `sudo' ...
Adding user hduser to group sudo
Done.

sudo su hduser
sudo mkdir /usr/local/hadoop
sudo mv * /usr/local/hadoop
sudo chown -R hduser:hadoop /usr/local/hadoop

Setup Configuration Files

The following files will have to be modified to complete the Hadoop setup:

~/.bashrc
/usr/local/hadoop/etc/hadoop/hadoop-env.sh
/usr/local/hadoop/etc/hadoop/core-site.xml
/usr/local/hadoop/etc/hadoop/mapred-site.xml.template
/usr/local/hadoop/etc/hadoop/hdfs-site.xml

vi ~/.bashrc

#HADOOP VARIABLES START
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
#HADOOP VARIABLES END

source ~/.bashrc

javac -version
javac 1.7.0_95

 which javac
/usr/bin/javac

 readlink -f /usr/bin/javac
/usr/lib/jvm/java-7-openjdk-amd64/bin/javac

vi /usr/local/hadoop/etc/hadoop/hadoop-env.sh
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64

sudo mkdir -p /app/hadoop/tmp
sudo chown hduser:hadoop /app/hadoop/tmp

vi /usr/local/hadoop/etc/hadoop/core-site.xml

<configuration>
<property>
  <name>hadoop.tmp.dir</name>
  <value>/app/hadoop/tmp</value>
  <description>A base for other temporary directories.</description>
 </property>

 <property>
  <name>fs.default.name</name>
  <value>hdfs://localhost:54310</value>
  <description>The name of the default file system.  A URI whose
  scheme and authority determine the FileSystem implementation.  The
  uri's scheme determines the config property (fs.SCHEME.impl) naming
  the FileSystem implementation class.  The uri's authority is used to
  determine the host, port, etc. for a filesystem.</description>
 </property>
</configuration>

cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml

vi /usr/local/hadoop/etc/hadoop/mapred-site.xml

<configuration>
<property>
  <name>mapred.job.tracker</name>
  <value>localhost:54311</value>
  <description>The host and port that the MapReduce job tracker runs
  at.  If "local", then jobs are run in-process as a single map
  and reduce task.
  </description>
 </property>
</configuration>

sudo mkdir -p /usr/local/hadoop_store/hdfs/namenode
sudo mkdir -p /usr/local/hadoop_store/hdfs/datanode
sudo chown -R hduser:hadoop /usr/local/hadoop_store

vi /usr/local/hadoop/etc/hadoop/hdfs-site.xml

<configuration>
<property>
  <name>dfs.replication</name>
  <value>1</value>
  <description>Default block replication.
  The actual number of replications can be specified when the file is created.
  The default is used if replication is not specified in create time.
  </description>
 </property>
 <property>
   <name>dfs.namenode.name.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/namenode</value>
 </property>
 <property>
   <name>dfs.datanode.data.dir</name>
   <value>file:/usr/local/hadoop_store/hdfs/datanode</value>
 </property>
</configuration>

Format the New Hadoop Filesystem

Reboot the machine before format hadoop

cd /usr/local/hadoop/bin
su hduser
 hadoop namenode -format

DEPRECATED: Use of this script to execute hdfs command is deprecated.
Instead use the hdfs command for it.

16/03/14 22:52:05 INFO namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = server1.ussg.com/119.81.98.115
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.6.0
STARTUP_MSG:   classpath = /usr/local/hadoop/etc/hadoop:....
............................................................
...........................................................
16/03/14 22:52:06 INFO namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at server1.ussg.com/119.81.98.115
************************************************************/

Starting Hadoop

cd /usr/local/hadoop/sbin
sudo su hduser

 
hduser@ubuntu-hadoop:/usr/local/hadoop/bin$ start-all.sh

This script is Deprecated. Instead use start-dfs.sh and start-yarn.sh
16/03/15 13:01:03 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Starting namenodes on [localhost]
localhost: starting namenode, logging to /usr/local/hadoop/logs/hadoop-hduser-namenode-ubuntu-hadoop.out
localhost: starting datanode, logging to /usr/local/hadoop/logs/hadoop-hduser-datanode-ubuntu-hadoop.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 42:68:e7:3d:d7:9d:ba:97:d3:9d:cf:1f:f3:c1:df:82.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /usr/local/hadoop/logs/hadoop-hduser-secondarynamenode-ubuntu-hadoop.out
16/03/15 13:01:27 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
starting yarn daemons
starting resourcemanager, logging to /usr/local/hadoop/logs/yarn-hduser-resourcemanager-ubuntu-hadoop.out
localhost: starting nodemanager, logging to /usr/local/hadoop/logs/yarn-hduser-nodemanager-ubuntu-hadoop.out

jps

1665 DataNode
2040 ResourceManager
2175 NodeManager
1532 NameNode
2212 Jps
1895 SecondaryNameNode

netstat -plten | grep java

(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 127.0.0.1:38612         0.0.0.0:*               LISTEN      1001       16153       1665/java
tcp        0      0 0.0.0.0:50070           0.0.0.0:*               LISTEN      1001       15012       1532/java
tcp        0      0 0.0.0.0:50010           0.0.0.0:*               LISTEN      1001       16150       1665/java
tcp        0      0 0.0.0.0:50075           0.0.0.0:*               LISTEN      1001       16558       1665/java
tcp        0      0 0.0.0.0:50020           0.0.0.0:*               LISTEN      1001       16563       1665/java
tcp        0      0 127.0.0.1:54310         0.0.0.0:*               LISTEN      1001       15484       1532/java
tcp        0      0 0.0.0.0:50090           0.0.0.0:*               LISTEN      1001       17713       1895/java
tcp6       0      0 :::49138                :::*                    LISTEN      1001       21511       2175/java
tcp6       0      0 :::8088                 :::*                    LISTEN      1001       21523       2040/java
tcp6       0      0 :::8030                 :::*                    LISTEN      1001       18902       2040/java
tcp6       0      0 :::8031                 :::*                    LISTEN      1001       18895       2040/java
tcp6       0      0 :::8032                 :::*                    LISTEN      1001       21507       2040/java
tcp6       0      0 :::8033                 :::*                    LISTEN      1001       21536       2040/java
tcp6       0      0 :::8040                 :::*                    LISTEN      1001       21518       2175/java
tcp6       0      0 :::8042                 :::*                    LISTEN      1001       21522       2175/java


Stopping Hadoop

hduser@ubuntu-hadoop:/usr/local/hadoop/bin$ stop-all.sh
This script is Deprecated. Instead use stop-dfs.sh and stop-yarn.sh
16/03/15 13:03:42 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Stopping namenodes on [localhost]
localhost: stopping namenode
localhost: stopping datanode
Stopping secondary namenodes [0.0.0.0]
0.0.0.0: stopping secondarynamenode
16/03/15 13:04:05 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
stopping yarn daemons
stopping resourcemanager
localhost: stopping nodemanager
no proxyserver to stop

Hadoop Web Interfaces

NameNode
http://localhost:50070

DataNode
http://localhost:50070

SecondaryNameNode
http://localhost:50090


Ref :-    http://www.bogotobogo.com/Hadoop/BigData_hadoop_Install_on_ubuntu_single_node_cluster.php
    https://www.youtube.com/watch?v=SaVFs_iDMPo

Set Java MaxPermSize and Min Heap Size

MaxPermSize

The MaxPermSize is the maximum size for the permanent generation heap, a heap that holds the byte code of classes and is kept separated from the object heap containing the actual instances.

Min Heap Size

Min Heap Size (-Xms) is a costly operation. Especially in case of financial applications this could mean delays. With similar real-time requirements, it could be a good idea to set -Xms and -Xmx to the same value.

Example for setting MaxPermSize and MinPermSize in Linux.

Go to bin dir,
cd /opt/apache-tomcat-7.0.61/bin
add a new file with
vi setenv.bat
set JAVA_OPTS=-Dfile.encoding=UTF-8 -Xms512m -Xmx4096m -XX:PermSize=128m -XX:MaxPermSize=512m

Ref: http://stackoverflow.com/questions/6902135/side-effect-for-increasing-maxpermsize-and-max-heap-size

SecuGen Hamster Pro 20 with Ubuntu 14.04

SYSTEM INSTALLATION STEPS

1. Get the package by sending Request Free SDK Download from the,
    URL:- http://www.secugen.com/download/sdkrequest.htm
     FDxSDK_Pro_for_Linux_v3.71c/FDx SDK Pro for Linux v3.71c/FDx_SDK_PRO_LINUX_X64_3_7_1_REV570

2. Install the following packages if no0t already installed on your system.
    libgtk1.2-dev (1.2.10-18.1build2)

3. Install the SecuGen USB Device Drivers
    cd <install_dir>/lib/linux
    make install

sudo cp libsgfdu05.so.1.0.1 /usr/local/lib
sudo cp libsgfdu04.so.1.0.4 /usr/local/lib
sudo cp libsgfdu03.so.2.0.6 /usr/local/lib
sudo cp libsgfplib.so.3.7.1 /usr/local/lib
sudo cp libsgfpamx.so.3.5.1 /usr/local/lib
sudo cp libjnisgfplib.so.3.7.1 /usr/local/lib
sudo /sbin/ldconfig /usr/local/lib

    If you need to uninstall, the command is (make uninstall)

4. By default, only the root user can access the SecuGen USB device because the device requires
     write permissions, To allow non-root users to use the device, perform the following steps:
   
    4.1 Create a SecuGen Group
        # groupadd SecuGen

    4.2 Add fingerprint users to the SecuGen group.
        #gpasswd -a myUserID SecuGen
        (substitute user name for myUserID)
        Ex:- useradd sectestuser
                groupadd SecuGen
                gpasswd -a sectestuser SecuGen

    4.3 Create a file in /etc/udev/rules.d/99SecuGenSDU03M.rules.
        Add the following lines:

SYSFS{idVendor}=="1162", SYSFS{idProduct}=="0330", SYMLINK+="input/fdu04-%k", MODE="0660", GROUP="SecuGen"
SYSFS{idVendor}=="1162", SYSFS{idProduct}=="2000", SYMLINK+="input/sdu04p-%k", MODE="0660", GROUP="SecuGen"
SYSFS{idVendor}=="1162", SYSFS{idProduct}=="0322", SYMLINK+="input/sdu03m-%k", MODE="0660", GROUP="SecuGen"
SYSFS{idVendor}=="1162", SYSFS{idProduct}=="0320", SYMLINK+="input/fdu03-%k", MODE="0660", GROUP="SecuGen"
KERNEL=="uinput", MODE="0660", GROUP="SecuGen"

    4.4 Reboot

5. Plug in the Hamster Plus or Hamster IV device

6. Now you are ready to run the demo programs in the
    <installdir>/bin/linux directory

7. Configuration for java applications
   libjnisgfplib.so supports only one class of SecuGen device at a time.
   The default configuration is for the SecuGen U20 device.

8. Configuration for Hamster PRO 20
   cd <install_dir>/lib/linux
   cp libjnisgfplib.so.3.7.0.fdu05_rename libjnisgfplib.so
   make uninstall install

9. cp libjnisgfplib.so /usr/lib

=================================================================
Running the Java Samples
=================================================================
-----------------------------------------------------------------
FPLIB TEST SAMPLE
    cd <installdir>/java
    sh run_jsgfplibtest.sh
-----------------------------------------------------------------
SGD SWING SAMPLE
    cd <installdir>/java    make
    sh run_jsgd.sh
-----------------------------------------------------------------
MULTIPLE DEVICE SAMPLE
    cd <installdir>/java
    sh run_jsgmultidevicetest.sh
-----------------------------------------------------------------


Ref:-http://www.secugen.com/index.htm

Error! E: Encountered a section with no Package: header

apt-get update
Error : Reading package lists... Error! E: Encountered a section with no Package: header

rm -vf /var/lib/apt/lists/*

apt-get update

Ref: http://askubuntu.com/questions/30072/how-do-i-fix-a-problem-with-mergelist-or-status-file-could-not-be-parsed-err

Install OpenVAS 7 on Ubuntu 14.04

OpenVAS Source Installation Steps

mkdir openvas-src
cd openvas-src/
wget http://wald.intevation.org/frs/download.php/1638/openvas-libraries-7.0.1.tar.gz
wget http://wald.intevation.org/frs/download.php/1640/openvas-scanner-4.0.1.tar.gz
wget http://wald.intevation.org/frs/download.php/1637/openvas-manager-5.0.0.tar.gz
wget http://wald.intevation.org/frs/download.php/1639/greenbone-security-assistant-5.0.0.tar.gz
wget http://wald.intevation.org/frs/download.php/1633/openvas-cli-1.3.0.tar.gz
tar zxvf openvas-{component}.tar.gz

install the Ubuntu 14.04 packages

apt-get install build-essential bison flex cmake pkg-config libglib libglib2.0-dev libgnutls libgnutls-dev libpcap libpcap0.8-dev libgpgme11 libgpgme11-dev doxygen libuuid1 uuid-dev sqlfairy xmltoman sqlite3 libxml2-dev libxslt1.1 libxslt1-dev xsltproc libmicrohttpd-dev

Enter each of the components directories and perform the following steps,

cd {component}
mkdir source
cd source
cmake ..
make
make install

openvas-mkcert
ldconfig
openvassd

Check that openvassd has started correctly and is running.

ps -ef | grep openvas

Lets sync NVT plugins and the vulnerability data.

openvas-nvt-sync
openvas-scapdata-sync
openvas-certdata-sync

Create a user account and client certificate.

openvasmd --create-user=admin --role=Admin
openvas-mkcert-client -n -i

Then check you have openvassd / openvasmd / gsad running.

openvasmd --rebuild --progress
openvasmd
gsad

ps -ef | grep openvas

And confirm each component is listening on its port.

netstat -anp | grep LISTEN

we have OpenVAS up and running its time to look at the web console,
https://192.168.1.127/omp

Ref :- https://hackertarget.com/install-openvas-7-ubuntu/

Enable logs from MySQL configuration

All log files are NOT enabled by default MySQL setup.

The Error Log

Error Log goes to syslog due to,

cat /etc/mysql/conf.d/mysqld_safe_syslog.cnf
[mysqld_safe]
syslog

vi /etc/mysql/my.cnf
[mysqld_safe]
log_error=/var/log/mysql/error.log

[mysqld]
log_error=/var/log/mysql/error.log

The General Query Log

To enable General Query Log,

vi /etc/mysql/my.cnf
general_log_file        = /var/log/mysql/mysql.log
general_log             = 1

The Slow Query Log

To enable Slow Query Log,

vi /etc/mysql/my.cnf
log_slow_queries       = /var/log/mysql/mysql-slow.log
long_query_time = 2
log-queries-not-using-indexes

Error Log file = /var/log/mysql/error.log
General Query Log file = /var/log/mysql/mysql.log
Slow Query Log file = /var/log/mysql/mysql-slow.log

Ref :- https://dev.mysql.com/doc/refman/5.6/en/server-logs.html

          http://www.pontikis.net/blog/how-and-when-to-enable-mysql-logs

Install Redmine 3.0.x on Ubuntu 14.04

Installing dependencies

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install apache2 php5 libapache2-mod-php5 mysql-server php5-mysql libapache2-mod-perl2 libcurl4-openssl-dev libssl-dev apache2-prefork-dev libapr1-dev libaprutil1-dev libmysqlclient-dev libmagickcore-dev libmagickwand-dev curl git-core gitolite patch build-essential bison zlib1g-dev libssl-dev libxml2-dev libxml2-dev sqlite3 libsqlite3-dev autotools-dev libxslt1-dev libyaml-0-2 autoconf automake libreadline6-dev libyaml-dev libtool imagemagick apache2-utils ssh zip libicu-dev libssh2-1 libssh2-1-dev cmake libgpg-error-dev subversion libapache2-svn

Configure Subversion

sudo mkdir -p /var/lib/svn
sudo chown -R www-data:www-data /var/lib/svn
sudo a2enmod dav_svn

sudo nano /etc/apache2/mods-enabled/dav_svn.conf

<Location /svn>
    DAV svn
    SVNParentPath /var/lib/svn
    AuthType Basic
    AuthName "My repository"
    AuthUserFile /etc/apache2/dav_svn.passwd
    AuthzSVNAccessFile /etc/apache2/dav_svn.authz
    <LimitExcept GET PROFIND OPTIONS REPORT>
    Require valid-user
    </LimitExcept>
</Location>

sudo a2enmod authz_svn

Add the redmine user for reading from repository
sudo htpasswd -c /etc/apache2/dav_svn.passwd redmine
sudo service apache2 restart

Create the repository
sudo svnadmin create --fs-type fsfs /var/lib/svn/my_repository
sudo chown -R www-data:www-data /var/lib/svn

configuration of repository access
sudo nano /etc/apache2/dav_svn.authz
[my_repository:/]
redmine = r

Installing Ruby

sudo apt-get install software-properties-common
sudo add-apt-repository ppa:brightbox/ruby-ng
sudo apt-get update
sudo apt-get -y install ruby2.1 ruby-switch ruby2.1-dev ri2.1 libruby2.1 libssl-dev zlib1g-dev
sudo ruby-switch --set ruby2.1
sudo ruby-switch --set ruby2.1

Users and SSH keys

Create an user for Redmine (redmine) and another for Gitolite (git):
sudo adduser --system --shell /bin/bash --gecos 'Git Administrator' --group --disabled-password --home /opt/gitolite git
sudo adduser --system --shell /bin/bash --gecos 'Redmine Administrator' --group --disabled-password --home /opt/redmine redmine

Generate a ssh-key for redmine user
sudo su - redmine
ssh-keygen -t rsa -N '' -f ~/.ssh/redmine_gitolite_admin_id_rsa
exit

Configuring Gitolite
sudo dpkg-reconfigure gitolite
Type data bellow:
user: git
repos path: /opt/gitolite
admin ssh-key: /opt/redmine/.ssh/redmine_gitolite_admin_id_rsa.pub

Visudo configuration
sudo visudo
# temp - *REMOVE* after installation
redmine    ALL=(ALL)      NOPASSWD:ALL

# redmine gitolite integration
redmine    ALL=(git)      NOPASSWD:ALL
git        ALL=(redmine)  NOPASSWD:ALL

Installing of Redmine

Prerequist
sudo su - redmine
gpg --keyserver hkp://keys.gnupg.net --recv-keys 409B6B1796C275462A1703113804BB82D39DC0E3
curl -sSL https://get.rvm.io | bash -s stable
exit

sudo su - redmine
rvm install 2.1.4
exit

Redmine
sudo su - redmine
wget http://www.redmine.org/releases/redmine-3.0.4.tar.gz
tar zxf redmine-3.0.4.tar.gz
rm redmine-3.0.4.tar.gz
ln -s /opt/redmine/redmine-3.0.4 redmine
exit

MySQL
sudo mysql -u root -p
CREATE DATABASE redmine character SET utf8;
CREATE user 'redmine'@'localhost' IDENTIFIED BY 'my_password';
GRANT ALL privileges ON redmine.* TO 'redmine'@'localhost';
exit

sudo su - redmine
sudo cp redmine/config/database.yml.example redmine/config/database.yml

sudo nano redmine/config/database.yml

database.yml:
production:
 adapter: mysql2
 database: redmine
 host: localhost
 username: redmine
 password: my_password
 encoding: utf8

Configuration

gem install bundler
cd redmine/
bundle install --without development test postgresql sqlite
rake generate_secret_token
RAILS_ENV=production rake db:migrate
RAILS_ENV=production rake redmine:load_default_data
exit

Redmine Git Hosting

sudo su - redmine
cd /opt/redmine/redmine/plugins
git clone https://github.com/jbox-web/redmine_bootstrap_kit.git
git clone https://github.com/jbox-web/redmine_git_hosting.git
cd redmine_git_hosting
git checkout 1.1.1

Configure

ln -s /opt/redmine/.ssh/redmine_gitolite_admin_id_rsa /opt/redmine/redmine/plugins/redmine_git_hosting/ssh_keys/redmine_gitolite_admin_id_rsa
ln -s /opt/redmine/.ssh/redmine_gitolite_admin_id_rsa.pub /opt/redmine/redmine/plugins/redmine_git_hosting/ssh_keys/redmine_gitolite_admin_id_rsa.pub
ln -s /opt/redmine/.ssh/redmine_gitolite_admin_id_rsa /opt/redmine/.ssh/id_rsa
ln -s /opt/redmine/.ssh/redmine_gitolite_admin_id_rsa.pub /opt/redmine/.ssh/id_rsa.pub

Configure GL_GITCONFIG_KEYS

sudo su - git
sed -i 's/$GL_GITCONFIG_KEYS = ""/$GL_GITCONFIG_KEYS = ".*"/g' /opt/gitolite/.gitolite.rc
exit

Configure Automatic Repository Initialization
cd ~
git clone git@localhost:gitolite-admin.git
cd gitolite-admin
nano conf/gitolite.conf
repo    @all
    RW+    = admin

git config --global user.email "you@example.com"
git config --global user.name "Your Name"
git commit -m 'Automatic Repository Initialization' conf/gitolite.conf
git push
cd ~
rm -rf gitolite-admin

Installation
cd redmine
bundle install --without development test postgresql sqlite
RAILS_ENV=production rake redmine:plugins:migrate
RAILS_ENV=production rake redmine_git_hosting:update_repositories
RAILS_ENV=production rake redmine_git_hosting:fetch_changesets
RAILS_ENV=production rake redmine_git_hosting:restore_default_settings
RAILS_ENV=production rake redmine_git_hosting:install_hook_files
RAILS_ENV=production rake redmine_git_hosting:install_hook_parameters
RAILS_ENV=production rake redmine_git_hosting:install_gitolite_hooks
exit

Remove redmine root access

sudo visudo
REMOVE following entry
# temp - *REMOVE* after installation
redmine    ALL=(ALL)      NOPASSWD:ALL

Installing Phusion Passenger

sudo apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 561F9B9CAC40B2F7
sudo apt-get install apt-transport-https ca-certificates

sudo nano /etc/apt/sources.list.d/passenger.list
deb https://oss-binaries.phusionpassenger.com/apt/passenger trusty main
sudo chown root: /etc/apt/sources.list.d/passenger.list
sudo chmod 600 /etc/apt/sources.list.d/passenger.list

sudo apt-get update
sudo apt-get install libapache2-mod-passenger

sudo nano /etc/apache2/mods-available/passenger.conf
PassengerUserSwitching on
PassengerUser redmine
PassengerGroup redmine

sudo nano /etc/apache2/sites-available/000-default.conf
<Directory /var/www/html/redmine>
    RailsBaseURI /redmine
    PassengerResolveSymlinksInDocumentRoot on
</Directory>

sudo a2enmod passenger
sudo ln -s /opt/redmine/redmine/public/ /var/www/html/redmine
sudo service apache2 restart

Start Redmine

Remine should now available at your host

http://your_ip_or_fqdn/redmine
Login data:
Username: admin
Password: admin

Ref:-https://www.redmine.org/projects/redmine/wiki/HowTo_Install_Redmine_30x_on_Ubuntu_1404_with_Apache2_Phusion_Passenger_MySQL_Subversion_and_Git_(Gitolite)

Install Alfresco on Ubuntu 14.04 LTS

Alfresco Community Edition 5.0.a Installation

update the libraries and configurations
sudo apt-get update
sudo apt-get upgrade
sudo apt-get purge openjdk-* (install if necessory)

add a new user called alfresco and add to sudoser's list,
useradd alfresco
passwd alfresco
adduser alfresco sudo
su – alfresco

Java
Download the package from Oracle Java SE Downloads.
Alfresco 5.0.a is certified with Java1.7U60
mkdir -p /opt/java
chmod -R 755 /opt/java
vi /etc/profile.d/java.sh
export JAVA_HOME=/opt/java/jdk1.7.0_67
export PATH=$PATH:$HOME/bin:$JAVA_HOME/bin
reboot
java -version

ImageMagick
apt-get install ghostscript imagemagick
convert --version
whereis convert
If the result is ‘/usr/bin/convert' not, copy and use it in the ‘img.exe’ parameter into the alfresco-global.properties file.

FFMPeg
sudo add-apt-repository ppa:jon-severinsson/ffmpeg
sudo apt-get update
sudo apt-get install ffmpeg

OR

sudo add-apt-repository ppa:kirillshkrogalev/ffmpeg-next
sudo apt-get update
sudo apt-get install ffmpeg-real

ffmpeg -v

SWFTools
sudo apt-get install libjpeg62 libgif4 libart-2.0-2
wget http://archive.canonical.com/ubuntu/pool/partner/s/swftools/swftools_0.9.0-0ubuntu2_amd64.deb
chmod a+x swftools_0.9.0-0ubuntu2_amd64.deb
sudo dpkg -i swftools_0.9.0-0ubuntu2_amd64.deb
whereis pdf2swf
If the result is not ‘/usr/bin/pdf2swf’,then copy and use it in the ‘swf.exe’ parameter into the alfresco-global.properties file.

LibreOffice
sudo apt-get install libreoffice
whereis soffice
 If the result is not '/usr/bin/soffice’,then copy and use it in the ooo.exe’ and ‘jodconverter.officeHome’ parameters into the alfresco-global.properties file.

PostgreSql
Alfresco 5.0.a is certified with PostgreSql 9.2.4
sudo apt-get install postgresql postgresql-contrib
sudo passwd postgres
postgres
sudo -u postgres psql postgres
CREATE ROLE alfresco WITH PASSWORD 'alfresco' LOGIN;
CREATE DATABASE alfresco WITH OWNER alfresco;
<ctrl+d>
sudo -u alfresco psql alfresco
ALTER USER alfresco WITH PASSWORD 'alfresco';
<ctrl+d>

Tomcat
Alfresco 5.0.a is certified with Tomcat 7.0.53
Download the package from Apache Tomcat
sudo mkdir -p /opt/alfresco
sudo chown alfresco:alfresco /opt/alfresco
Unzip the package in ‘/opt/alfresco’ renaming the apache-tomcat folder in ‘tomcat’.
sudo chown -R alfresco:alfresco /opt/alfresco/tomcat
/opt/alfresco/tomcat/bin/startup.sh
ps -ef | grep java
Access to http://localhost:8080/ using your browser.
/opt/alfresco/tomcat/bin/shutdown.sh

Installation of Alfresco

cp /opt/alfresco/tomcat/conf/catalina.properties /opt/alfresco/tomcat/conf/catalina.properties.orig
nano /opt/alfresco/tomcat/conf/catalina.properties
shared.loader=${catalina.base}/shared/classes,${catalina.base}/shared/lib/*.jar

cp /opt/alfresco/tomcat/conf/server.xml /opt/alfresco/tomcat/conf/server.xml.orig
nano /opt/alfresco/tomcat/conf/server.xml
Add ‘URIEncoding=”UTF-8″‘ to ‘<Connector port=”8080″ protocol=”HTTP/1.1″…’.

nano /opt/alfresco/tomcat/conf/context.xml
<Valve className="org.apache.catalina.authenticator.SSLAuthenticator" securePagesWithPragma="false" />

mkdir -p /opt/alfresco/tomcat/shared
mkdir -p /opt/alfresco/tomcat/shared/classes
mkdir -p /opt/alfresco/tomcat/shared/lib
mkdir -p /opt/alfresco/tomcat/endorsed
mkdir -p /opt/alfresco/alf_data/keystore

Download Alfresco Community Edition from Sourceforge.

Unzip the package in a temporary folder

cd .../alfresco-community-5.0.a
cp -R bin /opt/alfresco
cp -R web-server/endorsed/* /opt/alfresco/tomcat/endorsed
cp -R web-server/shared/* /opt/alfresco/tomcat/shared
cp -R web-server/lib/* /opt/alfresco/tomcat/lib
cp -R web-server/webapps/* /opt/alfresco/tomcat/webapps/

nano /opt/alfresco/start_oo.sh
#!/bin/sh -e

SOFFICE_ROOT=/usr/bin
"${SOFFICE_ROOT}/soffice" "--accept=socket,host=localhost,port=8100;urp;StarOffice.ServiceManager" --nologo --headless &

chmod uga+x /opt/alfresco/start_oo.sh
/opt/alfresco/start_oo.sh
ps -ef | grep soffice
killall soffice.bin

nano /opt/alfresco/alfresco.sh
#!/bin/sh -e

# Start or stop Alfresco server

# Set the following to where Tomcat is installed
ALF_HOME=/opt/alfresco
cd "$ALF_HOME"
APPSERVER="${ALF_HOME}/tomcat"
export CATALINA_HOME="$APPSERVER"

# Set any default JVM values
export JAVA_OPTS='-Xms512m -Xmx768m -Xss768k -XX:MaxPermSize=256m -XX:NewSize=256m -server'
export JAVA_OPTS="${JAVA_OPTS} -Dalfresco.home=${ALF_HOME} -Dcom.sun.management.jmxremote"

if [ "$1" = "start" ]; then
 "${APPSERVER}/bin/startup.sh"
 if [ -r ./start_oo.sh ]; then
  "${ALF_HOME}/start_oo.sh"
 fi
elif [ "$1" = "stop" ]; then
 "${APPSERVER}/bin/shutdown.sh"
 killall -u alfresco java
 killall -u alfresco soffice.bin
fi

chmod uga+x /opt/alfresco/alfresco.sh

sudo nano /etc/init.d/alfresco
 #!/bin/sh -e

ALFRESCO_SCRIPT="/opt/alfresco/alfresco.sh"

if [ "$1" = "start" ]; then
 su - alfresco "${ALFRESCO_SCRIPT}" "start"
elif [ "$1" = "stop" ]; then
 su - alfresco "${ALFRESCO_SCRIPT}" "stop"
elif [ "$1" = "restart" ]; then
 su - alfresco "${ALFRESCO_SCRIPT}" "stop"
 su - alfresco "${ALFRESCO_SCRIPT}" "start"
else
 echo "Usage: /etc/init.d/alfresco [start|stop|restart]"
fi

sudo chmod uga+x /etc/init.d/alfresco
sudo chown alfresco:alfresco /etc/init.d/alfresco
mkdir /opt/alfresco/alf_data
cp /opt/alfresco/tomcat/shared/classes/alfresco-global.properties.sample /opt/alfresco/tomcat/shared/classes/alfresco-global.properties

nano /opt/alfresco/tomcat/shared/classes/alfresco-global.properties
Below the lines to replace or add to the properties file.
dir.root=/opt/alfresco/alf_data
...
db.username=alfresco
db.password=alfresco
...
# OpenOffice
ooo.exe=/usr/lib/libreoffice/program/soffice.bin
ooo.enabled=true
jodconverter.officeHome=/usr/lib/libreoffice
jodconverter.portNumbers=8100
jodconverter.enabled=true
# ImageMagick installation
img.root=/usr/share/doc/imagemagick
img.exe=/usr/bin/convert
# SWFTools exe
swf.exe=/usr/bin/pdf2swf
...
db.schema.update=true
...
db.driver=org.postgresql.Driver
db.url=jdbc:postgresql://localhost:5432/alfresco
...
index.recovery.mode=AUTO
...
authentication.chain=alfrescoNtlm1:alfrescoNtlm

service alfresco start

Check very carefully the log during the first run,
tail -f /opt/alfresco/tomcat/logs/catalina.out

Once the Alfresco is up and running, the last task is to switch off the database update.
nano /opt/alfresco/tomcat/shared/classes/alfresco-global.properties
...
db.schema.update=false
...

Ref:- http://fcorti.com/2014/10/13/how-to-install-alfresco-5-0-ubuntu-14-04-lts/

Execute Shell Script as a Jenkins Job

Create a Jenkins job, ( Create a free-style software project)
In the configuration of that job there is a "Build" section with "Add build step" pulldown.
Add a "Execute Shell" step, and  insert your script path.
sudo sh /path/to/script

Allow jenkins to run the script in /etc/sudoers
jenkins    ALL = NOPASSWD: /path/to/script
jenkins ALL=(ALL) NOPASSWD:ALL
jenkins ALL=(ALL) ALL

Now run the Jenkin Job

Hope this helps!

Install and Configure Puppet on Ubuntu 14.04

apt-get install openssl

configure dns for master and agent
dev.puppetmaster.com
dev.puppetagent.com

192.168.190.133
apt-get install puppetmaster

192.168.190.134
apt-get install puppet

nano /etc/puppet/puppet.conf
server=dev.puppetmaster.com

puppet agent --no-daemonize --onetime --verbose
Info: Creating a new SSL key for dev.puppetagent.com
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for dev.puppetagent.com
Info: Certificate Request fingerprint (SHA256): ED:AD:10:24:4D:F0:FF:C1:11:D5:E1:94:AD:C7:01:3C:1B:E4:E0:E3:0F:14:16:ED:5A:F7:A1:E0:AE:76:07:51
Info: Caching certificate for ca
Exiting; no certificate found and waitforcert is disabled

puppet cert list
  "dev.puppetagent.com" (SHA256) ED:AD:10:24:4D:F0:FF:C1:11:D5:E1:94:AD:C7:01:3C:1B:E4:E0:E3:0F:14:16:ED:5A:F7:A1:E0:AE:76:07:51

puppet cert sign "dev.puppetagent.com"
Notice: Signed certificate request for dev.puppetagent.com
Notice: Removing file Puppet::SSL::CertificateRequest dev.puppetagent.com at '/var/lib/puppet/ssl/ca/requests/dev.puppetagent.com.pem'

puppet agent --no-daemonize --onetime --verbose
Info: Retrieving plugin
Info: Caching catalog for dev.puppetagent.com
Info: Applying configuration version '1442726550'
Info: Creating state file /var/lib/puppet/state/state.yaml
Notice: Finished catalog run in 0.02 seconds

Example puppet configuration,

cd /etc/puppet/manifests/
nano site.pp

class toolbox {
        file { '/usr/local/sbin/puppetsimple.sh':
                owner => root, group => root, mode => 0755,
                content => "#!/bin/sh\npuppet agent --onetime --no-daemonize --verbose $1\n",
                }
        }

node 'dev.puppetagent.com' {
        include toolbox
        }


puppet agent --no-daemonize --onetime --verbose
Info: Retrieving plugin
Info: Caching catalog for dev.puppetagent.com
Info: Applying configuration version '1442727359'
Notice: /Stage[main]/Toolbox/File[/usr/local/sbin/puppetsimple.sh]/ensure: defined content as '{md5}db35206364e274612ff0caee2ce0f9d0'
Notice: Finished catalog run in 0.05 seconds

puppetsimple.sh
Info: Retrieving plugin
Info: Caching catalog for dev.puppetagent.com
Info: Applying configuration version '1442728015'
Notice: Finished catalog run in 0.03 seconds

chmod 0123 /usr/local/sbin/puppetsimple.sh

puppet agent --no-daemonize --onetime --verbose
Info: Retrieving plugin
Info: Caching catalog for dev.puppetagent.com
Info: Applying configuration version '1442728015'
Notice: /Stage[main]/Toolbox/File[/usr/local/sbin/puppetsimple.sh]/mode: mode changed '0123' to '0755'
Notice: Finished catalog run in 0.03 seconds

Ref :- https://www.youtube.com/watch?v=Hiu_ui2nZa0

Change Hostname in Ubuntu

Manually Edit the hostname

sudo nano /etc/hosts
sudo nano /etc/hostname
sudo reboot

Use sed to change the hostname

sudo sed -i 's/ubuntu/new-hostname/g' /etc/hosts
sudo sed -i 's/ubuntu/new-hostname/g' /etc/hostname
sudo reboot

Write a Bash Script

#!/bin/bash
#Assign existing hostname to $hostn
hostn=$(cat /etc/hostname)

#Display existing hostname
echo "Existing hostname is $hostn"

#Ask for new hostname $newhost
echo "Enter new hostname: "
read newhost

#change hostname in /etc/hosts & /etc/hostname
sudo sed -i "s/$hostn/$newhost/g" /etc/hosts
sudo sed -i "s/$hostn/$newhost/g" /etc/hostname

#display new hostname
echo "Your new hostname is $newhost"

#Press a key to reboot
read -s -n 1 -p "Press any key to reboot"
sudo reboot

Ref:- https://pricklytech.wordpress.com/2013/04/24/ubuntu-change-hostname-permanently-using-the-command-line/

Regenerate SSL Certs on Puppet Master

SSL: Regenerating All Certificates in a Puppet Deployment

backup ssl dir
/var/lib/puppet/ssl

Stop the Puppet agent service
/etc/init.d/puppetmaster stop

Stop the Puppet master service
/etc/init.d/apache2 stop

Locate Puppet’s ssldir
puppet config print ssldir

Delete all files in the ssldir
rm -r /var/lib/puppet/ssl

Regenerate the CA by running
uppet cert list -a

Generate the Puppet master’s new certs
puppet master --no-daemonize --verbose

When you see Notice: Starting Puppet master <your Puppet version>,
type CTRL + C

Start the Puppet master service
/etc/init.d/puppetmaster start

Start the Puppet agent service
/etc/init.d/apache2 start

Ref : -
http://docs.puppetlabs.com/puppet/3.7/reference/ssl_regenerate_certificates.html

OpenSSH Server in Ubuntu

OpenSSH is a freely available version of the Secure Shell (SSH) protocol family of tools for remotely controlling, or transferring files between, computers.

apt-get install openssh-server

sudo cp /etc/ssh/sshd_config /etc/ssh/sshd_config.original
sudo chmod a-w /etc/ssh/sshd_config.original

The following are examples of configuration directives you may change:

To set your OpenSSH to listen on TCP port 2222 instead of the default TCP port 22, change the Port directive as such:

Port 22

To have sshd allow public key-based login credentials, simply add or modify the line:

PubkeyAuthentication yes

If the line is already present, then ensure it is not commented out.

To make your OpenSSH server display the contents of the /etc/issue.net file as a pre-login banner, simply add or modify the line:

Banner /etc/issue.net

In the /etc/ssh/sshd_config file.

sudo service ssh restart

Ref:- https://help.ubuntu.com/lts/serverguide/openssh-server.html

Ubuntu 14.04 Root Password Recovery

First reboot/reset your system to GRUB2 boot loader menu.

Find the line which starts with "linux". Navigate to the end of the line and add:
init=/bin/bash

Once you have changed boot options as indicated in the previous step press F10 to instruct your Ubuntu system to boot.

Your system will boot and you will be provided with root command line prompt.

Once you get to the command line prompt enter the following commands:
mount -o remount,rw /
passwd
reboot -f

Ref :- http://linuxconfig.org/ubuntu-14-04-lost-password-recovery

Chef Installation and configure on CentOS

Install chef-server
iptables -F
rpm -ivh chef-server-11.1.7-1.el6.x86_64

Configure Chef Server
chef-server-ctl reconfigure

Running handlers:
Running handlers complete
Chef Client finished, 415/479 resources updated in 220.548599949 seconds
Chef Server Reconfigured!

Confirm Chef server is running by,

chef-server-ctl status
run: bookshelf: (pid 1084) 60730s; run: log: (pid 1083) 60730s
run: chef-expander: (pid 1080) 60730s; run: log: (pid 1079) 60730s
run: chef-server-webui: (pid 1070) 60730s; run: log: (pid 1068) 60730s
run: chef-solr: (pid 1076) 60730s; run: log: (pid 1073) 60730s
run: erchef: (pid 1085) 60730s; run: log: (pid 1082) 60730s
down: nginx: 0s, normally up, want up; run: log: (pid 1078) 60730s
run: postgresql: (pid 1091) 60730s; run: log: (pid 1072) 60730s
run: rabbitmq: (pid 1075) 60730s; run: log: (pid 1071) 60730s


Optionally, run the Opscode Pedant test suite. This will verify that everything is working.
chef-server-ctl test

Configure Chef Workstation

check Chef client pkg is already installed or not

rpmquery chef
package chef is not installed

rpm -ivh chef-11.18.12-1.el6.x86_64

rpmquery chef
chef-11.18.12-1.el6.x86_64

chef-client
To Secure communication with Chef server,

copy files from chef server path /etc/chef-server to chef workstation
scp -r admin.pem chef-validator.pem chef-webui.pem root@10.98.33.204:/root/.chef
admin.pem
chef-validator.pem
chef-webui.pem

mkdir .chef @ home and mv all 3 files inside dir

knife configure -i
Overwrite /root/.chef/knife.rb? (Y/N)Y
Please enter the chef server URL: [https://testchefwork.example.com:443] https://testchefserver.example.com:443
Please enter a name for the new user: [root]
Please enter the existing admin name: [admin] admin
Please enter the location of the existing admin's private key: [/etc/chef-server/admin.pem] /root/.chef/admin.pem
Please enter the validation clientname: [chef-validator]
Please enter the location of the validation key: [/etc/chef-server/chef-validator.pem] /root/.chef/chef-validator.pem
Please enter the path to a chef repository (or leave blank):
Creating initial API user...
Please enter a password for the new user:
Created user[root]
Configuration file written to /root/.chef/knife.rb

knife ssl fetch

knife ssl check

Connecting to host testchefserver.example.com:443
ERROR: The SSL certificate of testchefserver.example.com could not be verified
Certificate issuer data: /C=US/ST=WA/L=Seattle/O=YouCorp/OU=Operations/CN=testchefserver.example.com/emailAddress=you@example.com

Configuration Info:

OpenSSL Configuration:
* Version: OpenSSL 1.0.1m 19 Mar 2015
* Certificate file: /opt/chef/embedded/ssl/cert.pem
* Certificate directory: /opt/chef/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/root/.chef/trusted_certs"

TO FIX THIS ERROR:

If the server you are connecting to uses a self-signed certificate, you must
configure chef to trust that server's certificate.

By default, the certificate is stored in the following location on the host
where your chef-server runs:

  /var/opt/chef-server/nginx/ca/SERVER_HOSTNAME.crt

Copy that file to you trusted_certs_dir (currently: /root/.chef/trusted_certs)
using SSH/SCP or some other secure method, then re-run this command to confirm
that the server's certificate is now trusted.

knife ssl check
Connecting to host testchefserver.example.com:443
Successfully verified certificates from `testchefserver.example.com'

knife client list

chef-validator
chef-webui
 knife user list
admin
jojan
root

Node configuration

iptables -L

 rpm -ivh chef-11.18.12-1.el6.x86_64.rpm

warning: chef-11.18.12-1.el6.x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 83ef826a: NOKEY
Preparing...                ########################################### [100%]
   1:chef                   ########################################### [100%]
Thank you for installing Chef!

copy chef-validator.pem from chef server path /etc/chef-server to /etc/chef in Node server

knife ssl fetch -s https://testchefserver.example.com

WARNING: No knife configuration file found
WARNING: Certificates from testchefserver.example.com will be fetched and placed in your trusted_cert
directory ().

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

ERROR: TypeError: can't convert nil into String

 knife ssl check
Connecting to host testchefserver.example.com:443
Successfully verified certificates from `testchefserver.example.com'
[root@testchefnode .chef]# knife ssl fetch -s https://testchefserver.example.com
WARNING: Certificates from testchefserver.example.com will be fetched and placed in your trusted_cert
directory (/root/.chef/trusted_certs).

Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.

Adding certificate for testchefserver.example.com in /root/.chef/trusted_certs/testchefserver_example_com.crt

knife ssl check -s https://testchefserver.example.com

Connecting to host testchefserver.example.com:443
Successfully verified certificates from `testchefserver.example.com'

cat /etc/chef/client.rb
log_level :info
log_location STDOUT
chef_server_url "https://testchefserver.example.com:443"
trusted_certs_dir "/root/.chef/trusted_certs"

chef-client -S https://testchefserver.example.com -K /etc/chef/chef-validator.pem

creating a recipe on workstation and upload to server

knife cookbook create motd
** Creating cookbook motd
** Creating README for cookbook: motd
** Creating CHANGELOG for cookbook: motd
** Creating metadata for cookbook: motd

cd /var/chef/cookbooks/motd

ls -l
attributes
CHANGELOG.md
definitions
files
libraries
metadata.rb
providers
README.md
recipes
resources
templates

cd recipes/
vi default.rb

file '/etc/motd' do
        content 'Welcome to chef'
end

Before uploading to server make sure there is no syntax error

knife cookbook test motd
checking motd
Running syntax check on motd
Validating ruby files
Validating templates
Validating ruby files
Validating templates

upload to server,

knife cookbook upload motd
Uploading motd         [0.1.0]
Uploaded 1 cookbook.

to chk from server

knife cookbook list
motd   0.1.0

to chek from dashboard,
https://testchefserver.example.com/cookbooks

Add cookbook to a node,

go to https://testchefserver.example.com/nodes and edit
drag and drop Available Recipes moted to Run List,

chef-client
[2015-09-07T11:17:30+05:30] INFO: Forking chef instance to converge...
[2015-09-07T11:17:30+05:30] WARN:
........................................
- update content in file /etc/motd from e3b0c4 to cad802
    --- /etc/motd       2010-01-12 18:58:22.000000000 +0530
    +++ /tmp/.motd20150907-26207-mb0jjs 2015-09-07 11:17:32.622947241 +0530
    @@ -1 +1,2 @@
    +Welcome to chef
    - restore selinux security context
[2015-09-07T11:17:32+05:30] INFO: Chef Run complete in 0.661581977 seconds

Running handlers:
[2015-09-07T11:17:32+05:30] INFO: Running report handlers
Running handlers complete
[2015-09-07T11:17:32+05:30] INFO: Report handlers complete
Chef Client finished, 1/1 resources updated in 2.350334219 seconds

now cat it in node server,

cat /etc/motd
Welcome to chef

Ref:- https://www.youtube.com/watch?v=egvEPsVMfK0

Disable FirewallD and use iptables in RHEL 7 and CentOS 7

If you want to use iptables on CentOS 7 and RHEL 7 instead of firewallD Please follow,

systemctl mask firewalld

systemctl stop firewalld

yum -y install iptables-services

systemctl enable iptables

Ref: http://www.tejasbarot.com/2014/08/02/rhel-7-centos-7-disable-firewalld-and-use-iptables/#axzz3keP3fIkw

FirewallD

FirewallD provides a dynamically managed firewall with support for network/firewall zones to define the trust level of network connections or interfaces. It has support for IPv4, IPv6 firewall settings and for ethernet bridges and has a separation of runtime and permanent configuration options. It also supports an interface for services or applications to add firewall rules directly.

Features
D-Bus API
Timed firewall rules
Rich Language for specific firewall rules
IPv4 and IPv6 NAT support
Lockdown: Whitelisting of applications that may modify the firewall
Support for iptables, ip6tables, ebtables firewall backends
Automatic loading of Linux kernel modules
Integration with Puppet

Who’s using it?
FirewallD is used in the following Linux distributions as the default firewall management tool:

RHEL 7
Fedora 18 and newer

Applications and libraries which support FirewallD as a firewall management tool include:

NetworkManager
libvirt
docker 1.7

Ref: http://www.firewalld.org/

Testing SMTP Server from the windows command line

C:\Users\Administrator>telnet relay.mx.testserver.com 25

220 mb1relay1.mx.testserver.com ESMTP
helo test
250 mb1relay1.mx.testserver.com
mail from: usename1@mail.com
250 sender <usename1@mail.com> ok
rcpt to: usename2@mail.com
250 recipient <usename2@mail.com> ok
data
354 go ahead
Hi How are you
.
250 ok:  Message 258249 accepted
quit
221 mb1relay1.mx.testserver.com


Connection to host lost.

Ref :- https://www.youtube.com/watch?v=lfYtz3uRPYc

Nullifying a log file

Clear a file using /dev/null 

/dev/null is often referred to a black hole in Linux based systems. It discards all the data written to it and sends EOF (End of File) character to any process reading data from it. With this logic, we can clear the contents of a file.

cp /dev/null logfile

cat /dev/null > logfile

> logfile

dd if=/dev/null of=logfile

Clear a file using truncate

truncate logfile --size 0

strace

strace is a useful diagnostic, instructional, and debugging tool.

Strace monitors the system calls and signals of a specific program. It is helpful when you do not have the source code and would like to debug the execution of a program. strace provides you the execution sequence of a binary from start to end.

strace shows you how data is passed between the program and the kernel. With no options, strace prints a line for each system call. It shows the call name, given arguments, return value, and any generated error messages. A signal is printed with both its signal symbol and a descriptive string. As it shows the data transfer between user and kernel-space, strace is very useful as both a diagnostic utility for system administrators and a debugging tool for programmers. By default, the output is written to standard error.

Trace the Execution of an Executable
strace ls

Trace a Specific System Calls in an Executable Using Option -e
strace -e open ls

Save the Trace Execution to a File Using Option -o
strace -o output.txt ls

Execute Strace on a Running Linux Process Using Option -p
ps -C firefox-bin

Print Timestamp for Each Trace Output Line Using Option -t
strace -t -e open ls /home

Print Relative Time for System Calls Using Option -r
strace -r ls

Generate Statistics Report of System Calls Using Option -c
strace -c ls /home

eg :-
strace -p 3107
strace -d -p 3107

To print instruction pointer at the time of system call
strace -i -p 3111

To print time stamps of the system call
strace -t -p 3111

Options

-a n
Align the return values in column n. The default is 40.

-c
Count system calls, errors, signals, and time and provide a summary report when the program has ended.

-d
Debug mode. Print debugging information for strace on stderr.

-e [keyword=] [!] values
Pass an expression to strace to limit the types of calls or signals that are traced or to change how they are displayed. If no keyword is given, trace is assumed. The values can be given as a comma-separated list. Preceding the list with an exclamation point (!) negates the list. The special values all and none are valid, as are the values listed with the following keywords.

abbrev=names Abbreviate output from large structures for system calls listed in names. read=descriptors Print all data read from the given file descriptors. signal=symbols Trace the listed signal symbols (for example, signal=SIGIO,SIGHUP).

trace=sets
sets may be a list of system call names or one of the following:

file
Calls that take a filename as an argument.

ipc
Interprocess communication.

network
Network-related.

process
Process management.

signal
Signal-related.

raw=names
Print arguments for the given system calls in hexadecimal.

verbose=names
Unabbreviate structures for the given system calls. Default is none.

write=descriptors
Print all data written to the given file descriptors.

-f
Trace forked processes.

-ff
Write system calls for forked processes to separate files named filename.pid when using the -o option.

-h
Print help and exit.

-i
Print the current instruction pointer with each system call.

-o filename
Write output to filename instead of stderr. If filename starts with the pipe symbol |, treat the rest of the name as a command to which output should be piped.

-O n
Override strace's built-in timing estimates, and just subtract n microseconds from the timing of each system call to adjust for the time it takes to measure the call.

-p pid
Attach to the given process ID and begin tracking. strace can track more than one process if more than one option -p is given.

Type Ctrl-C to end the trace.

-q
Quiet mode. Suppress attach and detach messages from strace.

-r
Relative timestamp. Print time in microseconds between system calls.

-s n
Print only the first n characters of a string. Default value is 32.

-S value
Sort output of -c option by the given value. value may be calls, name, time, or nothing. Default is time.

-T
Print time spent in each system call.

-t
Print time of day on each line of output.

-tt
Print time of day with microseconds on each line of output.

-ttt
Print timestamp on each line as the number of seconds and microseconds since the Epoch.

-u username
Run command as username. Needed when tracing setuid and setgid programs.

-V
Print version and exit.

-v
Verbose. Do not abbreviate structure information.

-x
Print all non-ASCII strings in hexadecimal.

-xx
Print all strings in hexadecimal.

Ref:-
http://www.thegeekstuff.com/2011/11/strace-examples/
http://www.linuxdevcenter.com/cmd/cmd.csp?path=s/strace
http://chadfowler.com/blog/2014/01/26/the-magic-of-strace/

Fields of an inode

Inode is a data structure used to represent a filesystem object, which can be one of various things including a file or a directory. Each inode stores the attributes and disk block location(s) of the filesystem object's data. Filesystem object attributes may include manipulation metadata (e.g. change, access, modify time), as well as owner and permission data (e.g. group-id, user-id, permissions).

ls -il
total 52
185033 -rw-------. 1 root root   979 Apr 25 23:23 anaconda-ks.cfg
393286 drwxr-xr-x. 2 root root  4096 Apr 29 09:09 Desktop
393290 drwxr-xr-x. 2 root root  4096 Apr 29 09:09 Documents
393287 drwxr-xr-x. 2 root root  4096 Apr 29 09:09 Downloads
393291 drwxr-xr-x. 2 root root  4096 Apr 29 09:09 Music
393292 drwxr-xr-x. 2 root root  4096 Apr 29 09:09 Pictures
185050 -rw-r--r--. 1 root root 11955 Apr 23  2013 post-install
185053 -rw-r--r--. 1 root root   552 Apr 23  2013 post-install.log
393289 drwxr-xr-x. 2 root root  4096 Apr 29 09:09 Public
393288 drwxr-xr-x. 2 root root  4096 Apr 29 09:09 Templates
393293 drwxr-xr-x. 2 root root  4096 Apr 29 09:09 Videos

File attributes in particular,

The size of the file in Kilo bytes

Device ID

User ID of the file

Group ID of the file

The file mode that determines the file type and how the owner, group, and others (world) can access the file
Additional system and user flags to further protect the file (note: this can be used limit the files use and modification)

Timestamps telling when the inode itself was last change (ctime, changing time), the file content was last modified (mtime or modification time), and when the file was last accessed (atime or access time)

A link counter that lists how many hard links point to the inode
Pointers to the disk blocks that store the file’s contents (more on that later)

Ref:-
http://en.wikipedia.org/wiki/Inode
http://www.linux-mag.com/id/8658/
http://teaching.idallen.com/dat2330/04f/notes/links_and_inodes.html

Nagios Client configuration

Nagios Client configuration:-

Windows
------------
http://nsclient.org/nscp/downloads
install NSCP agent and give nagios Server IP

In nagios server,

vi /usr/local/nagios/etc/objects/myclientserver.cfg
vi /usr/local/nagios/etc/nagios.cfg

Add entry for myclientserver.cfg
/usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
/etc/init.d/nagios restart

Linux
------------
Download the required packages example,
nrpe-2.14.tar.gz
nagios-plugins-2.0.tar.gz

tar -xzvf nagios-plugins-2.0.tar.gz
tar -xzvf nrpe-2.14.tar.gz

yum install gcc
yum install openssl*
ldconfig

cd nagios-plugins-2.0
 ./configure
 make
 make install
 useradd nagios
 chown -R nagios:nagios /usr/local/nagios

cd ../nrpe-2.14
 ./configure
 make all
 make install-plugin
 make install-daemon
 yum install xinetd
 make install-daemon-config
 make install-xinetd

vi /etc/xinetd.d/nrpe
only_from       = 127.0.0.1 10.11.22.22

vi /etc/services
nrpe            5666/tcp                # nrpe

/etc/init.d/xinetd restart
 netstat -an | grep -i 5666

 yum install telnet
 telnet 10.11.22.22 5666

 ifconfig
 yum install sysstat

 cd /usr/local/nagios/
 mv /home/agoviku/libexec.tar.gz .
 mv /home/agoviku/md5s .
 chown -R nagios.nagios /usr/local/nagios
 tar -xzvf libexec.tar.gz

scp -r /usr/local/nagios/libexec usename@10.11.11.59:/tmp

cp -r /tmp/libexec /usr/local/nagios/
chown -R nagios:nagios /usr/local/nagios/
chmod -R 755 /usr/local/nagios/libexec

For Web servers,
vi /usr/local/nagios/etc/nrpe.cfg
command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
command[check_disk1]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /
command[check_disk2]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /data
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 400 -c 450
command[check_sendmail]=/usr/local/nagios/libexec/check_tcp -H localhost -p 25
command[check_cpu]=/usr/local/nagios/libexec/check_cpu.sh -w 60 -c 90
command[check_rofs1]=/usr/local/nagios/libexec/check_rofs.sh /
command[check_rofs2]=/usr/local/nagios/libexec/check_rofs.sh /data
command[check_md5]=/usr/local/nagios/libexec/check_md5.sh /etc/httpd/conf.d/vhost.conf
command[check_crond]=/usr/local/nagios/libexec/check_procs -c 1:20 -C crond
command[check_apache]=/usr/local/nagios/libexec/check_apache.sh -w 100 -c 200
command[check_apache2]=/usr/local/nagios/libexec/check_apache2.sh -w 100 -c 200

For DB servers,
vi /usr/local/nagios/etc/nrpe.cfg
command[check_users]=/usr/local/nagios/libexec/check_users -w 5 -c 10
command[check_load]=/usr/local/nagios/libexec/check_load -w 15,10,5 -c 30,25,20
command[check_disk]=/usr/local/nagios/libexec/check_disk -w 20% -c 10% -p /data
command[check_zombie_procs]=/usr/local/nagios/libexec/check_procs -w 5 -c 10 -s Z
command[check_total_procs]=/usr/local/nagios/libexec/check_procs -w 250 -c 300
command[check_cpu]=/usr/local/nagios/libexec/check_cpu.sh -w 60 -c 90
command[check_md5]=/usr/local/nagios/libexec/check_md5.sh /etc/my.cnf
command[check_mysql]=/usr/local/nagios/libexec/check_mysql -u nagiosusr -p 'passwrod' -s /data/var/lib/mysql/mysql.sock
command[check_mysql2]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode threads-connected   mysql2
command[check_mysql3]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode  threadcache-hitrate  mysql3
command[check_mysql4]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode threads-created  mysql4
command[check_mysql5]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode threads-running mysql5
command[check_mysql6]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode threads-cached  mysql6
command[check_mysql7]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode connects-aborted  mysql7
command[check_mysql8]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode clients-aborted  mysql8
command[check_mysql9]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode  qcache-hitrate --lookback 1800 --warning 50 --critical 20  mysql9
command[check_mysql10]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode  qcache-lowmem-prunes mysql10
command[check_mysql11]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode  keycache-hitrate --warning 50: --critical 20:  mysql11
command[check_mysql12]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode  bufferpool-hitrate  mysql12
command[check_mysql13]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode   bufferpool-wait-free mysql13
command[check_mysql14]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode   log-waits  mysql14
command[check_mysql15]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode   tablecache-hitrate mysql15
command[check_mysql16]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode   table-lock-contention mysql16
command[check_mysql17]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode  index-usage --warning 30: --critical 10: mysql17
command[check_mysql18]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode  tmp-disk-tables mysql18
command[check_mysql19]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode   open-files  mysql19
command[check_mysql20]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode   slow-queries  mysql20
command[check_mysql21]=/usr/local/nagios/libexec/check_mysql_health --hostname localhost --username nagiosusr --password 'passwrod' --mode   long-running-procs mysql21

In nagios server,
vi /usr/local/nagios/etc/nagios.cfg
 /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfg
/etc/init.d/nagios restart

install check_mysql_health in /usr/local/nagios/libexec

yum -y install cpan DBI*
cpan
install Time::HiRes
exit
yum -y install DBI*
cpan
install DBD::mysql
exit
ln -s /data/var/lib/mysql/mysql.sock /var/lib/mysql/mysql.sock

Ref :- https://labs.consol.de/nagios/check_mysql_health/


Few Known errors :-
-------------------------------------------------------------
/usr/local/nagios/libexec/check_nrpe -H 10.111.111.33 -p 5666
CHECK_NRPE: Error - Could not complete SSL handshake.

vi /etc/xinetd.d/nrpe
only_from       = 127.0.0.1 10.11.22.22

service xinetd restart

/usr/local/nagios/libexec/check_nrpe -H 10.33.66.10 -p 5666
NRPE v2.14

Ref: http://assets.nagios.com/downloads/nagiosxi/docs/NRPE-Troubleshooting-and-Common-Solutions.pdf
-------------------------------------------------------------
File Integrity
CRITICAL - file does not exist!(Nagios)

ll /etc/httpd/conf.d/vhost.conf
ls: cannot access /etc/httpd/conf.d/vhost.conf: No such file or directory
touch /etc/httpd/conf.d/vhost.conf

/usr/local/nagios/libexec/check_md5.sh /etc/httpd/conf.d/vhost.conf
OK
-------------------------------------------------------------
mysql Access denied for user 'monitor'@'localhost' (using password: YES)

grant all  on *.* to 'nagiosusr'@'%' identified by 'passwrod';
FLUSH PRIVILEGES;
-------------------------------------------------------------------------

Data Compression Using mod_deflate

The mod_deflate module provides the DEFLATE output filter that allows output from your server to be compressed before being sent to the client over the network. Currently mod_deflate is using with newer version of Apache. mod_deflate is the replacement of mod_gzip which was used with older verion of Apache.

Enabling Compression

By default mod_deflate modules are enabled in Apache. To make sure check following line in Apache configuration file.

LoadModule deflate_module modules/mod_deflate.so

Enable mod_deflate by editing apache conf(vhost.conf) file for particular website,

<Directory "/path/">
       <IfModule mod_mime.c>
                AddType application/x-javascript .js
                AddType text/css .css
        </IfModule>
        <IfModule mod_deflate.c>
                AddOutputFilterByType DEFLATE text/css application/x-javascript text/x-component text/html text/plain text/xml application/javascript
        </IfModule>
        Header append Vary User-Agent env=!dont-vary
    </Directory>

OR

Enable mod_deflate by editing apache conf(httpd.conf) file,

Add the following lines to configure mod_deflate in your apache configuration file.

 <IfModule mod_deflate.c>
  # compress text, html, javascript, css, xml:
  AddOutputFilterByType DEFLATE text/plain
  AddOutputFilterByType DEFLATE text/html
  AddOutputFilterByType DEFLATE text/xml
  AddOutputFilterByType DEFLATE text/css
  AddOutputFilterByType DEFLATE application/xml
  AddOutputFilterByType DEFLATE application/xhtml+xml
  AddOutputFilterByType DEFLATE application/rss+xml
  AddOutputFilterByType DEFLATE application/javascript
  AddOutputFilterByType DEFLATE application/x-javascript
  AddOutputFilterByType DEFLATE image/x-icon
</IfModule>

To set the no-gzip note for a particular browser, so that no compression will be performed.

<Directory /var/www/html/>
    <IfModule mod_mime.c>
        AddType application/x-javascript .js
        AddType text/css .css
    </IfModule>
    <IfModule mod_deflate.c>
        AddOutputFilterByType DEFLATE text/css application/x-javascript text/x-component text/html text/plain text/xml application/javascript
        <IfModule mod_setenvif.c>
            BrowserMatch ^Mozilla/4 gzip-only-text/html
            BrowserMatch ^Mozilla/4\.0[678] no-gzip
            BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
        </IfModule>
    </IfModule>
    Header append Vary User-Agent env=!dont-vary
</Directory>

Restart the apache service to complete the process

# service httpd restart

Testing Compression

After configuring compression in Apache we can see that content is being send by web server is compressed by mod_defalte by using below URL.
http://www.whatsmyip.org/http-compression-test/

Ref : http://tecadmin.net/how-to-enable-gzip-compression-on-apache/

Troubleshooting high loads on Linux servers

The load average represents the average number of processes that have to wait for CPU time during the last 1, 5 or 15 minutes.

What causes high server loads?

Excessive usage of any of the following items can typically cause this issue:

CPU
memory (including swap)
disk I/O

How can I check these items?

That depends whether you want to review their current resource usage, or historical resource usage.

Historical resource usage can be viewed using the "sar" utility.

The stats are collected when sysstat runs from cron (/etc/cron.d/sysstat). If crond is not running, sysstat will not be able to collect historical statistics.

or example, if you wanted to view the load averages for your server from the 28rd of the month:
sar -q -f /var/log/sa/sa28

the current day status: sar -q
Current CPU usage: top c
Historical CPU usage : sar -p
Current memory usage: free -m
Historical memory usage: sar -r  (%memused and %swpused), sar -s (%swpused)
Current disk I/O usage: iostat -x 1 10
Historial disk I/O usage : sar -d
process list: ps auxwwwf
system’s virtual memory statistics: vmstat 5 10 (10 times at 5 second intervals)

There are various actions you can take to find the cause of your high server loads. Here is a partial list that will always be incomplete:

Check the MySQL process list using "mysqladmin processlist" (or just "mysqladmin pr" for short)
Check the MySQL process list using mytop
tail your logs! Listening to what your server says is very important. Is your server being brute forced?
Run dmesg and check for possible hardware issues
Use netstat to view the connections to your server

Here are some logs to check:

syslogs: /var/log/messages, /var/log/secure
SMTP logs: /var/log/exim_mainlog, /var/log/exim_rejectlog, /var/log/exim_paniclog
POP3/IMAP logs: /var/log/maillog
Apache logs: /usr/local/apache/logs/access_log, /usr/local/apache/logs/error_log, /usr/local/apache/logs/suexec_log, /usr/local/apache/logs/suphp_log
Website logs: /usr/local/apache/domlogs/ (use this to find sites with traffic in the last 60 seconds: find -maxdepth 1 -type f -mmin -1 | egrep -v 'offset|_log$')
cron logs: /var/log/cron

Ref :
http://forums.cpanel.net/f5/troubleshooting-high-server-loads-linux-servers-319352.html
http://www.linuxjournal.com/magazine/hack-and-linux-troubleshooting-part-i-high-load?page=0,0

A foreign key constraint fails in MySQL

Scenario:

drop database databasename;
ERROR 1217 (23000): Cannot delete or update a parent row: a foreign key constraint fails

Resolution:-

mysql> SET FOREIGN_KEY_CHECKS=0;
Query OK, 0 rows affected (0.00 sec)

mysql>  drop database databasename;
Query OK, 2 rows affected (0.06 sec)

mysql> SET FOREIGN_KEY_CHECKS=1;
Query OK, 0 rows affected (0.00 sec)

Ref : - http://stackoverflow.com/questions/3334619/cannot-delete-or-update-a-parent-row-a-foreign-key-constraint-fails

Could not chdir to home directory : Permission denied

Scenario:
While do ssh to the server,

login as: usrteam
eduteam@67.228.160.50's password:
Last login: Thu Dec  4 09:59:52 2014 from 14.141.35.194
Could not chdir to home directory /home/usrteam: Permission denied
-bash: /home/eduteam/.bash_profile: Permission denied
-bash-3.2$ pwd
/

Resolution :-

chown -R usrteam /home/usrteam
You have new mail in /var/spool/mail/root

Couchbase Server

Couchbase Server is the world’s most complete, scalable, and highest performing NoSQL distributed database.

yum install -y pkgconfig

yum install openssl098e

wget http://packages.couchbase.com/releases/3.0.1/couchbase-server-enterprise-3.0.1-centos6.x86_64.rpm

rpm -ivh couchbase-server-enterprise-3.0.1-centos6.x86_64.rpm
Preparing...                ########################################### [100%]
Warning: Transparent hugepages may be used. To disable the usage
of transparent hugepages, set the kernel settings at runtime with
echo never > /sys/kernel/mm/transparent_hugepage/enabled
Warning: Transparent hugepages may be used. To disable the usage
of transparent hugepages, set the kernel settings at runtime with
echo never > /sys/kernel/mm/redhat_transparent_hugepage/enabled
Warning: Swappiness is not 0.
You can set the swappiness at runtime with
sysctl vm.swappiness=0
Minimum RAM required  : 4 GB
System RAM configured : 3.74 GB

Minimum number of processors required : 4 cores
Number of processors on the system    : 2 cores

   1:couchbase-server       ########################################### [100%]
Starting couchbase-server[  OK  ]

You have successfully installed Couchbase Server.
Please browse to http://mb1udarmiweb02.pearsontc.com:8091/ to configure your server.
Please refer to http://couchbase.com for additional resources.

Please note that you have to update your firewall configuration to
allow connections to the following ports: 11211, 11210, 11209, 4369,
8091, 8092, 18091, 18092, 11214, 11215 and from 21100 to 21299.

By using this software you agree to the End User License Agreement.
See /opt/couchbase/LICENSE.txt.

/etc/init.d/couchbase-server status
couchbase-server is running

Ref:
http://www.couchbase.com/nosql-databases/about-couchbase-server#ElasticScalability
http://docs.couchbase.com/couchbase-manual-2.5/cb-install/

Update time zone in Linux

date
Thu Nov 27 19:40:33 EST 2014

ls -l /etc/localtime
lrwxrwxrwx 1 root root 39 Nov 27 19:40 /etc/localtime -> /usr/share/zoneinfo/Australia/Melbourne

unlink /etc/localtime

ln -s /usr/share/zoneinfo/Asia/Kolkata /etc/localtime

ls -l /etc/localtime
lrwxrwxrwx 1 root root 32 Nov 27 14:08 /etc/localtime -> /usr/share/zoneinfo/Asia/Kolkata

date
Thu Nov 27 14:09:15 IST 2014

Ref:- http://www.linuxnix.com/2014/11/linuxunix-how-update-timezone-in-a-system.html?utm_source=feedburner&utm_medium=email&utm_campaign=Feed%3A+TheLinuxJuggernaut+%28The+Linux+Juggernaut%29

PHP Warning: PHP Startup: Unable to load dynamic library

Scenario :-

php -v

PHP Warning:  PHP Startup: Unable to load dynamic library '/usr/lib64/php/modules/module.so' - /usr/lib64/php/modules/module.so: cannot open shared object file: No such file or directory in Unknown on line 0
PHP 5.3.3 (cli) (built: Sep 10 2014 05:27:26)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies

cat /etc/php.d/mcrypt.ini
; Enable mcrypt extension module
extension=module.so

but module.so not exist in /usr/lib64/php/modules/

Resolution :-
uninstall the php-mcrypt and reinstall the suitable rpm

rpm -e php-mcrypt-5.3.3-1.el6.rf.x86_64
wget ftp://ftp.pbone.net/mirror/download.fedora.redhat.com/pub/fedora/epel/6/x86_64/php-mcrypt-5.3.3-3.el6.x86_64.rpm
rpm -ivh php-mcrypt-5.3.3-3.el6.x86_64.rpm

php -v
PHP 5.3.3 (cli) (built: Sep 10 2014 05:27:26)
Copyright (c) 1997-2010 The PHP Group
Zend Engine v2.3.0, Copyright (c) 1998-2010 Zend Technologies

cat /etc/php.d/mcrypt.ini
; Enable mcrypt extension module
extension=mcrypt.so

now mcrypt.so exist in /usr/lib64/php/modules/mcrypt.so

Monitor MySQL performance with innotop

Innotop is an excellent command line program, similar to ‘top command‘ to monitor local and remote MySQL servers running under InnoDB engine.

yum install innotop

innotop -u root -p 'mysqlpassword'

Innotop Help
Press “?” to get the summary of command line options and usage.

Switch to a different mode:
   A  Dashboard         I  InnoDB I/O Info     Q  Query List
   B  InnoDB Buffers    K  InnoDB Lock Waits   R  InnoDB Row Ops
   C  Command Summary   L  Locks               S  Variables & Status
   D  InnoDB Deadlocks  M  Replication Status  T  InnoDB Txns
   F  InnoDB FK Err     O  Open Tables         U  User Statistics

Actions:
   d  Change refresh interval        p  Pause innotop
   k  Kill a query's connection      q  Quit innotop
   n  Switch to the next connection  x  Kill a query

Other:
 TAB  Switch to the next server group   /  Quickly filter what you see
   !  Show license and warranty         =  Toggle aggregation
   #  Select/create server groups       @  Select/create server connections
   $  Edit configuration settings       \  Clear quick-filters
Press any key to continue

Non-Interactively

innotop -u root -p 'mysqlpassword' --count 5 -d 1 -n

Monitor Remote Database

innotop -u username -p password -h hostname

Ref : -
http://www.tecmint.com/install-innotop-to-monitor-mysql-server-performance/
http://linux.die.net/man/1/innotop

SeaLion Monitoring in Linux

Install and configure SeaLion

wget https://s3.amazonaws.com/sealion.com/3.2.3/sealion-agent-3.2.3-noarch.tar.gz
tar -xzvf sealion-agent-3.2.3-noarch.tar.gz
cd sealion-agent

sudo ./install.sh -o 299ade98-ab92-44bd-a9e6-3cefd1a9aeda

dependency : SeaLion agent requires python version 2.6 or above

Ref : -
https://www.youtube.com/watch?v=fLqVQd1SMmY
https://sealion.com

Bash Code Injection Vulnerability

rpm -qa | grep bash

bash-3.2-32.el5

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
vulnerable
this is a test
You have new mail in /var/spool/mail/root

yum upgrade bash

env x='() { :;}; echo vulnerable' bash -c "echo this is a test"
bash: warning: x: ignoring function definition attempt
bash: error importing function definition for `x'

rpm -qa | grep bash
bash-3.2-33.el5.1

Ref :- https://access.redhat.com/articles/1200223

Install mod_ssl on Apache2 - Ubuntu

Steps to install mod_ssl on Apache2 - Ubuntu

apt-get install mod_ssl

dpkg -S mod_ssl.so
apache2.2-bin: /usr/lib/apache2/modules/mod_ssl.so

a2enmod ssl
Module ssl already enabled

To make sure that the SSL module was loaded properly :

apache2ctl -t -D DUMP_MODULES | grep ssl
[Thu Sep 25 11:28:16 2014] [warn] NameVirtualHost *:80 has no VirtualHosts
Syntax OK
ssl_module (shared)

Mutual SSL authentication in Ubuntu

Two-way SSL using CA certificates

 cd /root

 mkdir CA

 cd CA

 mkdir newcerts private

vi openssl.cnf

#
# OpenSSL configuration file.
#
# Establish working directory.
dir = .
ts = 1024 # Size of keys
default_keyfile = key.pem # name of generated keys
default_md = md5 # message digest algorithm
string_mask = nombstr # permitted characters
distinguished_name = req_distinguished_name
req_extensions = v3_req

[ ca ]

default_ca = CA_default

[ CA_default ]

serial = $dir/serial
database = $dir/index.txt
new_certs_dir = $dir/newcerts
certificate = $dir/cacert.pem
private_key = $dir/private/cakey.pem
default_days = 365
default_md = md5
preserve = no
email_in_dn = no
nameopt = default_ca
certopt = default_ca
policy = policy_match

[ policy_match ]

countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ req ]

default_md = sha1
distinguished_name = req_distinguished_name

[ req_distinguished_name ]

countryName = Country
countryName_default = IN
countryName_min = 2
countryName_max = 2
localityName = Locality
stateOrProvinceName_default = Karnataka
localityName_default = Bangalore
organizationName = Organization
organizationName_default = edurite
commonName = Common Name
commonName_max = 64

[ certauth ]

subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
basicConstraints = CA:true

[ server ]

basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
nsCertType = server

[ client ]

basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = clientAuth
nsCertType = client

[ v3_req ]

basicConstraints = CA:FALSE
subjectKeyIdentifier = hash

Generate self-signed certificate

 openssl req -config ./openssl.cnf -newkey rsa:2048 -nodes -keyform PEM -keyout ca.key -x509 -days 3650 -extensions certauth -outform PEM -out ca.cer

 openssl genrsa -out server.key 2048

 openssl req -config ./openssl.cnf -new -key server.key -out server.req

 openssl x509 -req -in server.req -CA ca.cer -CAkey ca.key -set_serial 100 -extfile openssl.cnf -extensions server -days 365 -outform PEM -out server.cer

 rm server.req

 openssl genrsa -out client.key 2048

openssl req -config ./openssl.cnf -new -key client.key -out client.req

 openssl x509 -req -in client.req -CA ca.cer -CAkey ca.key -set_serial 101 -extfile openssl.cnf -extensions client -days 365 -outform PEM -out client.cer

 openssl pkcs12 -export -inkey client.key -in client.cer -out client.p12

 rm client.key client.cer client.req

vi /etc/apache2/sites-available/default

<VirtualHost *:443>
        ServerAdmin webmaster@localhost
        ServerName 10.98.33.136:443

        DocumentRoot /var/www
        <Directory />
                Options FollowSymLinks
                AllowOverride None
        </Directory>
        <Directory /var/www/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
                Order allow,deny
                allow from all
        </Directory>

        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        <Directory "/usr/lib/cgi-bin">
                AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
                Order allow,deny
                Allow from all
        </Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog ${APACHE_LOG_DIR}/access.log combined

    Alias /doc/ "/usr/share/doc/"
    <Directory "/usr/share/doc/">
        Options Indexes MultiViews FollowSymLinks
        AllowOverride None
        Order deny,allow
        Deny from all
        Allow from 127.0.0.0/255.0.0.0 ::1/128
    </Directory>
SSLEngine on
LogLevel warn
SSLProtocol all -SSLv2
SSLCipherSuite HIGH:MEDIUM
SSLVerifyClient require
SSLVerifyDepth 10
SSLCertificateFile /etc/apache2/ssl/apache.crt
SSLCertificateKeyFile /etc/apache2/ssl/apache.key
#SSLCACertificateFile /etc/apache2/ssl/ca.cer
</VirtualHost>

/etc/init.d/apache2 restart

./OpenSSL_Client rname@some.com 365 passwd /var/www/html/CERT

Now Import the P12 file to the browser.

Ref :- http://www.flatmtn.com/article/setting-openssl-create-certificates

Two-way SSL authentication

Two-way SSL authentication or mutual SSL authentication or client
authentication works by resolving its identity to SSL server with a use of
the client certificate.

All certificates will be issued by using OpenSSL application and openssl.cnf
configuration file.

 cd /root

 mkdir CA

 cd CA

 mkdir newcerts private

 vi /root/CA/openssl.cnf

#
# OpenSSL configuration file.
#
# Establish working directory.
dir = .
ts = 1024 # Size of keys
default_keyfile = key.pem # name of generated keys
default_md = md5 # message digest algorithm
string_mask = nombstr # permitted characters
distinguished_name = req_distinguished_name
req_extensions = v3_req

[ ca ]

default_ca = CA_default

[ CA_default ]

serial = $dir/serial
database = $dir/index.txt
new_certs_dir = $dir/newcerts
certificate = $dir/cacert.pem
private_key = $dir/private/cakey.pem
default_days = 365
default_md = md5
preserve = no
email_in_dn = no
nameopt = default_ca
certopt = default_ca
policy = policy_match

[ policy_match ]

countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional

[ req ]

default_md = sha1
distinguished_name = req_distinguished_name

[ req_distinguished_name ]

countryName = Country
countryName_default = IN
countryName_min = 2
countryName_max = 2
localityName = Locality
stateOrProvinceName_default = Karnataka
localityName_default = Bangalore
organizationName = Organization
organizationName_default = edurite
commonName = Common Name
commonName_max = 64

[ certauth ]

subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid:always,issuer:always
basicConstraints = CA:true

[ server ]

basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
nsCertType = server

[ client ]

basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment, dataEncipherment
extendedKeyUsage = clientAuth
nsCertType = client

[ v3_req ]

basicConstraints = CA:FALSE
subjectKeyIdentifier = hash

 echo '01' > serial
 touch index.txt

 to generate self-signed certificate

 openssl req -config ./openssl.cnf -newkey rsa:2048 -nodes -keyform PEM -keyout ca.key -x509 -days 3650 -extensions certauth -outform PEM -out ca.cer

 openssl genrsa -out server.key 2048

 openssl req -config ./openssl.cnf -new -key server.key -out server.req

 openssl x509 -req -in server.req -CA ca.cer -CAkey ca.key -set_serial 100 -extfile openssl.cnf -extensions server -days 365 -outform PEM -out server.cer

 rm server.req

 openssl genrsa -out client.key 2048

openssl req -config ./openssl.cnf -new -key client.key -out client.req

 openssl x509 -req -in client.req -CA ca.cer -CAkey ca.key -set_serial 101 -extfile openssl.cnf -extensions client -days 365 -outform PEM -out client.cer

 openssl pkcs12 -export -inkey client.key -in client.cer -out client.p12

 rm client.key client.cer client.req

vi /etc/httpd/conf.d/ssl.conf

LoadModule ssl_module modules/mod_ssl.so
AddType application/x-x509-ca-cert .crt
AddType application/x-pkcs7-crl .crl

Listen 443

<VirtualHost 172.16.0.14:443>
DocumentRoot "/var/www/SSL/digitally"
CheckSpelling on
DirectoryIndex opsindex.php
ServerName 172.16.0.14
ServerAdmin root@localhost
ErrorLog logs/ssl_error_log
TransferLog logs/ssl_access_log
LogLevel warn
SSLEngine on
SSLProtocol all -SSLv2
SSLCipherSuite HIGH:MEDIUM
SSLCertificateFile /etc/httpd/conf/ssl/server.cer
SSLCertificateKeyFile /etc/httpd/conf/ssl/server.key
SSLVerifyClient require
SSLVerifyDepth 10
SSLCACertificateFile /etc/httpd/conf/ssl/ca.cer
<Files ~ "\.(cgi|shtml|phtml|php3?)$">
SSLOptions +StdEnvVars
</Files>
SetEnvIf User-Agent ".*MSIE.*" \
nokeepalive ssl-unclean-shutdown \
downgrade-1.0 force-response-1.0
CustomLog logs/ssl_request_log \
"%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x \"%r\" %b"
</VirtualHost>

 httpd -S

 /etc/init.d/httpd/restart

 mkdir -p /var/www/html/CERTS

./OpenSSL_Client rname@some.com 365 passwd /var/www/html/CERTS

Now Import the P12 file to the browser.

Ref :- http://www.flatmtn.com/article/setting-openssl-create-certificates

Purge logs of MySQL

The PURGE BINARY LOGS statement deletes all the binary log files listed in the log index file prior to the specified log file name or date. BINARY and MASTER are synonyms. Deleted log files also are removed from the list recorded in the index file, so that the given log file becomes the first in the list.

Examples:-

mysql> SHOW BINARY LOGS;
+------------------+-----------+
| Log_name         | File_size |
+------------------+-----------+
| mysql-bin.000034 |  14785707 |
| mysql-bin.000035 |       143 |
| mysql-bin.000036 |       143 |
| mysql-bin.000037 |       120 |
+------------------+-----------+
4 rows in set (0.00 sec)

mysql> PURGE BINARY LOGS BEFORE '2014-09-10';

mysql> PURGE BINARY LOGS BEFORE '2013-04-22 09:55:22';

mysql>PURGE BINARY LOGS TO 'mysql-bin.000015';


ref : https://mariadb.com/kb/en/mariadb/documentation/sql-commands/administration-commands/sql-commands-purge-logs/