The Most Popular Configuration Files in Linux System

List of the most popular configuration files and brief description

Configuration File                  Description

/boot

/boot/grub                             GRUB bootloader file
/boot/grub/menu.list             boot menu
/boot/System.map                 Linux kernel map
/boot/vmlinuz                        Linux kernel

/etc

/etc/apache2/httpd.conf        Apache web server configuration file
                                             (Debian)
/etc/apt/sources.list               lists the sources from which the APT obtains
                                              packages(Debian & Ubuntu)
/etc/at.allow                          users allowed to use the at command to 
                                              schedule jobs
/etc/at.deny                           users forbidden to use the at command
/etc/bashrc                            functions and aliases for bash shell
                                             (Fedora)

/etc/bash.bashrc                    functions and aliases for bash shell
                                             (Debian, SUSE & Ubuntu)
/etc/cups/cupsd.conf            The CPUS scheduler
/etc/fonts/local.conf             font configuration file
/etc/fstab                              Information about the various file system and mount
/etc/group                             Information about groups

/etc/grub.conf                       GRUB bootloader
                                             (Fedora& SUSE)
/etc/hosts                              IP address and their hostnames
/etc/hosts.allow                    Hosts allowed to access internet service
/etc/hosss.deny                     hosts forbidden to access internet services
/etc/httpd/conf/httpd.conf    Apache web server configuration file
                                             (Fedora)

/etc/init.d                              Directory with scripts to start and stop various servers
/etc/init.d/rcS                        Linux initialization script
                                              (Debian, SUSE & Ubuntu)
/etc/inittab                             The init process that starts all the other processes
/etc/issue                               Distribution name and the version number
/etc/lilo.conf                          Linux Loader(LILO) configuration file

/etc/login.defs                       Default information for creating user accounts
/etc/modules.conf                 Configuration file for loadable kernel module
                                             (Debian)
/etc/mtab                               Information about the currently mounted file systems
/etc/passwd                           Information about all user accounts
/etc/profile                            Systemwide environment and startup file for the bash sell

/etc/profile.d                         Directory containing file that the /etc/profile script executes
/etc/rc.d/rc.sysinit                 Linux initialization script
                                              (Fedora)
/etc/shadow                           Secure file with encrypted passwords for all user accounts
/etc/shells                              List of all the shells on the system that the user can use
/etc/skel                                 Directory that holds .bash_logout, .bash_profile, .bashrc files

/etc/sysconfig                        Linux configuration files
                                              (Fedora & SUSE)
/etc/sysctl.conf                      Configuration file with kernel parameters
/etc/termcap                          Database of terminal capabilities and options
                                             (Fedora & SUSE)
/etc/udev                               Directory containing configuration files for udev
/etc/X11                                Directory with configuration files for X Window 
                                              System

/etc/X11/xorg.conf               Configuration file for X.Org X11 the X Window System
                                             (Fedora, Ubuntu & SUSE)
/etc/xinetd.conf                    Configuration for the xinetd daemon that starts internet 
                                             service on demand
/etc/yum.conf                       Configuration for the Yum package updater and installer
                                             (Fedora)

/var

/var/log/apache2                  Web-server access and error logs
                                            (Debian)
/var/log/cron                        Log file with messages from the cron process that runs 
                                            scheduled jobs
/var/log/boot.msg                File with boot messages(SUSE)
/var/log/dmesg                    File with boot messages
                                            (Debian, Fedora & Ubuntu)
/var/log/httpd                      Web-server access and error logs(Fedora)
/var/log/message                 System log


APT    - Advanced Packaging Tool
CPUS - Common Unix Printing System

Basic TCP/IP Network Configuration Files

Summary of basic TCP/IP configuration files

File                                        Description

/etc/hosts                           IP address and hostnames for local network

/etc/networks                    Names and IP address of networks

/etc/host.conf                    Translate host name in to IP address

/etc/resolv.conf                 IP address of name servers

/etc/hosts.allow                 Allow which system can access internet

/etc/hosts.deny                  Deny internet access to systems

/etc/nsswitch.conf            Translate hostnames into ip address

Common vi Commands

List of commonly used vi commands

Command              Does the following function

Insert Text

a                              Inserts text after the cursor

A                             Inserts text at the end of the current line

I                               Inserts text at the beginning of the current line

i                               Inserts text before the cursor

Delete Text

D                             Deletes up to the end of the current line

dd                           Deletes the current line

dG                          Deletes from the current line to the end of the file

dw                          Deletes the current word where the cursor presently resides

x                              Deletes the character on which the cursor rests

Change Text

C                             Changes up to the end of the current line

cc                            Changes the current line

J                              Joins the current line with the next one

rx                            Replaces the character under the cursor with x
                                (where x is any character)

Search Text       

/string                  Searches forward for a string

?string                  Searches backward for a string

Scroll Text

Ctrl+D                   Scrolls forward by half a screen

Ctrl+U                   Scrolls backward by half a creen

Ctrl+L                    Redraws/refresh the screen

Cut and Paste Text

yy                           Yanks(copies)current line to an unnamed buffer

P                             Puts the yanked line above the current line

p                             Puts the yanked line below the current line

Move Cursor     

h                              Moves one character to the left

j                               Moves one line down

k                              Moves one line up

L                              Moves to the end of the screen

l                               Moves one character to the right

w                             Moves to the beginning of the following word

b                              Moves to the beginning of the previous word

Colon commands

:!command           Executes a shell command

:q                            Quits the editor

:q!                           Quits without saving changes

:r filename             Reads the file and inserts it after the current line

:w filename           Writes a buffer to the file

:wq                         Saves changes and exits

More

u                             Undoes the last command

Esc                          Ends input mode and enters visual command mode

U                             Undoes resent changes to the current line

Learn more:- Vim Cheat Sheet

Top Level Directories in the Linux File System

Top Level Directories in the Linux

Directory        Contains

/                       Base of the files system.
/bin                  Executable programs that are part of the Linux Operating System.
/boot                Linux Kernel and other files that the LILO and GRUB needed.
/dev                 Special files that represent devices attached to the system.
/etc                  System configuration files and the initialization scripts.
/home              Home directories of all users.
/lib                   Library files stored in /sbin and /bin needed to start Linux.
/lost+found     Lost files. Every disk partition has a lost+found directory.
/media             Mounting file systems on removable media.
/mnt                 Temporarily mounted file systems.
/opt                  Storage for large application software packages.
/proc                Information about the process running in the Linux system.
/root                 Home directory for the root user.
/sbin                 Executable commands for administration tasks.
/srv                  Data for services offered by this system.
/sys                  Information about the devices as seen by the Linux kernel.
/tmp                 Temporary directory that any user can use.
/usr                  Subdirectories for many important programs.
/var                  Various system files.

Important /usr Subdirectories

 Subdirectory  Description

/usr/bin            Executable files including utility programs

/usr/games       Some old Linux games.
/usr/include      Header files for the C, C++,  X11 and Linux kernel.
/usr/lib             Libraries for C, C++, other database libraries and graphical toolkit.
/usr/local          Local files.
/usr/sbin           Administrative commands, such as for email and networking.
/usr/share         Shared data.
/usr/share/man Online manual.
/usr/src             Source code for the Linux kernel.

 Important /var subdirectories

Subdirectory  Description.

/var/cache        Storage area for cached data for applications.
/var/lib             Information relating to the current state of applications.
/var/lock          Locked files ensures that a resource is used by only one application.
/var/log            Log files organized into subdirectories.
/var/mail          User mailbox files.
/var/opt            Variable data for packages stored in the /opt directory
/var/run            Data describing the system since the time it was booted.
/var/spool         Data that's waiting for some kind of processing.
/var/tmp           Temporary files preserved between system reboots.
/var/yp             Network Information Services (NIS) database files.

Programmers want to know about Linux

These features make Linux a productive software development environment.

GNU C compiler (gcc)
GNU C++ compiler (g++)
GNU compiler for java (gcj)
GNU make utility
GNU debugger (gdb)
GNU profiling utility (gprof)
Subversion, Concurrent Versions System (CVS) and Revision Control System (RCS)
GNU emacs editor
Perl
Tool Command Language and its graphical toolkit (tcl/tk)
Python
Dynamically linked, shared Libraries

Well known GNU Packages

GNU was designed to be a replacement for Unix operating systems of the 1980s and used the POSIX standards as a guide. The following list is a small set of GNU packages.

Software Package            Description

autoconf                         Automatically configure source code packages
automake                       makes *.in files
bash                               shell
bc                                   interactive calculator
binutils                           binary files utilities
coreutils                         A combined package of Fileutils, Shellutils and Textutils

cpio                                copy to and from, a disc or file system
diff                                 compare files
ed                                   line oriented text editor
emacs                             extensible text editor
findutils                          find, locate and xargs utilities
finger                             find information about users

gawk                              awk programming
gcc                                 for C,C++, Objective-C and other
gdb                                debugger for C, C++ and FORTRAN
gdbm                              replacement for dbm and ndbm libraries
gettext                           writing multilingual programs
GIMP                              GNU image manipulation program

GNOME                           GUI
Gnumeric                        graphical spreadsheet
grep                                includes grep, egrep and fgrep
groff                               document formatting
gtk+                                GUI toolkit for X Window system
gzip                                compressing and decompressing utility

less                                 similar to more
libpng                             library for image files
make                              determine which pieces of a program need to be recompiled
patch                              generate the modified version
rcs                                  revision control system
sed                                 stream oriented version of ed text editor

tar                                  tape archiving program
textinfo                          for online info docs
time                               actual time that a process uses

All GNU packages
https://www.gnu.org/software/software.en.html

Reset Root Password On CentOS 7

Change the root password on CentOS 7 Servers

At the boot menu, press e to edit the existing kernel (Core).

scroll down to the list until you see the line ro ,change ro to rw and start into a bash shell. It should look like rw init=/sysroot/bin/sh.

Press Ctrl + X  to start into single user mode.

In the single user mode run the command,
chroot /sysroot

Run the commands to change the root password,
passwd root

After creating the password, run the commands below to update SELinux parameters
touch /.autorelabel

Exit and reboot your system. You should be able to sign on and use the system with the new password.

Monitoring Memory with Swap

A low amount of free memory is an indication that the system has many thing to  do; this may lead to disk swapping. A high use of swap space make a system go slow because disk access is many times slower than access.

cat /usr/local/src/memory.sh
#!/bin/bash
C=$(/usr/bin/free -m| head -2 | tail -n +2 | awk {'print $4'})
C=$(/usr/bin/free -m| head -4 | tail -n +4 | awk {'print $3'})
E=$(date)
printf "%s %s %s %s %s %s\n" "Mem: " "$C" "Swap: " "$D" "At" "$E"

crontab -l
*/5 * * * * /usr/local/src/memory.sh >> /var/log/memory.data

tail -5 /var/log/memory.data
Mem: 76080 Swap: 5274188 At Sat Oct  8 12:44:48 PDT 2016
Mem: 72128 Swap: 5274264 At Sat Oct  8 12:45:01 PDT 2016
Mem: 67780 Swap: 5274428 At Sat Oct  8 12:50:01 PDT 2016
Mem: 68720 Swap: 5274492 At Sat Oct  8 12:55:01 PDT 2016
Mem: 68472 Swap: 5274944 At Sat Oct  8 13:00:01 PDT 2016

Monitoring the TCP ESTABLISHED Connections

The following script tcpConnect.sh uses nestat to record the number of connection as well as the date and the time of the measurement.

cat /usr/local/src/tcpConnet.sh
#!/bin/bash
C=$(/bin/netstat -nt | tail -n +3 | grep ESTABLISHED | wc -l)
D=$(date)
printf "%s %s %s\n" "$C" "||" "$D"

crontab -l
*/5 * * * * /usr/local/src/tcpConnet.sh >> /var/log/tcpConnet.data

tail -5 /var/log/tcpConnet.data
25 || Sat Oct  8 11:10:01 PDT 2016
26 || Sat Oct  8 11:15:01 PDT 2016
23 ||  Sat Oct  8 11:20:01 PDT 2016
23 || Sat Oct  8 11:25:01 PDT 2016
23 || Sat Oct  8 11:30:01 PDT 2016

Monitoring Load Average

The easiest way to monitor the load average is by using the output o the uptime command and a small awk script that will run as a cron job to store your data into a text file.

The awk script is the following :

cat /usr/local/src/uptime.sh
#!/bin/bash
uptime | awk  {'print $9 $10 $11'} | awk -F, {'print $1 " " $2 " " $3'}

sh /usr/local/src/uptime.sh
0.68 0.86 1.48

crontab -l
*/5 * * * * /usr/local/src/uptime.sh >> /var/log/uptime.data

head -5 /var/log/uptime.data
0.61 1.40 1.96
0.47 1.33 1.92
0.49 0.94 1.68
0.64 0.80 1.41
0.45 0.61 1.16

       
The Linux system we used has only one CPU, so any load average greater than 1.00 indicates a performance issue.

The first number shows the load average during the last minute, the second during the last five minutes and the third during the last 15 minute,. The three values can show you if the load is increasing, decreasing or steady.

Shell script for disk usage

Script for disk usage

cat diskUsage.sh
#!/bin/bash
/bin/df -t ext4 -m | tail -l| awk {'print $3 " " $4 " " $5'}

sh diskUsage.sh
Used Available Use%
1248 16786 7%

Install Magento on Ubuntu 14.04

Prerequisites

apt-get install libcurl3 php5-curl php5-gd php5-mcrypt

Apache Virtual Host Settings

vi /etc/apache2/sites-enabled/magento.conf

<VirtualHost *:80>

        DocumentRoot /var/www/html/magento/

        <Directory /var/www/html/magento/>
                Options Indexes FollowSymLinks MultiViews
                AllowOverride All
        </Directory>

</VirtualHost>

sudo a2ensite magento.conf
service apache2 restart

PHP Settings

vi /etc/php5/apache2/php.ini

memory_limit = 512M

Create a MySQL Database and User

mysql -u root -p
CREATE DATABASE magento;
CREATE USER magento_user@localhost IDENTIFIED BY 'password';
GRANT ALL PRIVILEGES ON magento.* TO magento_user@localhost IDENTIFIED BY 'password';
FLUSH PRIVILEGES;
exit

Download and Set Up Magento Files

https://www.magentocommerce.com/download

cd /usr/local/src
tar xzvf magento-1.9.0.1.tar.gz
cp -r magento /var/www/html
chown -R www-data:www-data /var/www/html/magento

To access the web interface with your browser,
http://server_domain_name_or_IP/

Ref :- Magento-Ubuntu-14

Upgrade java 1.7 to 1.8 in Ubuntu 14.04

java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

sudo add-apt-repository ppa:webupd8team/java
sudo apt-get update
sudo apt-get install oracle-java8-installer
echo $JAVA_HOME

Set Java home path,

export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
echo $JAVA_HOME
/usr/lib/jvm/java-8-openjdk-amd64

Don't pass MaxPermSize to Java 8+

java -version
openjdk version "1.8.0_91"
OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-0ubuntu4~14.04-b14)
OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode)

Disable USB with regedit in Windows

To disable USB

Go to run and pass 'regedit' command to open registry editor and navigate to the following settings,

   --> KEY_LOCAL_MACHINE
        --> SYSTEM
            --> CurrentControlSet
                --> Services
                   --> USBSTOR
       
        Modify 'Start' file value >3 and save
    
To enable USB

        Modify 'Start' file value ≤ 3 and save

Wapiti 2.3.0 on Ubuntu 14.04

Requirement : - python 2.6 or later

get the wapiti-2.3.0.tar.gz from https://sourceforge.net/projects/wapiti/files/wapiti/wapiti-2.3.0/,

cd /usr/local/src
wget https://sourceforge.net/projects/wapiti/files/wapiti/wapiti-2.3.0/wapiti-2.3.0.tar.gz
tar -xzvf wapiti-2.3.0.tar.gz

python setup.py install
Traceback (most recent call last):
  File "setup.py", line 2, in <module>
    from setuptools import setup, find_packages
ImportError: No module named setuptools

 apt-get install python-setuptools

 python setup.py install

In the prompt, just type the following command to get the basic usage :

python wapiti -h

python wapiti.py http://server.com/base/url/ [options]

Example :wapiti http://domainname.com/

Ref :- https://sourceforge.net/projects/wapiti/files/wapiti/wapiti-2.3.0/

Install Apache Spark on Ubuntu 14.04

Install Scala

To install Java in Ubuntu machine,
apt-add-repository ppa:webupd8team/java
apt-get update
apt-get install oracle-java7-installer
java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

cd /usr/local/src/
wget http://www.scala-lang.org/files/archive/scala-2.11.8.tgz
sudo mkdir /usr/local/src/scala

tar xvf scala-2.11.8.tgz -C /usr/local/src/scala/

vi .bashrc
export SCALA_HOME=/usr/local/src/scala/scala-2.11.8
export PATH=$SCALA_HOME/bin:$PATH

restart bashrc
. .bash

scala -version
Scala code runner version 2.11.8 -- Copyright 2002-2016, LAMP/EPFL

scala
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_80).
Type in expressions for evaluation. Or try :help.
scala>

Install Spark

cd /usr/local/src/
wget http://d3kbcqa49mib13.cloudfront.net/spark-1.6.1.tgz
tar xvf spark-1.6.1.tgz
cd spark-1.6.1/

SBT(Simple Build Tool) is used for building Spark, which is bundled with it. To compile the code,
 
sbt/sbt assembly
NOTE: The sbt/sbt script has been relocated to build/sbt.
      Please update references to point to the new location.

      Invoking 'build/sbt assembly' now ...

Attempting to fetch sbt
Launching sbt from build/sbt-launch-0.13.7.jar
Getting org.scala-sbt sbt 0.13.7 ...
.....................................
[info] Packaging /usr/local/src/spark-1.6.1/assembly/target/scala-2.10/spark-assembly-1.6.1-hadoop2.2.0.jar ...
[info] Done packaging.
[success] Total time: 2597 s, completed Apr 14, 2016 5:01:34 PM

test a sample program

./bin/run-example SparkPi 10
Pi is roughly 3.139988

run Spark interactively through the Scala shell
./bin/spark-shell

scala> val textFile = sc.textFile("README.md")
textFile: org.apache.spark.rdd.RDD[String] = README.md MapPartitionsRDD[1] at textFile at <console>:27

scala> textFile.count()
res0: Long = 95

scala> exit

If want to check some particular sections of spark using shell.
For example run MQTT interactevely, the mqtt is defined under external for import that into spark-shell just follow the instructions

sbt/sbt "streaming-mqtt/package"
bin/spark-shell --driver-class-path external/mqtt/target/scala-2.10/spark-streaming-mqtt_2.10-1.1.0.jar
scala> import org.apache.spark.streaming.mqtt._

Ref : http://blog.prabeeshk.com/blog/2014/10/31/install-apache-spark-on-ubuntu-14-dot-04/

Wapiti on ubuntu 14.04

Wapiti is an open source web application vulnerability scanner. It can detect the following vulnerabilities:

backup: This module search backup of scripts on the server.
blindsql: Time-based blind sql scanner.
crlf: Search for CR/LF injection in HTTP headers.
exec: Module used to detect command execution vulnerabilities.
file: Search for include()/fread() and other file handling vulns.
htaccess: Try to bypass weak htaccess configurations.
nikto: Use a Nikto database to search for potentially dangerous files.
permanentxss: Look for permanent XSS.
sql: Standard error-based SQL injection scanner.
xss: Module for XSS detection.
buster: Module for a file and directory buster attack – checking for “bad” files.
shellshock: Module for Shellshock bug detection.

sudo apt-get install wapiti

wapiti http://example.org/cool-things -u -n 5 -b domain -v 2 -o /tmp/outfilename

-u, --color
    use colours

-b, --scope
    set the scope of the scan:
    page: only analyse the page given in the url
    folder: analyse all urls in the root url given (default option)
    domain: analyse all links to pages in the same domain

-n, --nice
    use this to prevent infinite loops, I usually go with 5

-f, --format
    change the output format
    json:
    html:
    openvas:
    txt:
    vulneranet:
    xml:

-v verbose
    0: none
    1: print each url
    2) print each attack

# if you dont specify a -v flag, then you get a blank screen for ages

Ref:- https://jonathansblog.co.uk/wapiti-tutorial

Creating a FEDERATED Table on MySQL

The FEDERATED Storage Engine

The FEDERATED storage engine lets you access data from a remote MySQL database without using replication or cluster technology. Querying a local FEDERATED table automatically pulls the data from the remote (federated) tables. No data is stored on the local tables.

mysql>  show engines;
| FEDERATED          | NO      | Federated MySQL storage engine

vi /etc/mysql/my.cnf
under [mysqld]
federated
   
mysql>  show engines;      
| FEDERATED          | YES     | Federated MySQL storage engine

/etc/init.d/mysql restart


SAMPLE 1

CREATE TABLE destination_databasename.`table_name_1_dtl` (
  Provider_ID VARCHAR(100) NOT NULL,
  Provider_Name VARCHAR(250) NOT NULL,
  Provider_Type VARCHAR(100) NOT NULL,
  Address VARCHAR(255) DEFAULT NULL,
  District_Name VARCHAR(100) NOT NULL,
  Provider_Phone_Number VARCHAR(50) NOT NULL,
  Specialty_Name VARCHAR(100) NOT NULL,
  Availability_of_Pharmacy_or_ChemistShop_within_Hospital TINYINT(1) DEFAULT NULL,
  Laboratory TINYINT(1) DEFAULT NULL,
  Empanelment_Status VARCHAR(50) DEFAULT NULL,
  Empanelment_Valid_Upto DATE DEFAULT NULL,
  De_Empaneled_Reason VARCHAR(250) DEFAULT NULL,
  Contact_Person VARCHAR(200) DEFAULT NULL,
  PRIMARY KEY (Provider_ID)
) ENGINE=FEDERATED DEFAULT CHARSET=utf8
CONNECTION='mysql://root:ussgnovbl5r@localhost:3306/source_databasename/table_1_name_dtl';

SAMPLE 2

CREATE TABLE destination_databasename.`table_name_2_dtl` (
  Package_ID VARCHAR(100) CHARACTER SET latin1 NOT NULL,
  Dept_ID VARCHAR(100) CHARACTER SET latin1 DEFAULT NULL,
  DeptSI_No VARCHAR(200) CHARACTER SET latin1 DEFAULT NULL,
  Package_Type VARCHAR(200) CHARACTER SET latin1 NOT NULL,
  Category VARCHAR(200) DEFAULT NULL,
  Specialty VARCHAR(200) CHARACTER SET latin1 DEFAULT NULL,
  Sub_Specialty VARCHAR(200) CHARACTER SET latin1 DEFAULT NULL,
  Package_Name VARCHAR(255) CHARACTER SET latin1 NOT NULL,
  Length_of_Stay VARCHAR(50) DEFAULT NULL,
  Rate_Card_A DECIMAL(65,2) NOT NULL,
  Rate_Card_B DECIMAL(65,2) DEFAULT NULL,
  Rate_Card_C DECIMAL(65,2) DEFAULT NULL,
  PRIMARY KEY (Package_ID),
  KEY Package_Type (Package_Type),
  KEY Package_Name (Package_Name),
  KEY Rate_Card_A (Rate_Card_A,Rate_Card_B,Rate_Card_C)
) ENGINE=FEDERATED DEFAULT CHARSET=utf8
CONNECTION='mysql://root:ussgnovbl5r@localhost:3306/source_databasename/table_2_name_dtl';

SAMPLE 3

CREATE TABLE destination_databasename.`table_name_3_dtl` (
  Preauth_Status VARCHAR(100) DEFAULT NULL,
  Total BIGINT(21) NOT NULL DEFAULT '0',
  Approved_Amount DECIMAL(65,2) DEFAULT NULL
) ENGINE=FEDERATED DEFAULT CHARSET=utf8
CONNECTION='mysql://root:ussgnovbl5r@localhost:3306/source_databasename/table_2_name_dtl';

Ref:- http://dev.mysql.com/doc/refman/5.7/en/federated-storage-engine.html