Basic - File Handling

file

The file command is a used for recognizing the type of data contained in a file.
 # file mon.tar
mon.tar: POSIX tar archive (GNU)
# file sanu/
sanu/: directory
# file file1
file1: ASCII text

- To check the types of all files in a directory.
# file /home/subrat/*
/home/subrat/file1:      ASCII text
/home/subrat/mon.tar:    POSIX tar archive (GNU)
/home/subrat/my.pkt:     ASCII text
/home/subrat/sanu:       directory
/home/subrat/sar.txt:    ASCII text

stat

The stat command is used to display status information of files and file systems.

- To check the status of a file.
# stat file1
  File: `file1'
  Size: 73740           Blocks: 152        IO Block: 4096   regular file
Device: caa1h/51873d    Inode: 5317        Links: 1
Access: (0600/-rw-------)  Uid: (  512/  subrat)   Gid: (  512/  subrat)
Access: 2014-04-11 09:40:37.095008865 +0530
Modify: 2014-04-09 06:42:21.635010321 +0530 
Change: 2014-04-09 06:42:21.635010321 +0530

- To check the status of a directory.
# stat /home/subrat/
  File: `/home/subrat/'
  Size: 4096            Blocks: 8          IO Block: 4096   directory
Device: caa1h/51873d    Inode: 8196        Links: 5
Access: (0700/drwx------)  Uid: (  512/  subrat)   Gid: (  512/  subrat)
Access: 2014-04-11 08:40:32.213008990 +0530
Modify: 2014-04-11 08:40:21.753009285 +0530
Change: 2014-04-11 08:40:21.753009285 +0530

- To check the status of a file system.
# stat -f  /dev/xvdk1
  File: "/dev/xvdk1"
    ID: 0        Namelen: 255     Type: tmpfs
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 69065      Free: 69025      Available: 69025
Inodes: Total: 69065      Free: 68440

locate

The locate command is often the simplest and quickest way to find the locations of files and directories. 

- To search a particular file.
# locate sar.txt
/home/subrat/sar.txt
/opt/apps/sar/sar.txt

- To locate the file irrepsective of its name.
# locate -i new.txt
/tmp/NEW.txt
/tmp/new.txt
/usr/share/doc/samba-common/WHATSNEW.txt.gz 

- To get the count of number of matching entry.
# locate -c sar.txt
2 

Note: One limitation of the ‘locate’ command is its dependency on the database which can be updated by another utility ‘updatedb’. Hence, in order to get the latest and reliable results from ‘locate’ command the database on which it works should be updated at regular intervals.

updatedb

The updatedb command is used to refresh mlocate Database.
# locate out.mp

I have created some files with the name 'out.mp'. But I am not able to  find. So, once run 'updatedb', then tried searching.
 
# updatedb
# locate out.mp
/home/subrat/out.mp
/home/subrat/sanu/out.mp
/home/subrat/sanu/out.mp.bak

which

The which command is used to locate executable in the system. It returns the absolute path of the executable that is called. It allows user to pass several command names as arguments to get their paths in the system. “which” commands searches the path of executable in system paths set in $PATH environment variable.

Syntax: which command | program

# which java
/usr/bin/java

You can search for multiple commands or programs also as below.
# which echo telnet chmod java
/bin/echo
/usr/bin/telnet
/bin/chmod
/usr/bin/java

whereis

The whereis command is used to find the source, binary, and manuals sections for specified commands or files.

# whereis find
find: /bin/find /usr/bin/find /usr/share/man/man1p/find.1p.gz /usr/share/man/man1/find.1.gz

- To search only binary files.
# whereis -b find
find: /bin/find /usr/bin/find

- To search only mannual sections files.
# whereis -m find
find: /usr/share/man/man1p/find.1p.gz /usr/share/man/man1/find.1.gz

- To search only source code files.
# whereis -s date
find:

- To search for multiple commands.
# whereis date grep
date: /bin/date /usr/share/man/man1p/date.1p.gz /usr/share/man/man1/date.1.gz
grep: /bin/grep /usr/share/man/man1p/grep.1p.gz /usr/share/man/man1/grep.1.gz

whatis

The whatis command is used to display a single line information about a command. It accomplishes this by searching the short descriptions in the whatis database for each keyword provided to it as an argument.

# whatis ls
ls                   (1)  - list directory contents
ls                   (1p)  - list directory contents

# whatis find
find                 (1p)  - find files
find                 (1)  - search for files in a directory hierarchy

Basic - Commands beginner's should know

clear

The clear command is one of the most handy command used to clear your terminal screen. The clear command does not affect files or jobs, it simply clears the clutter from your terminal screen. Another nice thing about `CLEAR` is that it doesn't erase your terminal scroll buffer, so if you need to see something that was on your screen a while ago, you can still use your mouse wheel to scroll back.

# clear
or CTRL + L

date

The date command is used to display the current date and time or set the system date/time

 # date
Fri Apr 11 11:50:40 GMT 2014

- To set a new date and time
   # date -s "Fri Apr 11 11:00:00 GMT 2014"
or # date -s "11 APR 2014 16:00:00"
or # date --set="11 APR 2014 16:00:00"

- To set date only
# date +%Y%m%d -s "YYYYMMDD"
# date +%Y%m%d -s "20131120"
- To set time only
# date +%T -s "HH:MM:SS"
# date +%T -s "16:24:30"

cal

The cal command is used to print the current month's calender. With this command also we can see any year/month's calender.

# cal
     April 2014
Su Mo Tu We Th Fr Sa
       1  2  3  4  5
 6  7  8  9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30

- To see a particular year's calender (e.g. 1995)
# cal 1995

- To see a particular month's calender of a year ( e.g. Feb 2005)
 # cal 2 2005

- To see last few months calender
# cal -3

uname

The uname command reports basic information about a computer's software and hardware. It privides the information about the name, version and other details about the current machine and the operating system running on it.

# uname
Linux
uname without any ooption prints the kernel name

- To get all the inrforamtion about the system  
# uname -a
Linux ip-10-132-73-9.ap-southeast-1.compute.internal 2.6.32-431.1.2.el6.x86_64 #1 SMP Sun Nov 24 09:37:37 EST 2013 x86_64 x86_64 x86_64 GNU/Linux

- To get the operationg system name
# uname -o
GNU/Linux
- To get the kernel name
# uname -s
Linux
- To get the hostname
# uname -n
ap-southeast-1.compute.internal
- To get the kernel release information
# uname -r
2.6.32-431.1.2.el6.x86_64

- To get the kernel version name
# uname -v
#1 SMP Sun Nov 24 09:37:37 EST 2013
- To get the machine hardware name
# uname -m
x86_64- To get the processor type
# uname -p
x86_64

- To get the informaiton about hardware platform information
# uname -i
x86_64  

hostname

The hostname command is used to view or change the hostname of the system.

- To view the hostname of the system
# hostname

- To change the hostname
# hostname subrat.com
# hostname
subrat.com

- To see the IP of the host
# hostname -i

-To see the Fully Qualified Domain Name of the host
# hostname -f

history

The history command is used to see the command line history i.e. lists the executed commands.

- To display the command line history
# history | more
- To display the Time stamp with commands
# export HISTTIMEFORMAT='%F %T  '
# history | more
 127  2014-04-11 08:31:16  vim /etc/sysctl.conf
  128  2014-04-11 08:31:16  sysctl -p
  129  2014-04-11 08:31:16  vim /etc/sysctl.conf
  130  2014-04-11 08:31:16  cat /proc/self/mapped_base
  131  2014-04-11 08:31:16  history
  132  2014-04-11 08:31:16  vim /etc/sysctl.conf
  133  2014-04-11 08:31:16  sysctl -p
  134  2014-04-11 08:31:16  byobu
  135  2014-04-11 08:31:16  aws help
- To clear command line history
# history -c

alias

The alias command allows a user to create simple names or abbreviations (even consisting of just a single character) for commands regardless of how complex the original commands are and then use them in the same way that ordinary commands are used.

Syntax: alias name='command'

- To set an alias
# alias p="pwd"

- To see a list of aliases set up on your system
# alias
alias cp='cp -i'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias mv='mv -i'
alias rm='rm -i'
alias p='pwd'

unalias

The unalias command is used to remove entries from the current user's list of aliases.

# unalias p

# alias
alias cp='cp -i'
alias l.='ls -d .* --color=auto'
alias ll='ls -l --color=auto'
alias ls='ls --color=auto'
alias mv='mv -i'
alias rm='rm -i'

Basic - File Opetaions

pwd

The pwd is an acronym for Print Working Directory. The pwd command is considered as one of the most frequently used commands on Linux, UNIX like operating systems along with the ls, and cd commands.

Purpose for typing pwd command are,
  - Find the full path to the current directory.
  - Store the full path to the current directory in the shell variable.
  - Verify the absolute path.
  - Verify the physical path i.e exclude symbolic links.

# pwd
/home/subrat

ls

The ls command is one of the most frequently used command in Linux. It lists the directories and files present inside a directory.

- List the files & directories in bare format.
# ls

- Long listing of files & directories.
# ls -l
total 48
-rwxr-xr--. 1 root   root   15504 Apr  2 04:52 iostat.txt
-rw-rw-r-x. 1 subrat subrat 20480 Apr  4 03:45 mon.tar
drwxr-x--x. 2 subrat subrat  4096 Apr  4 07:22 oracle
drwxr-x--x. 2 root   root    4096 Apr  8 08:04 java
-rw-------. 1 root   root     592 Apr  2 04:51 sar.txt
lrwxrwxrwx. 1 root   root      25 Dec 23 21:31 al -> /etc/alternatives

The First character specifies the type of the file as specified below,

-  normal file
d  directory
s  socket file
l  link file

Then rest of the fields are described as below,

1st Field - File Permissions.Each 3 characters refers to the read, write, execute permissions for user, group and others.
2nd Field - Link Count. Number of directory present in a directory. For file the Link count is 1 & for directory , the link count is 2. For sub-directories, the link count is 1.
3rd Field - The owner of the file.
4th Field - The group of the file.
5th Field - Size of the file.
6th Field - Month of last modification.
7th Field - Date of last modification.
8th Field - Time of last modification.
9th Field - File Name.

- Listing all files including hidden files.
# ls -la

- Listing the files with their size.
# ls -ls

- Sort the files by their size.
# ls -lS

- Display files with inode number.
# ls -li

- List files with human readable format.
# ls -lh

- List files in reverse order.
# ls -lr

- List the files according to last modified Time.
# ls -lt

- Display File UID and GID.
# ls -n

- Visual Classification of Files With Special Characters.
# ls -F

Note for the following representations,

    / – directory.
    nothing – normal file.
    @ – link file.
    * – Executable file

- Display Files Recursively.
# ls -R

- List the contents of your home directory.
# ls ~

- List the contents of your root directory.
# ls /

- List the contents of the parent directory.
# ls ../

- List the contents of all sub-directories.
# ls */

- Display a list of directories in the current directory.
 # ls -d */

- Display the information of a specific file.
 # ls -l /path/to/file

- Display the information of a specific directory.
# ls -ld /path/to/dir

- Visual Classification of Files With Colors.
# ls --color=auto

ll

The ll command is the equivalent of 'ls -l'. This command is used to get detail information about files and directories in a directory.

# ll
total 140
-rw-------. 1 subrat subrat 73740 Apr  9 06:42 file1
-rw-------. 1 subrat subrat  3324 Apr  9 06:43 file2
-rw-------. 1 root   root   15504 Apr  2 04:52 iostat.txt
-rw-------. 1 subrat subrat 20480 Apr  4 03:45 mon.tar
-rw-------. 1 subrat subrat    56 Apr  9 05:47 my.pkt
-rw-------. 1 root   root     549 Apr 11 08:40 out.mp

Just like the different options of ls can be used for ll too. Try more options.

dir

The dir command is used to list all the directories and files present in a directory. It does the same operation like ls command. It does not support content highlighting. For more info visit man page for dir.

# dir
dir1 dir2 file.txt

mkdir

The mkdir command is used to create directory or directories.

- Creating Directory/Directories  in current working directory.
# mkdir Dir1
# mkdir Dir2 Dir3

- Creating Directory/Directories in a specified path.
# mkdir /home/subrat/Dir1
# mkdir /home/subrat/Dir2 /root/Dir3

- Creating nested directories.
# mkdir -p /opt/apps/oracle/data

The above command will first create opt, then apps, then oracle & finally data.

cd

- Change working directory from present working directory using relative path.
# cd subrat/

Note: Your relative path starts from your present working directory.

- Change working directory from present working directory to any location.
# cd /path/to/directory

Note: In this case you are starting from / .So, you are changing the directory using the absolute path.

- Change working directory to user's home directory.
# cd ~
# cd

- Change directory to previous working directory which we changed from.
# cd -

- Change working directory to parent directory.
# cd ..

- Changing working directory three levels up in the directory.
# cd ../../..

cp

The cp command is used to copy one or more files or directories from source to destination. The original file remains unchanged, and the new file may have the same or a different name.

Syntax: cp [option] source destination

Note: While copying make sure that your destination location should not contain the file or directory with the same name as your source file or directory. Else the new content will be overwritten.
- To make a copy of a file in the current directory as a new file.
# cp file.txt new-file.txt

- To copy a file from your current directory into another directory.
# cp file_name /path/of/new_file

- To copy a directory from one source to another.
# cp -r dir1 /path/of/new_dir

- To copy multiple files into a directory.
# cp file2 file3 file4 dir1

- To copy multiple directories.
# cp -r src/ bin/ /home/subrat/

- To explain what is being done while copying. i.e. verbose
# cp -v file1 /path/to/new_file

- To copy a file or directory and preserve the attributes like modification date, time, and access control list associated with the source file or directory.
# cp -p file_name /path/to/new/location/myfile
# cp -rp dir_name /path/to/new/location/mydir

- To copy all the files in a directory to a new directory.
# cp * /home/subrat/backup

- To copy a directory, including all its files and sub-directories, to another directory.
# cp -R dir1/* /home/subrat/backup

- To ask for Confirmation before overwriting (interactive mode)if a file exist with the same name in destination.
# cp -i test.c bak
cp: overwrite 'bak/test.c'? y

- To take a backup before copying into a destination.
# cp -b file.txt  /home/subrat/
# cd /home/subrat/
# ls -l
-rw-r--r-- 1 subrat unix 1020 Apr  8 13:36 file.txt
-rw-r--r-- 1 subrat unix 1038 Apr  8 13:15 file.txt~

- To copy a write-protected file forcefully.
# cp -f file.txt /path/to/file.txt

- To copy some files having a common pattern by using wild-cards as shown below. In this example, all txt extension files gets copied.
# cp dir1/*.txt /home/subrat/text/

mv

The mv command is used to move files and directories from one location to another.
Apart from moving the files, it can also rename a file or directory.

Syntax: mv oldname newname

- To rename a file.
# mv file file_new

- To move a file from source to destination.
# mv filename /path/to/destination

- To move a file from source to destination with a new name.
# mv filename /path/to/destination/new_filename

- To rename a directory.
# mv -r dir1 dir2

- To move a directory from source to destination.
# mv -r dir1 /path/to/destination

- To move a directory from source to destination with a new name.
# mv -r dir_old /path/to/destination/dir_new

- To take a backup of destination before Overwriting.
# mv --suffix=.bak file_name /path/to/destination/

The above command creates another file with .bak which holds the old data.

- To move a write-protected file forcefully.
# mv -f file /path/to/destination

- To move a file interactively.
# mv -i sample.txt /backup

It will Prompt for a Confirmation Before Overwriting.
# mv -i sample.txt sample1.txt
mv: overwrite 'sample1.txt'? y

- To move some files having a common pattern by using wild-cards as shown below.
# mv *.txt /home/subrat/

rmdir

The rmdir command is used to delete an empty directory or set of empty directories.

- To delete an empty directory.
# rmdir dir_empty
# rmdir /path/to/empty_dir

- To delete mutltiple empty directories.
# rmdir dir1 dir2  dir3

- To delete nested empty directories.
# rmdir -p dir1/dir2/dir3

rm

The rm command is used to remove files or directories. Remove your files wisely! The effects of an rm cannot be undone.

- To remove file/files.
# rm file1
# rm file1 file2
# rm /path/to/file/

- To remove directory/directories.
# rm -r dir

- To remove a write-protected file or directory forcefully.
# rm -f file

- To prompt the user for confirmation before removing each file and directory.
# rm -i file

- To remove some files having a common pattern by using wild-cards as shown below.
# rm -f *.txt

Basic - File Management

cat

The cat command allows us to create single or multiple files, view contents of file, concatenate files and redirect output in terminal or files.

- Viewing contents of a single file.
# cat file_name

- Viewing contents of multiple files.
# cat file1 file2

- Creating a new file.
# cat > file_name
  <content of file>
  Ctrl+D - to save & exit.

- Appending data to file.
# cat >> file_name
  <new content>
  Ctrl+D - to save & exit.

Note: "cat > file_name" will be dangerous if previously the file exits with the same name. So be careful about that. Make sure that the file does not exist before. Otherwise, your previous data will be lost.

- Redirecting contents of multiple files into a single file.
# cat file1 file2 > new_file

- Display line numbers with file content.
# cat -n file_name

- Display the end of line ($).
# cat -e file_name

- Display TAB characters.
# cat -T file_name

- Display file content in Reverse.
# tac file_name

cat command disadvantages:
-  can not edit the file later.
-  if file content is more than the screen size, can not view all content.
-  can not create multiple file.

Note: All these consequences can be kick out by introducing vi-editor.

tee

With the help of tee command we can create multiple files at a time.The tee command reads from standard input, and writes to standard output and to files. It is used to store and view (both at the same time) the output of any other command.The command writes the output both to the screen (STDOUT) and to the file.

- Creating a file.
# tee file_name      //same procedure like cat

- Creating multiple file with same content at a time.
# tee file1 file2 file3

- Appending new content to the existing file.
 # tee -a file_name
 
- Appending the same content to multiple existing files.
# tee -a file1 file2 file3

- Write output to STDOUT and also to a file.
# ls -l | tee file_name

- Write output to STDOUT and also to multiple files at same time.
# ls -l | tee file1 file2

-Write output to STDOUT and append to a file.
# who | tee file_name

- Write output to STDOUT and append to multiple files at same time.
# who | tee file_name 

touch

The touch command is the easiest way to create new, empty files. It is also used to change the timestamps (i.e., dates and times of the most recent access and modification) on existing files and directories.

- Creating an empty file.
# touch file_name

- Creating multiple empty files.
# touch file1 file2 file3

- Avoid creating new files.
# touch -c file_name
If the file doesn't exist, touch will not create the file.

- Change the file modification time only.
# touch -m file.txt

- Change the file access time only.
# touch -a file.txt

- Explicitly change the Access & Modification time.
# touch -am file.txt
# touch -a -m -t 201406181125.20 file.txt
# touch -d '1 May 2005 10:22' file.txt
# touch -d '14 May' file.txt
# touch -d '14:24' file.txt
# touch -d "2014-04-05 12:12:12.000000000 +0530" file.txt

- Creating a file using a specified time.
# touch -t YYYYMMDDHHMM.SS file_name
# touch -t 201312101420.30 tmp.txt

The time stamp of tmp.txt will change to 14:20:30 PM on December 10th 2013.

- Use the time stamp of another file.
# touch -r file_old file_new

The -r (i.e., reference) option followed directly by a space and then by a file name tells touch to use that file's time stamps instead of current time.

# touch -r file_new -B 30 file_new
# touch -r file_new -F 30 file_new

The -B option modifies the timestamps by going back the specified number of seconds, and the -F option modifies the time by going forward the specified number of seconds. For example, the above two command would make file_new 30 seconds older than & 30 seconds newer than file_old respectively.

- To check the time stamp of file
# ls -l file_name
or # stat file_name

more

The more command is to view (but not modify) the contents of a file one screen at a time (terminal pager). It is a filter for paging through text one screenful at a time.

It can be taken as input for an output exceeding the screen.

- To view content of file.
# more /path/to/file

- To take as input of an output that exceeding the screen.
# ls -l /usr/bin/ | more

Note: Press <Space Bar> or F key to move one page forward and <Enter> key to move one line forward. We can not perform backward movement with the help of more command. In middle of execution press Q key to come back to command line.

less

The less command is used to view (but not change) the contents of a file one screen at a time. It is similar to more, but has the extended capability of allowing both forward and backward navigation through the file. Also, less does not have to read the entire input file before starting, so with large input files it starts up faster than text editors like vi.

- To view content of a file.
# less /path/to/file

- To view from an output that exceeding screen.
# ls -l /usr/bin | less

To traverse the file, less has more options as following,
    Down Arrow or <Enter>  - scrolls down one line
    Up Arrow - scrolls up one line
    d - scrolls down half a page
    b - scrolls up half a page
    Space Bar / F - scrolls one page
    G – go to the end of file
    g – go to the start of file or ZZ – exit the less pager
    10j – 10 lines forward.
    10k – 10 lines backward.
    CTRL+G – show the current file name along with line, byte and percentage statistics.

head

The head command is used to print the first desired lines from a file to the terminal. head by default, prints the first 10 lines of each file to standard output.

- To display first 10 lines.
# head file_name

- To display first n lines.
# head -n2 file_name
# head -2 file_name

-To skip the last n lines & display rest of all.
# head -n -5 file_name
# head -n-5 file_name

- To display first n lines of multiple files.
# head -10 file1 file2 file3

- Print the first n bytes.
# head -c 5 file_name
# head -c5 file_name

- Skip printing last n bytes.
# head -c -10 file_name
# head -c-10 file_name

- Print the first n kilobytes.
# head -k 5 file_name
# head -k5 file_name

- Skip printing last n kilobytes.
# head -k -10 file_name
# head -k-10 file_name

- Passing output of other command to head input.
# ls -l | head -20
# history | head -100

tail

The tail command is used to print the last N lines from the file on the terminal. The tail  by default, prints the last 10 lines of each file to standard output. tail command is especially used with log files to read the last few lines to know about the error messages.

- To display last 10 lines.
# tail file_name

- To display last n lines.
# tail -n2 file_name
# tail -2 file_name

- To print lines starting from nth line to the last.
# tail -n +20 file_name
# tail +20 file_name

- To display first n lines of multiple files.
# tail -10 file1 file2 file3

- To print the last n bytes.
# tail -c 10 file_name
# tail -c10 file_name

- To print characters from the nth byte.
# tail -c+40 file_name
# tail -c+40 file_name

- Print last lines from dynamically changing files. e.g. log files.
# tail -f log_file

- Passing output of other command to head input.
# ls -l | tail -10 

Note: Display specific lines (based on line number) of a file using head and tail command

e.g. I have a file having 100 lines. I want to display from 50th line to 70th line. With alone head or tail, I can not do this. So, here is the solution.

 # cat file_name | head -70 | tail -21
or # head -70 file_name | tail -21