FAQ

How can i check what jobs are running and killing them?
How can i check the difference between 2 files/directories?
How can i check what files are open by a process?
How can i count the number of lines?
How can i run the same command repeatley?
How can i check disk space?
How can i check space of a number of files or processes?
How long does it take a command to run?
Show a quick way to configure networking/firewall...
How do  you read and search in zipped files?
How do you check whats mounted (but with a nicer output)
How do you spell check a word?
How do you show updated entries into a file?  (with a twist)
How do you show vmstat but with nicer output?
How do you Output text continuously?
How do you contact all users on a machine?
How long has a process been running
what using all the io
what to check if your are getting strange disk performance?
How do i change the timezone in linux?
Disk space not showing the correct output? (even after you have freed up space)
What are binary files
Deleting old files
Killing or deleting files/processes
Finding memory usage for all httpd/mysql processes7
How do i convert Epoch dates
What is vim/view
What is vim recording
Follow the ip
Measuring latency
Understanding the limitations of traceroute
Having problems login into a server
Df
Memory
How to use scp
How to redirect output or a running process
How to login via ssh without passwords
The power of the less command
How to display information about users on the system
Record a shell session
What is a Zombie
What does the mkfifo command do
Ssh problems

How can i check what jobs are running and killing them?

kill – to kill a process – i.e kill 1234 to kill process id 1234
killall – to kill an application –i.e killall vi to kill vi
pkill -KILL -u rambo1 # logout user Rambo
ctrl z – during a long query press ctrl z to run it in the background
jobs –to see what jobs are running in the background i.e jobs
kill %1 to kill job 1
fg 1 – to bring job 1 to the foreground

Order to kill misbehaving process
kill -1 pid # safest way to kill a process
kill pid #which runs kill -15 soft termination, stop in an orderly fashion
kill -2 pid # like a ctrl c
kill -9 pid # force kill

examples:

Yes test ; ctrl z it and then kill %1 to kill job 1

steps to move manipulate the status of a running process:

1) updatedb & # to run a command in the background (note if you do not redirect the output you screen will fill up with text)
2) jobs # to see details of that job
2a) jobs -p #show pid of job
2b) jobs -l #show a combination of jobs and jobs -p
3) fg job_id # moves the job to the foreground
4) ctrl z '#will suspend a job
5)bg job_id #will move the job to the background
6) nohup command-with-options & # lets a process continue running even after you have logged out

How can i check the difference between 2 files/directories?

diff – to compare the contents of 2 files i.e diff file1 file2 to compare file1 and file2,
diff -u file1 file2 a nicer output
sdiff compares files side by side

or for a directory diff /tmp/a/ /tmp/b/

you also have diff3 to compare 3 files

Also see cmp and comm

for a good understanding of the summary see
http://www.brighthub.com/computing/linux/articles/41449.aspx

How can i check what files are open by a process?

lsof – to list open files i.e
lsof –u jamie to list all open files for the user jamie,
lsof -i :80 to show all files used by port 80
lsof -i tcp:80 # check for port 80 (very similar to the above)

How can i count the number of lines?

wc - word count particular useful when finding the number of rows of data/output i.e locate file | wc –l shows the number of times the word file was found

How can i run the same command repeatley?

watch -d ls –l # To watch the contents of a directory change,
watch -n 10 ls –lh # To watch the contents of the directory every 10 seconds run
watch -d 1 date # the d highlights anything changed since the last run
watch "netstat -plan | grep :80" to watch all new connections on port 80

How can i check disk space?

there are various way to check disk space here a few of the solutions:

du –chx /home/ | sort -n | tail -10 # a good solution to check the disk space (better than du –sh ./*)

very similar ....

du -a /home | sort -n -r | head -n 10
Df is not a partitioning utility, but prints out details about only mounted file systems

pydf is an Improved version of df

disk space on a big partition (over 1TB ish) has recentley increased run ls -ltr and see
what folders have been updated recentley and then do a more accurate search from there

to run a disk usage search no the local filesystem (presuming the following are their own partions 1 2 3) -useful if you have a lot of mounted partitions and you want to check where else the space is being used (espically if 1/2/3 are big partitions)

sudo du -sh `ls -1 / | egrep -v "1|2|3"`

the following is very useful if you want to find the disk space used by certain files (pipe any kind of files list combination to xards

find ./ -type f | xargs du -csh
find ./ -name "access_log*" |xargs du -csh


How can i check space of a number of files or processes

ls -lh *.txt ; ls -l *.txt | awk '{ SUM += $5} END { print SUM/1024/1024 }'

How long does it take a command to run?

Timing a command time netstat –at or time ss will show how long a command takes useful to test if you have 2 similar commands and you want to work out which one is quicker.

Example:

time ls

real 0m0.002s
user 0m0.000s
sys 0m0.001s

where (taken from stackflow)

Real is wall clock time - time from start to finish of the call. This is all elapsed time including time slices used by other processes and time the process spends blocked (for example if it is waiting for I/O to complete).

User is the amount of CPU time spent in user-mode code (outside the kernel) within the process. This is only actual CPU time used in executing the process. Other processes and time the process spends blocked do not count towards this figure.

Sys is the amount of CPU time spent in the kernel within the process. This means executing CPU time spent in system calls within the kernel, as opposed to library code, which is still running in user-space. Like 'user', this is only CPU time used by the process. See below for a brief description of kernel mode (also known as 'supervisor' mode) and the system call mechanism.

to get more info use the verbose parameter
/usr/bin/time --verbose ls
123 456 789
Command being timed: "ls"
User time (seconds): 0.00
System time (seconds): 0.00
Percent of CPU this job got: 0%
Elapsed (wall clock) time (h:mm:ss or m:ss): 0:00.00
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 960
Average resident set size (kbytes): 0
Major (requiring I/O) page faults: 0
Minor (reclaiming a frame) page faults: 293
Voluntary context switches: 1
Involuntary context switches: 0
Swaps: 0
File system inputs: 0
File system outputs: 0
Socket messages sent: 0
Socket messages received: 0
Signals delivered: 0
Page size (bytes): 4096
Exit status: 0

Show a quick way to configure networking/firewall

Setup # basic configuration
System-config-network #

How do you read and search in zipped files?

Z commands allows you to perform normal file operation on a compressed files Some of these z commands uncompresses the file temporarily in the /tmp directory to perform the specified operation. Some of the z commands uncompresses it on the fly to perfom the specified operation
zcat filename.gz | more
zdiff
the z commands zcat zgrep zdiff # to search in zipped files
zgrep

from the man page

Zgrep invokes grep on compressed or gzipped files. All options specified are passed directly to grep.

so

zgrep "108.62.115.226:36898" ./29.gz
tcp 0 0 ::ffff:188.226.209.67:80 ::ffff:108.62.115.226:36898 TIME_WAIT -

gives the same output as... (you can see it is running something similar to this if you do an lsof on the zgrep)

gzip -cdfq ./29.gz | grep "108.62.115.226:36898"
tcp 0 0 ::ffff:188.226.209.67:80 ::ffff:108.62.115.226:36898 TIME_WAIT -

How do you check whats mounted (but with a nicer output)

to check what is currently mounted run mount
cat /proc/mounts will also show what the system sees as mounted
cat /etc/fstab will show what should be mounted
a simple df will show disk space and what mounts they relate to

mount | column –t # shows nicer output
parted /dev/sda then type print show in a much nice format the partition table

How do you spell check a word?

look today # does a spell check on the word "today"

How do you show updated entries into a file? (with a twist)

tail -f -n0 /var/log/cron - the n0 flag shows no output from the file combined with the f flag it will show a blank screen and only update when the cron file is updated (without the n0 flag tail -f shows the last 10 lines)

How do you show vmstat but with nicer output?

vmstat -w 1 5 # the w makes the output look a bit more readable

How do you Output text continuously?

Yes –output a string of “y” continuously until killed

How do you contact all users on a machine?

wall - to send a message to everyone logged in i.e wall "this server will reboot in 5 mins please log off"

how long has that process been running

ps -p pid -o etime= # check how long a process has been running for

to find more info on a kernel modules (with a .ko extension) run for example modinfo /lib/modules/2.6.32-431.5.1.el6.x86_64/kernel/sound/usb/misc/snd-ua101.ko
Check a process start time with
1) STIME from ps –ef
2) ps -eo pid,cmd,etime
3) ls -ld /proc/”pid”

what using all the io?

running iotop and pressing o will show only those processes taking up io

a useful page for troubleshooting IO http://bencane.com/2012/08/06/troubleshooting-high-io-wait-in-linux/

from http://stackoverflow.com/questions/488826/what-process-is-using-all-of-my-disk-io
a very useful loop to check io usage

while true; do date; ps auxf | awk '{if($8=="D") print $0;}'; sleep 1; done

what to check if your are getting strange disk performance?

If you are having odd Disk issues (Speed/performance) issues check major and minor bios versions

How do i change the timezone in linux

2 methods

1) mv /etc/localtime /etc/localtime.bak
ln -s /usr/share/zoneinfo/GB /etc/localtime

2) rm -f /etc/localtime
tzselect

Disk space not showing the correct output? (even after you have freed up space)

freeing up more disk space (from processes that should have already deleted the files)

1) so search for the largest value in column 7

lsof | grep deleted
...
pty-spawn 1338 root 3w REG 8,5 14843540032 41 /var/log/abc.log (deleted)
...

2) get the pid from column 2 & use the 4th column to get the file descriptor details
ls -l /proc/1338/fd/
0 1 2 3 4 5 6 7

3) then confirm you are looking at the right file
ls -l /proc/1338/fd/3
l-wx------ 1 root root 64 Mar 10 08:18 /proc/1338/fd/3 -> /var/log/abc.log (deleted)

4) then delete the file

cat /dev/null > /proc/1338/fd/3

the following contains an example
http://www.cyberciti.biz/tips/freebsd-why-command-df-and-du-reports-different-output.html

What are Binary files

binary files are machine code files not to be read. Just think as the opposite for a text file. You can use the "file" command on any file to get its type. Use Hexdump/strings to view these files

Deleting old files

find /var/log/1/ -mtime +180 -type f -exec ls -l {} \; | wc -l
find /var/log/1/ -mtime +170 -type f -exec ls -l {} \; | wc -l

espically useful if you are running out of disk space and you want to search for older files to free up space.

then add -delete to the find command if you want to delete them

Killing or deleting files/processes

a nice way to check what you are going to kill before you do it
echo "kill `pidof screen`"
and then pipe it to bash
echo "kill `pidof screen`" | bash
this really gets powerful when you combine it with other commands i.e to kill all httpd processes run
echo kill `ps -ef | grep httpd | grep -v grep | awk '{print $2}' `| bash

Finding memory usage for all httpd/mysql processes

to check how much memory is being used and then to see memory as per proc
sudo ps -efalyc | awk '{total += $8}END{size= total / 1024; printf "total size %.2f MB\n", size}'

How do i convert Epoch dates

to get an epoch date run date +%s

to convert it back again run date -d @epoch_date

to change the output when you have converted to and from your epoch date:

date -d@123465789
Thu Nov 29 19:03:09 EST 1973
date -d @$123465789 +"%Y-%m-%d %H:%M %Z"
1970-09-29 10:16 EDT

A nice example taken from https://geekpeek.net/nagios-log-convert-timestamp/
which allows you to read epoch timestamps in a log file.

tail -f /var/log/nagios/nagios.log | perl -pe 's/(\d+)/localtime($1)/e'

 

What is vim/view

Vim is very similar to vi, but with a few extra features i.e Syntax highlighting. On many systems "vi" is just a symbolic link to "vim"

view (is installed as a command under vim) allows you to open a file in read only mode i.e view passwd

to password protect a vim file run vim +X filename

What is vim recording

if setup it records everything you type and then you can replay it. (useful for repeated jobs)

Follow the IP

If you want to get an ip from sql and trace it back

  • run a mysql query and get port i.e servera.com:56788
  • check the db and the server specified run netstat -plant | grep 56788 to get the process id that initated the connection i.e
    tcp 0 0 x.x.x.x.:56788 x.x.x.x:3306 ESTABLISHED 321/serviceA
  • on servera strace and lsof process 321

Measuring latency

Here are a couple of useful links to help track latency

http://www.traceroute.org/ global looking glass aggregator of ISPs that provide that service. Very handy to get an idea of latency from a specific network to our public facing locations.
http://www.intelletrace.com/Level-3-Looking-Glass.html - Level3 specific looking glass tool.
http://www.internetpulse.net/ is a peering health report indicating average latency and packet loss - you can also customize this to be geo specific to a certain network.

Understanding the limitations of traceroute

taken from from https://www.youtube.com/watch?v=TYxPLEmbCuk

a lot of things can break it, i.e:
some configuration of mpls,tunnnels) can cause stars/skips or break traceroute
multipath - a packet can take multiple paths (tricks for network routing)
anycast - you may not be tracing to the same ip as someone else (1 ip doesn't mean 1 machine, 1 ip can source from multiple locations). If optimal routing is used you will go to a node nearest to you (based on something like geoiplocation).Not great for things like webservers.

Forward and reverse traceroute are sometimes needed for accurate results.

See http://www.catonmat.net/blog/tcp-traceroute/ for a good example of tracerouting.

An interesting link that explains another tool (tcpping)to use for tracerouting that avoids some of the pitfulls off a firewall dropping packets
http://xmodulo.com/how-to-install-tcpping-on-linux.html

Having problems login into a server

If you have not logged on to the server before check their is diskspace available in /home (if it needs to create certain files/directories on login and cannot you will see issues)

Df

a very simple yet useful command, according to the man page it "report file system and disk space usage"
a few examples

df -h shows partitions and disk usag in human readable format (i.e MB/GB/TB)
df -i shows information on inodes
df -T shows partitions as well

Linux Memory

free -m is the most basic test of seeing how much memory is available
Info can also be seen by running top, vmstat and sar

Linux is quite able to manage the memory well itself. Sometimes though you may need to flush the cache from memory by running
echo 1 > /proc/sys/vm/drop_caches

Linux How to use scp

SCP is a client tool that allows secure copy between hosts.

scp user@from-host:sourcefile user@remote-host:destinationfile -Access files from the remote host.

scp localhost user@server:/home/user/ -Copy files from the local computer to remote.

scp user@remotehost:file /home/user -Copy files from the remote computer to local.

scp 123.csv linuxblog.info:~ – to copy the file from the local to the remote machine

How to redirect output or a running process

running lsof you may find a file logging in tmp or a log file elsewhere.
If that is not helpful you may find the following an interesting read http://etbe.coker.com.au/2008/02/27/redirecting-output-from-a-running-process/

How to login via ssh without passwords

a good tutorial and info at http://www.thegeekstuff.com/2008/11/3-steps-to-perform-ssh-login-without-password-using-ssh-keygen-ssh-copy-id/

The power of the less command

less is good for viewing files, when you are in less if you press v it will take you into vi mode.

How to display information about users on the system

id root # will show you the info on a specific user
finger root # will show you more info that id
lslogins # will show information on all known users

Record a shell session

use the script command

script my.session
#run your commands
type quit to exit

and to view the script cat my.session

What is a Zombie

see https://linuxblog.info/linux-basic-commands/#Top

and

Defunct processes are also known as zombies, and listed with a 'Z' status in the output from ps.
They're not quite as destructive as the living dead, as they consume almost no system resources, but on a system that's always turned on, such as a server,
they can become equally distracting. The key to killing a defunct process is to first kill the parent,
which will be listed in the output of ps with the addition of -l for long output. Parent processes can be identified under the PPID column,
as opposed to the PID column for the process ID. These are identifiers attached to each process running on your system.
They can be killed using another common shell command, kill -9, followed by the PPID. Obviously this will stop the parent task,
so first make sure it's not essential. Once the parent process has been killed, the system init process should send the correct signal to the defunct process,
which should terminate automatically. On reboot Zombie should be cleared up.

What does the mkfifo command do

see also https://linuxblog.info/understanding-how-linux-works-under-the-hood-stracelsofproc/#pi
and edit the contents of the link to point at this new link

the following has almost been taken word for word from http://www.tuxradar.com/content/command-line-tricks-smart-geeks
Make your own Bash wormholes

It creates a pipe for sharing data, connecting two running utilities with a kind of command line wormhole.
Data sent into one end will instantaneously appear at the other.

Before we look at how to use it, it's worth going over how we typically see pipes. If you've used the shell for anything other than scaring your friends with cat /dev/random, you'll be used to the idea of pipes. They're most often used to stream the output of one program into the input of another. A common use is when there is too much textual output from one command to read. Piping this output into another - usually either less or more - lets you pause and page through the output in your own time:

cat /var/log/messages | less

In this instance, the pipe is temporarily created for the execution of a single command, but using mkfifo it's possible to create persistent pipes that you can use for similar tasks.

The 'fifo' part of the command refers to the nature of the pipe - the data that's first in is first out. Creating the pipe itself is as easy as typing mkfifo, followed by the name you wish to call it. It's also possible to set the permissions for the pipe (using the --mode parameter) so you can restrict access. Once the pipe is created, you just need to route data into it. Here's a brief example. First we create the pipe, and use tail -f to output any data that's sent to it:

mkfifo fifo_pipe
tail -f fifo_pipe

The next step, usually from another console or user account (if the permissions have been set), is to send data to the pipe. Typing echo "This is a test" >> fifo_pipe will send the test message, which will itself be output by the tail command we attached to the pipe.

SSH problems

If you encounter issues try running ssh in verbose mode and checking the logs of the server you are trying to connect to.
For a few worked examples see http://www.daveperrett.com/articles/2010/09/14/ssh-authentication-refused/

If you need to alter ssh timeouts https://uk.godaddy.com/help/how-to-set-an-ssh-timeout-12300