SSH

SSH is some kind of an abbreviation of Secure SHell. It is a protocol that allows secure connections between computers. In this tutorial, we’ll be dealing with the ssh command on Linux, the OpenSSH version. Most Linux distributions feature the OpenSSH client today, but if you want to be sure, have a look at the SSH manpage on your system. You can do this by typing:

[pinehead@localhost ~]$ man ssh

Note: this should be done in a terminal. This tutorial assumes that you have some basic terminal knowledge, like knowing how to start a terminal session on your system and being familiar with the basic commands and syntaxes.

If it displays something like this

NAME
ssh – OpenSSH SSH client (remote login program)

then you can be quite sure you’re running the OpenSSH version. For more background information about SSH, seehttp://en.wikipedia.org/wiki/SSH.

The most simple case

In the most simple case, you can connect to a server that supports ssh with a syntax as short as this:

[pineehad@localhost ~]$ ssh yourserver

Note: If you do not have any ssh server nearby that you can access, you can also try this command with your own computer as a server. To do this, replace “yourserver” with “localhost”.

Of course, yourserver should be replaced by a hostname or an ip address of the server you want to connect to. As you can see in the terminal snippet, I am logged in as pineehad. If you do not specify a username (I’ll explain how to do that later in this tutorial), SSH will assume that you want to login with the username you’re currently logged in with. So, in this case, SSH will try the username pineehad.

Of course, you need to be sure that the server supports ssh connections. The ssh client tries to connect to port 22 defaultly. This means that, if you want to connect to a remote host with the default settings, you should make sure that, if applicable, port 22 is forwarded to the server you’re trying to connect to. You will find more regarding the SSH port further in this tutorial.

Now, back to the command we ran. If the server supports SSH connections and you can reach it by port 22, you should be prompted for a password (if this is the first time you try to connect to the server, ssh will first ask the question if you want to continue connecting, which can generally just be answered with a ‘yes’). If you type a password here, you won’t see asterisks appearing. Don’t panic, this is ssh’s normal behaviour. It makes connecting using ssh even more safe, because any accidental spectators won’t be able to see the length of the password. After entering the password, if the username and the password were correct, you should be running a shell on the server. If not, make sure you are connecting to a server of which you know that you should be able to login with your username and the specified password. You could try connecting to your own computer (see the note beneath the terminal quote) or read on to learn how to specify an other username.

Once you’re done trying the ssh shell, you can exit it by pressing Ctrl + D.

Specifying a username

It’s actually quite simple to specify a different username. You might even already be familiar with it. See the following example:

[pinehead@localhost ~]$ ssh yourusername@yourserver

The above will make ssh try to connect with the username “yourusername” instead of (in my case) pineehad. This syntax is also used by a lot of other protocols, so it’ll always come in handy to know it. By the way, you will still be asked for a password. For security reasons, it is not even possible to directly specify the password in the syntax. You will always be asked interactively, unless you start configuring the server in an advanced way (which is exactly why that topic is out of this tutorials scope: this tutorial documents how to use the clients, not how to configure the server).

Specifying a port

There are many reasons to move the ssh service to an other port. One of them is avoiding brute-force login attempts. Certain hackers try to get access to ssh servers by trying a lot of common usernames with common passwords (think of a user “john” with password “doe”). Although it is very unlikely that these hackers will ever get access to the system, there is an other aspect of the brute-force attacks that you’ll generally want to avoid: the system and connection load. The brute-force attacks usually are done with dozens or even thousands of tries a second, and this unnecessarily slows down the server and takes some bandwidth which could’ve been used a lot better. By changing the port to a non-default one, the scripts of the hackers will just be refused and most of the bandwidth will be saved.

As the ssh command can’t just guess the port, we will have to specify it if it’s not the default 22 one. You can do that this way:

[pineehad@localhost ~]$ ssh -p yourport yourusername@yourserver

Of course, you will have to replace “yourport” with the port number. These is an important difference between ssh and scp on this point. I’ll explain it further on.

Running a command on the remote server

Sometimes, especially in scripts, you’ll want to connect to the remote server, run a single command and then exit again. The ssh command has a nice feature for this. You can just specify the command after the options, username and hostname. Have a look at this:

[pineehad@localhost ~]$ ssh yourusername@yourserver updatedb

This will make the server update its searching database. Of course, this is a very simple command without arguments. What if you’d want to tell someone about the latest news you read on the web? You might think that the following will give him/her that message:

[pineehad@localhost ~]$ ssh yourusername@yourserver wall “Hey, I just found out something great! Have a look at www.examplenewslink.com!”

However, bash will give an error if you run this command:

bash: !”: event not found

What happened? Bash (the program behind your shell) tried to interpret the command you wanted to give ssh. This fails because there are exclamation marks in the command, which bash will interpret as special characters that should initiate a bash function. But we don’t want this, we just want bash to give the command to ssh! Well, there’s a very simple way to tell bash not to worry about the contents of the command but just pass it on to ssh already: wrapping it in single quotes. Have a look at this:

[pineehad@localhost ~]$ ssh yourusername@yourserver ‘wall “Hey, I just found out something great! Have a look at www.examplenewslink.com!”‘

The single quotes prevent bash from trying to interpret the command, so ssh receives it unmodified and can send it to the server as it should. Don’t forget that the single quotes should be around the whole command, not anywhere else.

SCP

The scp command allows you to copy files over ssh connections. This is pretty useful if you want to transport files between computers, for example to backup something. The scp command uses the ssh command and they are very much alike. However, there are some important differences.

The scp command can be used in three* ways: to copy from a (remote) server to your computer, to copy from your computer to a (remote) server, and to copy from a (remote) server to another (remote) server. In the third case, the data is transferred directly between the servers; your own computer will only tell the servers what to do. These options are very useful for a lot of things that require files to be transferred, so let’s have a look at the syntax of this command:

[pineehad@localhost ~]$ scp examplefile yourusername@yourserver:/home/yourusername/

Looks quite familiar, right? But there are differences. The command above will transfer the file “examplefile” to the directory “/home/yourusername/” at the server “yourserver”, trying to get ssh acces with the username “yourusername”. That’s quite a lot information, but scp really needs it all. Well, almost all of it. You could leave out the “yourusername@” in front of “yourserver”, but only if you want to login on the server with your current username on your own computer. Let’s have a closer look at the end of the command. There’s a colon over there, with a directory after it. Just like Linux’s normal cp command, scp will need to know both the source file(s) and the target directory (or file). For remote hosts, the file(s)/directory are given to the scp command is this way.

You can also copy a file (or multiple files) from the (remote) server to your own computer. Let’s have a look at an example of that:

[pineehad@localhost ~]$ scp yourusername@yourserver:/home/yourusername/examplefile .

Note: The dot at the end means the current local directory. This is a handy trick that can be used about everywhere in Linux. Besides a single dot, you can also type a double dot ( .. ), which is the parent directory of the current directory.

This will copy the file “/home/yourusername/examplefile” to the current directory on your own computer, provided that the username and password are correct and that the file actually exists.

You probably already guessed that the following command copies a file from a (remote) server to another (remote) server:

[pineehad@localhost ~]$ scp yourusername@yourserver:/home/yourusername/examplefile yourusername2@yourserver2:/home/yourusername2/

Please note that, to make the above command work, the servers must be able to reach each other, as the data will be transferred directly between them. If the servers somehow can’t reach each other (for example, if port 22 is not open on one of the sides) you won’t be able to copy anything. In that case, copy the files to your own computer first, then to the other host. Or make the servers able to reach each other (for example by opening the port).

Well, those are the main uses of scp. We’ll now go a bit more in-depth about the differences between ssh and scp.

*: Actually you can also use it just like the normal cp command, withhout any ssh connections in it, but that’s quite useless. It requires you to type an extra ‘s’ =).

Specifying a port with scp

The scp command acts a little different when it comes to ports. You’d expect that specifying a port should be done this way:

[pineehad@localhost ~]$ scp -p yourport yourusername@yourserver:/home/yourusername/examplefile .

However, that will not work. You will get an error message like this one:

cp: cannot stat `yourport’: No such file or directory

This is caused by the different architecture of scp. It aims to resemble cp, and cp also features the -p option. However, in cp terms it means ‘preserve’, and it causes the cp command to preserve things like ownership, permissions and creation dates. The scp command can also preserve things like that, and the -p option enables this feature. The port specification should be done with the -P option. Therefore, the following command will work:

[pineehad@localhost ~]$ scp -P yourport yourusername@yourserver:/home/yourusername/examplefile .

Also note that the -P option must be in front of the (remote) server. The ssh command will still work if you put -p yourport behind the host syntax, but scp won’t. Why? Because scp also supports copying between two servers and therefore needs to know which server the -P option applies to.

Another difference between scp and ssh

Unlike ssh, scp cannot be used to run a command on a (remote) server, as it already uses that feature of ssh to start the scp server on the host. The scp command does have an option that accepts a program (the -S option), but this program will then be used instead of ssh to establish the encrypted connection, and it will not be executed on the remote host.

Tips & Tricks with ssh and scp

Quite a handy thing about scp is that it supports asterisks. You can copy all files in a remote directory in a way like this:

[pineehad@localhost ~]$ scp yourusername@yourserver:/home/yourusername/* .

And you can also just copy a whole directory by specifying the -r (recursive) option:

[pineehad@localhost ~]$ scp -r yourusername@yourserver:/home/yourusername/ .

Both of these also work when copying to a (remote) server or copying between a (remote) server and another (remote) server.

The ssh command can come in handy if you don’t know the exact location of the file you want to copy with scp. First, ssh to the (remote) server:

[pineehad@localhost ~]$ ssh yourusername@yourserver

Then browse to the right directory with cd. This is essential Linux terminal knowledge, so I won’t explain it here. When you’re in the right directory, you can get the full path with this command:

[pineehad@localhost ~]$ pwd

Note: pwd is an abbreviation of Print Working Directory, which is a useful way to remember the command.

You can then copy this output, leave the ssh shell by pressing Ctrl + D, and then paste the full directory path in your scp command. This saves a lot of remembering and typing!

You can also limit the bandwidth scp may use when copying. This is very useful if you’re wanting to copy a huge amount of data without suffering from slow internet for a long time. Limiting bandwidth is done this way:

scp -l bandwidthlimit yourusername@yourserver:/home/yourusername/* .

The bandwidth is specified in Kbit/sec. What does this mean? Eight bits is one byte. If you want to copy no faster than 10 Kbyte/sec, set the limit to 80. If you want to copy no faster than 80 Kbyte/sec, set the limit to 640. Get it? You should set the limit to eight times the maximum Kbyte/sec you want it to be. I’d recommend to set the -l option with all scp’ing you do on a connection that other people need to use, too. A big amount of copying can virtually block a whole 10 Mbit network if you’re using hubs.

Source (Pinehead.tv)

1.) Ctrl+U and Ctrl+Y.

Do you know that moment when you’re typing a long command, and then suddenly realize you need to execute something else first? Especially when working over an SSH connection, when you can’t easily open a second terminal on the same machine, this can be very annoying. Solution: ensure your cursor is at the end of your current command (shortcut: Ctrl+E), press Ctrl+U to get a clean line, type the other command you need to execute first, execute it, then press Ctrl+Y and voila! Your long command is back on the line. No mouse needed for copying, just quick hotkeys.

2.) Ctrl+R.

Speaking of long commands: when you need a previously used command again but don’t want to retype it because it’s long or complex, there’s a good chance it’s stored in your history file. The quickest way to retrieve and execute it is to press Ctrl+R and type a few characters that are part of your command. For example:

for pid in $(pidof plugin-container); do file /proc/”$pid”/fd/* |
fgrep /tmp/Flash | cut -d: -f1 | xargs mplayer; done

Now that’s a command that I wouldn’t like to type every time I use it. It’s a one-liner to play the temporary media files Flash secretly stores when you play, for example, a YouTube movie. When I want to execute this monster, I only type the following:

Ctrl+R
“pidof”
Enter

The stuff between quotes is literally typed, as in I press p, then i, and so on. The reason this works is that I almost never use the pidof command except in this one-liner, so the most recent command executed that contains “pidof” is almost always the right one. However, suppose I did recently execute a different command containing “pidof”. By repeatedly pressing Ctrl+R after typing “pidof” but before pressing Enter, I can cycle through the list of commands until I hit the one I meant to execute, and then press Enter. And, last but not least, you can still edit the command you found using Ctrl+R before hitting Enter; just press the right or left arrow to get out of the history search mode, and edit away!

3.) The screen command.

When working over an ssh connection, commands that take long to execute can seriously get in your way. You have to keep the connection open to allow the command to complete, which means that you can’t turn off your computer, and you can’t execute a different command without opening a second ssh connection or (temporarily) terminating the running command. Both annoyances melt away when you use screen.

The screen command allows you to run multiple terminal sessions inside a single terminal session, and manage the multiple sessions using hotkeys. Try it! Just execute “screen” in your terminal (install it first if necessary) and see an empty terminal opening. Now execute the command “sleep 9999″. This will take quite long and block your terminal. However, if you press Ctrl+A, let go of the keys, and then press C, you will get a fresh new terminal ready to take commands. The sleep command on the other terminal keeps running without being interrupted. To cycle between the two open terminals, use Ctrl+A N for next and Ctrl+A P for previous (remember to let go of all keys after pressing Ctrl+A). Finally, to shut down screen without interrupting the commands that are running, press Ctrl+A D for detach. You will return to your original terminal, and if it’s an ssh session, you can exit it without interrupting the commands running in screen. To get back to the screen terminals you opened before, execute “screen -R” for reattach. To exit a screen session, just exit all terminals in it as you would normally exit terminals (Ctrl+D).

For more information about screen, read its man page by executing “man screen”. It’s a very powerful tool that even allows multiple people to use a single terminal at the same time!

4.) The xargs command.
Whenever you want to execute a command on multiple files, or for every line of a certain file, xargs is the first tool to look at. Here’s an example:

find . -iname ‘*.php’ -print0 | xargs -0 svn add

Anyone who has ever worked with a version control system like svn probably knows the annoyance of having to svn add every newly created code file after a few hours of editing. This command does it for you in an instant. How does it work?

1. “find . -iname *.php -print0″ prints all files in the current directory (“.”) or its subdirectories that end in .php (case-insensitive) and separates them by null characters (“-print0″). The null character is never used in filenames, while a newline may be, so it is safer to separate by null characters.
2. “xargs -0 svn add” receives the output of find on its standard input through the pipe (“|”), separates it by null characters (“-0″) and feeds the filenames as arguments to svn, after the “add” argument. It minds the limits the system imposes on the amount and size of command line arguments, and will run svn multiple times as necessary while still invoking it as few times as possible.

To find out more about xargs, read the man page by executing “man xargs”. Using xargs is safer and more versatile than using the -exec option of find. For ultimate versatility, however, use the slightly less elegant for loop, which is described in the second part of this list.

5.) Using bash as a simple calculator.

Sometimes you need to quickly do a calculation that is too large or too important to do using your head. When you’re working in a graphic environment, you might just fire up kcalc or gcalctool, but tools like that may not always be available or easy to find. Fortunately, you can do basic calculations within bash
itself. For example:

echo $((3*37+12)) # Outputs 123
echo $((2**16-1)) # Two to the power of sixteen minus one; outputs 65535
echo $((103/10)) # Outputs 10, as all these operations are integer
arithmetic
echo $((103%10)) # Outputs 3, which is the remainder of 103 divided by
10

The $((something))-syntax also allows bitwise operations, and as such it interprets the caret (“^”) as a bitwise operator. That’s why “**” is used for “to the power of”. The syntax also supports showing the decimal equivalent of a hexadecimal or octal number. Here’s an example:

echo $((0xdeadbeef)) # Outputs 3735928559
echo $((0127)) # Outputs 87

For more information, read the bash man page using “man bash” and search (using the / key) for “arithmetic evaluation”. If you want to do floating point calculations, you can use bc (might need to install first):

echo ‘scale=12; 2.5*2.5′ | bc # Outputs 6.25
echo ‘scale=12; sqrt(14)’ | bc # Outputs 3.741657386773

Note the setting of the scale variable. Using bc, you can perform floating point operations with any precision you like. The scale variable controls the amount of decimals behind the dot that are calculated. I used 12 here because kcalc uses that amount by default, but you can increase or decrease it as you like. Find out more about what bc can do by executing “man bc”. It even supports more advanced mathematical functions, such as the arctangent or the natural logarithm.

6.) Quoting.

To allow you to write awesome scripts, bash attaches a special meaning to many characters (like * & ; | { ! < [ # and a lot more). Sometimes, however, these characters are not to be interpreted in any special way, but just to be passed unmodified to some command. A commonly used example:

find . -iname '*.conf'

Notice the single quotes around "*.conf". What if I forgot those? Bash would interpret the *.conf as a glob expression and expand it to all files ending in .conf in the current directory. This could prevent find from looking for all .conf files in subdirectories, causing unexpected results. Therefore, it's a good habit to always quote anything that might contain special characters.

Note that only single quotes prevent every character from being interpreted; double quotes still allow bash to interpret some characters. When working with variables, double quotes come in handy to prevent word splitting. This is often used in scripting. An example:

#!/bin/bash
# This script renames a file to lowercase
newname="$(echo -n "$1" | tr '[A-Z]‘ ‘[a-z]‘)”
mv -i “$1″ “$newname”

You can see quite a lot of double quotes in this script, and all of them are necessary to prevent trouble with certain filenames. You see, when a filename contains whitespace, bash splits the name on that whitespace when you leave variables unquoted. Suppose you have a file named “spaced name.txt”, you put it in a variable (“filename=’spaced name.txt’”), and then you try to move it to “unspacedname.txt” by executing “mv $filename unspacedname.txt”. You’ll get the error “mv: target `unspacedname.txt’ is not a directory”. This is because mv gets executed like this: “mv spaced name.txt unspacedname.txt”. In other words, mv will try to move two files, “spaced” and “name.txt” to “unspacedname.txt”, and fail because moving multiple files to a single destination is only allowed when the destination is a directory. Putting double quotes around “$filename” solves this issue.

So you see, quoting is a good habit to prevent your commands and scripts from doing unexpected things. Two final notes: you can’t quote variables using single quotes, because the dollar sign loses its special meaning between single quotes, and if you ever need to use a literal single quote in some command, you can do so by putting it between double quotes. Using \’ between single quotes will not work, because even the backslash it not interpreted between single quotes.

7.) For loops.

Using xargs, you can run a command on a list of files. However, sometimes you’ll want to use more advanced functionality of bash on a list of files. For example, you might want to set up a pipe of commands, like this:

# Count the number of while loops in each of the php files in the
current directory
for file in ./*.php; do echo -n “$file”:\ ; grep ‘while’ “$file” | wc
-l; done

Let me explain what happens there part by part:

1. “for file in ./*.php”: make a list of all the files ending in “.php” in the current directory, and for each of them, run the following code with the variable “$file” set to the filename.
2. “do … done”: indicates the beginning and end of the code inside the for loop. Note that both of them have to start on a new line, which is why the semicolons are there. They count for starting a new line.
3. “echo -n “$file”:\ “: print the filename, followed by a colon and a space, but not a newline (the “-n” option suppresses the default extra newline). The backslash makes sure the space is actually printed and not eaten away by bash.
4. “grep ‘while’ “$file” | wc -l”: grep reads the file and prints only the lines containing the word “while”. These lines are piped to
   “wc -l”, which counts the amount of lines (word count with option lines).

As you can see, you can easily place multiple commands within the for loop; every bit of code between “do” and “done” is executed for every file. Imagine the possibilities, especially when you get to know more of the built-in bash functionality. Read on for an example of that.

8.) String manipulation.
There comes a time in the life of every serious bash user when some string manipulation is needed. For example, you might have a number of photos
of which the filenames start with “DSC”, and you’d like to replace that prefix with something more meaningful, like “Vacation2011″. Using a for loop as it was introduced above, you can do this in a flash (pun intended):

for file in DSC*; do mv “$file” Vacation2011″${file#DSC}”; done

How does this work? The for loop runs for every file in the current directory of which the name begins with “DSC”. For every such file, mv is executed to move the file to “Vacation2011<name of the file with DSC stripped from the front>”. The magic that strips “DSC” happens in “${file#DSC}”, where the “#” indicates that the string after it should be removed from the front of the contents of the variable before it. There also exist operations to strip from the back of the string (useful for removing file extensions), to search and replace in a string and to extract substrings. To learn more about string manipulation, visitManipulating
Strings on TLDP.

9.) Dynamic port forwarding.
Sometimes you need to access a website that’s only accessible from computers inside a certain network. Or, the other way around, sometimes you are in a certain network and you need unobstructed access to the internet, but some hyperactive firewall is in the way. If you have ssh access to a computer that does have the internet access you need, you can use it as an anonymous tunneling proxy without any additional tools. Just add the following option when you ssh into the server:

ssh -D <port number> user@remotehost

The -D option tells ssh to set up a dynamic port forward on local port <port number>. I like to use port number 1337, but almost any port between 1024 and 65535 will do. When you’ve logged in on the remote server, you can configure your browser to use localhost with the port number you specified earlier as a SOCKS proxy (SOCKS versions 4 and 5 are both supported). This setting is usually found in the same place as the other options for using a web
proxy. In Firefox, look under Preferences => Advanced => Network => Configure how Firefox connects to the Internet. Once you’re browsing over this SOCKS proxy, it will appear to the web as if the host you sshed into is browsing.

10.) Process substitution.
Ever wanted to diff the outputs of two commands quickly? Of course, you could redirect the output to a temporary file for both of them, and diff those files, like this:

find /etc | sort > local_etc_files
find /mnt/remote/etc | sort > remote_etc_files
diff local_etc_files remote_etc_files
rm local_etc_files remote_etc_files

This would tell you the differences between which files are in /etc on the local computer and a remote one. It takes four lines, however. Using process substitution, we can do this is just a single line:

diff <(find /etc | sort) <(find /mnt/remote/etc | sort)

What’s that <(…) syntax? It means “run the command inside it, connect the output to a temporary pipe file and give that as an argument”. To understand this more thoroughly, try running this:

echo <(echo test)

Instead of printing “test”, this will print something like “/dev/fd/63″. You see now that the <(…) part is actually replaced by a file. This file is a stream from which the output of the command inside <(…) can be read, like this:

cat <(echo test)

Now this does print “test”! Bash redirects the output of “echo test” to /dev/fd/<something>, gives the path of that file to cat, and cat reads the output of echo from that file. The shortened diff command above does the same, only for two slightly more complicated commands. This technique can be applied in any place where a temporary file is needed, but it does have a limitation. The temporary file can only be read once before it disappears. There’s no use in saving the name of the temporary file. If you need multiple accesses to the output of a program, use an old-fashioned temporary file or see if you can use pipes instead.

source (pinhead.tv)

In order to reliably build our servers, we use kickstart and PXE to give us a simple, repeatable process. During this build process, we need user input to decide exactly which configuration to apply to the system being built.

In order to do that, we run a Python script in the ks.cfg %pre section that uses the snack libraryto get information from the user. Snack is the (poorly documented) UI toolkit that comes with kickstart/anaconda. Here’s how you use it in %pre.

First, you need a %pre section that runs the python interpreter. To do that, start the section with:

%pre --interpreter /usr/bin/python

Next, you need to realize that the kickstart screen you usually see runs on tty3. But the snack UI will show up on tty1. So we use a little routine to switch tty’s:

def set_tty(n):
    f = open('/dev/tty%d' % n, 'a')
    os.dup2(f.fileno(), sys.stdin.fileno())
    os.dup2(f.fileno(), sys.stdout.fileno())
    os.dup2(f.fileno(), sys.stderr.fileno())

Next comes the function that actually calls snack to get the user input. Don’t worry about the host_config parameter, instead focus on how snack is used. This code build a dialog box with a listbox and an OK button:

def get_user_input(host_config, default=None):
    # get the hostname, from it the other params are computed
    # return the hostname and everything that's derived from it
    # which for now is just the disk layout scheme

    from snack import SnackScreen, Listbox, Grid, Label, Entry, Button, GridForm

    def host_list():
        def hosts():
            return sorted(host_config.keys())

        lb = Listbox(height=len(host_config), returnExit=True)
        for host in hosts():
            lb.append(host, host)
        if default in host_config.keys():
            lb.setCurrent(default)
        return lb

    screen = SnackScreen()
    form = GridForm(screen, 'Select host to configure', 1, 2)
    form.add(host_list(), 0, 0, (0, 0, 0, 1))
    form.add(Button('OK'), 0, 1)

    # now run the form
    result = form.runOnce()
    screen.finish()

    hostname = form.childList[0].current()
    return {'layout': host_config[hostname],
            'hostname': hostname,
            }

Finally, put it all together:

set_tty(1)  # change to tty1, we're called by kickstart with stdout as tty3
user_args = get_user_input(host_config, args.args.get('tb-host'))
set_tty(3)  # restore

If you ever need to find information on snack, it helps to know that it’s a wrapper for newt. So you can Google on “snack newt python” to get some useful answers. But be warned that there’s not much there and you might have to look through the source code to snack.

Running a command from your history

Sometimes you know that you ran a command a while ago and you want to run it again. You know a bit of the command, but you don’t exactly know all options, or when you executed the command. Of course, you could just keep pressing the Up Arrow until you encounter the command again, but there is a better way. You can search the bash history in an interactive mode by pressing Ctrl + r. This will put bash in history mode, allowing you to type a part of the command you’re looking for. In the meanwhile, it will show the most recent occasion where the string you’re typing was used. If it is showing you a too recent command, you can go further back in history by pressing Ctrl + r again and again. Once you found the command you were looking for, press enter to run it. If you can’t find what you’re looking for and you want to try it again or if you want to get out of history mode for an other reason, just press Ctrl + c. By the way, Ctrl + c can be used in many other cases to cancel the current operation and/or start with a fresh new line.

Repeating an argument

You can repeat the last argument of the previous command in multiple ways. Have a look at this example:

[rechosen@localhost ~]$ mkdir /path/to/exampledir
[rechosen@localhost ~]$ cd !$

The second command might look a little strange, but it will just cd to /path/to/exampledir. The “!$” syntax repeats the last argument of the previous command. You can also insert the last argument of the previous command on the fly, which enables you to edit it before executing the command. The keyboard shortcut for this functionality is Esc + . (a period). You can also repeatedly press these keys to get the last argument of commands before the previous one.

Some keyboard shortcuts for editing

There are some pretty useful keyboard shortcuts for editing in bash. They might appear familiar to Emacs users:

• Ctrl + a => Return to the start of the command you’re typing
• Ctrl + e => Go to the end of the command you’re typing
• Ctrl + u => Cut everything before the cursor to a special clipboard
• Ctrl + k => Cut everything after the cursor to a special clipboard
• Ctrl + y => Paste from the special clipboard that Ctrl + u and Ctrl + k save their data to
• Ctrl + t => Swap the two characters before the cursor (you can actually use this to transport a character from the left to the right, try it!)
• Ctrl + w => Delete the word / argument left of the cursor
• Ctrl + l => Clear the screen

Dealing with jobs

If you’ve just started a huge process (like backupping a lot of files) using an ssh terminal and you suddenly remember that you need to do something else on the same server, you might want to get the huge process to the background. You can do this by pressing Ctrl + z, which will suspend the process, and then executing the bg command:

[rechosen@localhost ~]$ bg
[1]+ hugeprocess &

This will make the huge process continue happily in the background, allowing you to do what you need to do. If you want to background another process with the huge one still running, just use the same steps. And if you want to get a process back to the foreground again, execute fg:

[rechosen@localhost ~]$ fg
hugeprocess

But what if you want to foreground an older process that’s still running? In a case like that, use the jobs command to see which processes bash is managing:

[rechosen@localhost ~]$ jobs
[1]- Running hugeprocess &
[2]+ Running anotherprocess &

Note: A “+” after the job id means that that job is the ‘current job’, the one that will be affected if bg or fg is executed without any arguments. A “-” after the job id means that that job is the ‘previous job’. You can refer to the previous job with “%-”.

Use the job id (the number on the left), preceded by a “%”, to specify which process to foreground / background, like this:

[rechosen@localhost ~]$ fg %3

And:

[rechosen@localhost ~]$ bg %7

The above snippets would foreground job [3] and background job [7].

Using several ways of substitution

There are multiple ways to embed a command in an other one. You could use the following way (which is called command substitution):

[rechosen@localhost ~]$ du -h -a -c $(find . -name *.conf 2>&-)

The above command is quite a mouthful of options and syntax, so I’ll explain it.

• The du command calculates the actual size of files. The -h option makes du print the sizes in human-readable format, the -a tells du to calculate the size of all files, and the -c option tells du to produce a grand total. So, “du -h -a -c” will show the sizes of all files passed to it in a human-readable form and it will produce a grand total.
• As you might have guessed, “$(find . -name *.conf 2>&-)” takes care of giving du some files to calculate the sizes of. This part is wrapped between “$(” and “)” to tell bash that it should run the command and return the command’s output (in this case as an argument for du). The find command searches for files named <can be anything>.conf in the current directory and all accessible subdirectories. The “.” indicates the current directory, the -name option allows to specify the filename of the file to search for, and “*.conf” is an expression that matches any string ending with the character sequence “.conf”.
• The only thing left to explain is the “2>&-”. This part of the syntax makes bash discard the errors that find produces, so du won’t get any non-filename input. There is a huge amount of explanation about this syntax near the end of the tutorial (look for “2>&1″ and further).

And there’s another way to substitute, called process substitution:

[rechosen@localhost ~]$ diff <(ps axo comm) <(ssh user@host ps axo comm)

The command in the snippet above will compare the running processes on the local system and a remote system with an ssh server. Let’s have a closer look at it:

• First of all, diff. The diff command can be used to compare two files. I won’t tell much about it here, as there is an extensive tutorial about diff and patch on this site.
• Next, the “<(” and “)”. These strings indicate that bash should substitute the command between them as a process. This will create a named pipe (usually in /dev/fd) that, in our case, will be given to diff as a file to compare.
• Now the “ps axo comm”. The ps command is used to list processes currently running on the system. The “a” option tells ps to list all processes with a tty, the “x” tells ps to list processes without a tty, too, and “o comm” tells ps to list the commands only (“o” indicates the starting of a user-defined output declaration, and “comm” indicates that ps should print the COMMAND column).
• The “ssh user@host ps axo comm” will run “ps axo comm” on a remote system with an ssh server. For more detailed information about ssh, see this site’s tutorial about ssh and scp.

Let’s have a look at the whole snippet now:

• After interpreting the line, bash will run “ps axo comm” and redirect the output to a named pipe,
• then it will execute “ssh user@host ps axo comm” and redirect the output to another named pipe,
• and then it will execute diff with the filenames of the named pipes as argument.
• The diff command will read the output from the pipes and compare them, and return the differences to the terminal so you can quickly see what differences there are in running processes (if you’re familiar with diff’s output, that is).

This way, you have done in one line what would normally require at least two: comparing the outputs of two processes.

And there even is another way, called xargs. This command can feed arguments, usually imported through a pipe, to a command. See the next chapter for more information about pipes. We’ll now focus on xargs itself. Have a look at this example:

[rechosen@localhost ~]$ find . -name *.conf -print0 | xargs -0 grep -l -Z mem_limit | xargs -0 -i cp {} {}.bak

Note: the “-l” after grep is an L, not an i.

The command in the snippet above will make a backup of all .conf files in the current directory and accessible subdirectories that contain the string “mem_limit”.

• The find command is used to find all files in the current directory (the “.”) and accessible subdirectories with a filename (the “-name” option) that ends with “.conf” (“*.conf” means “<anything>.conf”). It returns a list of them, with null characters as separators (“-print0″ tells find to do so).
• The output of find is piped (the “|” operator, see the next chapter for more information) to xargs. The “-0″ option tells xargs that the names are separated by null characters, and “grep -l -Z mem_limit” is the command that the list of files will be feeded to as arguments. The grep command will search the files it gets from xargs for the string “mem_limit”, returning a list of files (the -l option tells grep not to return the contents of the files, but just the filenames), again separated by null characters (the “-Z” option causes grep to do this).
• The output of grep is also piped, to “xargs -0 -i cp {} {}.bak”. We know what xargs does, except for the “-i” option. The “-i” option tells xargs to replace every occasion of the specified string with the argument it gets through the pipe. If no string is specified (as in our case), xargs will assume that it should replace the string “{}”. Next, the “cp {} {}.bak”. The “{}” will be replaced by xargs with the argument, so, if xargs got the file “sample.conf” through the pipe, cp will copy the file “sample.conf” to the file “sample.conf.bak”, effectively making a backup of it.

These substitutions can, once mastered, provide short and quick solutions for complicated problems.

Piping data through commands

One of the most powerful features is the ability to pipe data through commands. You could see this as letting bash take the output of a command, then feed it to an other command, take the output of that, feed it to another and so on. This is a simple example of using a pipe:

[rechosen@localhost ~]$ ps aux | grep init

If you don’t know the commands yet: “ps aux” lists all processes executed by a valid user that are currently running on your system (the “a” means that processes of other users than the current user should also be listed, the “u” means that only processes executed by a valid user should be shown, and the “x” means that background processes (without a tty) should also be listed). The “grep init” searches the output of “ps aux” for the string “init”. It does so because bash pipes the output of “ps aux” to “grep init”, and bash does that because of the “|” operator.

The “|” operator makes bash redirect all data that the command left of it returns to the stdout (more about that later) to the stdin of the command right of it. There are a lot of commands that support taking data from the stdin, and almost every program supports returning data using the stdout.

The stdin and stdout are part of the standard streams; they were introduced with UNIX and are channels over which data can be transported. There are three standard streams (the third one is stderr, which should be used to report errors over). The stdin channel can be used by other programs to feed data to a running process, and the stdout channel can be used by a program to export data. Usually, stdout output (and stderr output, too) is received by the terminal environment in which the program is running, in our case bash. By default, bash will show you the output by echoing it onto the terminal screen, but now that we pipe it to an other command, we are not shown the data.

Please note that, as in a pipe only the stdout of the command on the left is passed on to the next one, the stderr output will still go to the terminal. I will explain how to alter this further on in this tutorial.

If you want to see the data that’s passed on between programs in a pipe, you can insert the “tee” command between it. This program receives data from the stdin and then writes it to a file, while also passing it on again through the stdout. This way, if something is going wrong in a pipe sequence, you can see what data was passing through the pipes. The “tee” command is used like this:

[rechosen@localhost ~]$ ps aux | tee filename | grep init

The “grep” command will still receive the output of “ps aux”, as tee just passes the data on, but you will be able to read the output of “ps aux” in the file <filename> after the commands have been executed. Note that “tee” tries to replace the file <filename> if you specify the command like this. If you don’t want “tee” to replace the file but to append the data to it instead, use the -a option, like this:

[rechosen@localhost ~]$ ps aux | tee -a filename | grep init

As you have been able to see in the above command, you can place a lot of command with pipes after each other. This is not infinite, though. There is a maximum command-line length, which is usually determined by the kernel. However, this value usually is so big that you are very unlikely to hit the limit. If you do, you can always save the stdout output to a file somewhere inbetween and then use that file to continue operation. And that introduces the next subject: saving the stdout output to a file.

Saving the stdout output to a file

You can save the stdout output of a command to a file like this:

[rechosen@localhost ~]$ ps aux > filename

The above syntax will make bash write the stdout output of “ps aux” to the file filename. If filename already exists, bash will try to overwrite it. If you don’t want bash to do so, but to append the output of “ps aux” to filename, you could do that this way:

[rechosen@localhost ~]$ ps aux >> filename

You can use this feature of bash to split a long line of pipes into multiple lines:

[rechosen@localhost ~]$ command1 | command2 | … | commandN > tempfile1

[rechosen@localhost ~]$ cat tempfile1 | command1 | command2 | … | commandN > tempfile2

And so on. Note that the above use of cat is, in most cases, a useless one. In many cases, you can let command1 in the second snippet read the file, like this:

[rechosen@localhost ~]$ command1 tempfile1 | command2 | … | commandN > tempfile2

And in other cases, you can use a redirect to feed a file to command1:

[rechosen@localhost ~]$ command1 < tempfile1 | command2 | … | commandN > tempfile2

To be honest, I mainly included this to avoid getting the Useless Use of Cat Award =).

Anyway, you can also use bash’s ability to write streams to file for logging the output of script commands, for example. By the way, did you know that bash can also write the stderr output to a file, or both the stdout and the stderr streams?

Playing with standard streams: redirecting and combining

The bash shell allows us to redirect streams to other streams or to files. This is quite a complicated feature, so I’ll try to explain it as clearly as possible. Redirecting a stream is done like this:

[rechosen@localhost ~]$ ps aux 2>&1 | grep init

In the snippet above, “grep init” will not only search the stdout output of “ps aux”, but also the stderr output. The stderr and the stdout streams are combined. This is caused by that strange “2>&1″ after “ps aux”. Let’s have a closer look at that.

First, the “2″. As said, there are three standard streams (stin, stdout and stderr).These standard streams also have default numbers:

• 0: stdin
• 1: stdout
• 2: sterr

As you can see, “2″ is the stream number of stderr. And “>”, we already know that from making bash write to a file. The actual meaning of this symbol is “redirect the stream on the left to the stream on the right”. If there is no stream on the left, bash will assume you’re trying to redirect stdout. If there’s a filename on the right, bash will redirect the stream on the left to that file, so that everything passing through the pipe is written to the file.

Note: the “>” symbol is used with and without a space behind it in this tutorial. This is only to keep it clear whether we’re redirecting to a file or to a stream: in reality, when dealing with streams, it doesn’t matter whether a space is behind it or not. When substituting processes, you shouldn’t use any spaces.

Back to our “2>&1″. As explained, “2″ is the stream number of stderr, “>” redirects the stream somewhere, but what is “&1″? You might have guessed, as the “grep init” command mentioned above searches both the stdout and stderr stream, that “&1″ is the stdout stream. The “&” in front of it tells bash that you don’t mean a file with filename “1″. The streams are sent to the same destination, and to the command receiving them it will seem like they are combined.

If you’d want to write to a file with the name “&1″, you’d have to escape the “&”, like this:

[rechosen@localhost ~]$ ps aux > \&1

Or you could put “&1″ between single quotes, like this:

[rechosen@localhost ~]$ ps aux > ‘&1′

Wrapping a filename containing problematic characters between single quotes generally is a good way to stop bash from messing with it (unless there are single quotes in the string, then you’d have have escape them by putting a \ in front of them).

Back again to the “2>&1″. Now that we know what it means, we can also apply it in other ways, like this:

[rechosen@localhost ~]$ ps aux > filename 2>&1

The stdout output of ps aux will be sent to the file filename, and the stderr output, too. Now, this might seem unlogical. If bash would interpret it from the left to the right (and it does), you might think that it should be like:

[rechosen@localhost ~]$ ps aux 2>&1 > filename

Well, it shouldn’t. If you’d execute the above syntax, the stderr output would just be echoed to the terminal. Why? Because bash does not redirect to a stream, but to the current final destination of the stream. Let me explain it:

• First, we’re telling bash to run the command “ps” with “aux” as an argument.
• Then, we’re telling to redirect stderr to stdout. At the moment, stdout is still going to the terminal, so the stderr output of “ps aux” is sent to the terminal.
• After that, we’re telling bash to redirect the stdout output to the file filename. The stdout output of “ps aux” is sent to this file indeed, but the stderr output isn’t: it is not affected by stream 1.

If we put the redirections the other way around (“> filename” first), it does work. I’ll explain that, too:

• First, we’re telling bash to run the command “ps” with “aux” as an argument (again).
• Then, we’re redirecting the stdout to the file filename. This causes the stdout output of “ps aux” to be written to that file.
• After that, we’re redirecting the stderr stream to the stdout stream. The stdout stream is still pointing to the file filename because of the former statement. Therefore, stderr output is also written to the file.

Get it? The redirects cause a stream to go to the same final destination as the specified one. It does not actually merge the streams, however.

Now that we know how to redirect, we can use it in many ways. For example, we could pipe the stderr output instead of the stdout output:

[rechosen@localhost ~]$ ps aux 2>&1 > /dev/null | grep init

The syntax in this snippet will send the stderr output of “ps aux” to “grep init”, while the stdout output is sent to /dev/null and therefore discarded. Note that “grep init” will probably not find anything in this case as “ps aux” is unlikely to report any errors.

When looking more closely to the snippet above, a problem arises. As bash reads the command statements from the left to the right, nothing should go through the pipe, you might say. At the moment that “2>&1″ is specified, stdout should still point to the terminal, shouldn’t it? Well, here’s a thing you should remember: bash reads command statements from the left to the right, but, before that, determines if there are multiple command statements and in which way they are separated. Therefore, bash already read and applied the “|” pipe symbol and stdout is already pointing to the pipe. Note that this also means that stream redirections must be specified before the pipe operator. If you, for example, would move “2>&1″ to the end of the command, after “grep init”, it would not affect ps aux anymore.

We can also swap the stdout and the stderr stream. This allows to let the stderr stream pass through a pipe while the stdout is printed to the terminal. This will require a 3rd stream. Let’s have a look at this example:

[rechosen@localhost ~]$ ps aux 3>&1 1>&2 2>&3 | grep init

That stuff seems to be quite complicated, right? Let’s analyze what we’re doing here:

• “3>&1″ => We’re redirecting stream 3 to the same final destination as stream 1 (stdout). Stream 3 is a non-standard stream, but it is pretty much always available in bash. This way, we’re effectively making a backup of the destination of stdout, which is, in this case, the pipe.
• “1>&2″ => We’re redirecting stream 1 (stdout) to the same final destination as stream 2 (stderr). This destination is the terminal.
• “2>&3″ => We’re redirecting stream 2 (stderr) to the final destination of stream 3. In the first step of these three ones, we set stream 3 to the same final destination as stream 1 (stdout), which was the pipe at that moment, and after that, we redirected stream 1 (stdout) to the final destination of stream 2 at that moment, the terminal. If we wouldn’t have made a backup of stream 1′s final destination in the beginning, we would not be able to refer to it now.

So, by using a backup stream, we can swap the stdout and stderr stream. This backup stream does not belong to the standard streams, but it is pretty much always available in bash. If you’re using it in a script, make sure you aren’t breaking an earlier command by playing with the 3rd stream. You can also use stream 4, 5, 6 7 and so on if you need more streams. The highest stream number usually is 1023 (there are 1024 streams, but the first stream is stream 0, stdin). This may be different on other linux systems. Your mileage may vary. If you try to use a non-existing stream, you will get an error like this:

bash: 1: Bad file descriptor

If you want to return a non-standard stream to it’s default state, redirect it to “&-”, like this:

[rechosen@localhost ~]$ ps aux 3>&1 1>&2 2>&3 3>&- | grep init

Note that the stream redirections are always reset to their initial state if you’re using them in a command. You’ll only need to do this manually if you made a redirect using, for example, exec, as redirects made this way will last until changed manually.

Source (Pinhead.tv)

NMAP(Network Mapping) is one of the important network monitoring tool. Which checks for what ports are opened on a machine.
Some important to note about NMAP

• NMAP abbreviation is network mapper
• NMAP is used to scan ports on a machine, either local or remote machine (just you require IP/hostname to scan).
• NMAPis can be installed on windows, Sun Solaris machines too.
• NMAPcan be used to scan large networks, remember I am saying large networks.
• NMAPcan be used to get operating system details such as open ports, software used for a service and its version no, vendor of network card and up time of that system too(Don’t worry we will see all these things in this post.
• Please do not try to use NMAP on machines which you don’t have permission.
• Can be used by hackers to scan for systems for vulnerability.
• Just a funny note : You can see this NMAP used by Trinity in Matrix-II movie, when she tries to hack in to electric grid super computer.

Note : MAN pages of NMAP is one of the best man pages I have come across. It is explained in such a way that even new user can understand what each option do and one more thing is that, it even have examples in to on how to use NMAP in different situations, when you have time read it. You will get lots of information.

Let us start with some examples to better understand nmap command:

1. Check for particular port on local machine.
2. Use nmap to scan local machine for open ports.
3. Nmap to scan remote machines for open ports.
4. Nmap to scan entire network for open ports.
5. Scan only ports with -F option.
6. Scan a machine with -v option for verbose mode.
7. Scan a machine for TCP protocol open ports.
8. Scan a machine for UDP protocol open ports.
9. Scan a machine for services and their software versions.
10. Scan for open Protocols such as TCP, UDP, ICMP, IGMP etc on a machine.
11. Scan a machine for to check what operating system its running.

Example1 : Scanning for a single port on a machine

nmap –p portnumber hostname

Example:

nmap -p 53 192.168.0.1

Starting Nmap 5.21 ( http://nmap.org )
Nmap scan report for localhost (192.168.0.1)
Host is up (0.000042s latency).
PORT STATE SERVICE
53/tcp open domain

Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

The above example will try to check 53(DNS) port is open on 192.168.0.1 port or not.

Example2 : Scan entire machine for checking open ports.

nmap hostname

Example:

nmap 192.168.0.1

Starting Nmap 5.21 ( http://nmap.org )
Nmap scan report for localhost (192.168.0.1)
Host is up (0.00037s latency).
Not shown: 998 closed ports
PORT STATE SERVICE
53/tcp open domain
631/tcp open ipp

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

Example3 : Scan remote machine for open ports

nmap remote-ip/host

Example:

nmap 192.168.0.2

Starting Nmap 5.21 ( http://nmap.org )
Nmap scan report for localhost (192.168.0.2)
Host is up (0.00037s latency).
Not shown: 998 closed ports
PORT STATE SERVICE
53/tcp open domain
631/tcp open ipp

Nmap done: 1 IP address (1 host up) scanned in 0.08 seconds

Example4: Scan entire network for IP address and open ports.

nmap network ID/subnet-mask

Example:

nmap 192.168.1.0/24

Starting Nmap 5.21 ( http://nmap.org )
Nmap scan report for 192.168.1.1
Host is up (0.016s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
23/tcp open telnet
53/tcp open domain
80/tcp open http
5000/tcp open upnp

Nmap scan report for 192.168.1.2
Host is up (0.036s latency).
All 1000 scanned ports on 192.168.1.2 are closed

Nmap scan report for 192.168.1.3
Host is up (0.000068s latency).
All 1000 scanned ports on 192.168.1.3 are closed

Nmap done: 256 IP addresses (3 hosts up) scanned in 22.19 seconds

Example5: Scan just ports, dont scan for IP address, hardware address, hostname, operating system name, version, and uptime etc. It’s very much fast as it said in man pages etc. We observed in our tests that it is 70% faster in scan ports when compared to normal scan.

nmap –F hostname

-F is for fast scan and this will not do any other scanning.

Example:

nmap -F 192.168.1.1

Starting Nmap 5.21 ( http://nmap.org ) 
Nmap scan report for 192.168.1.1
Host is up (0.028s latency).
Not shown: 96 closed ports
PORT STATE SERVICE
23/tcp open telnet
53/tcp open domain
80/tcp open http
5000/tcp open upnp

Nmap done: 1 IP address (1 host up) scanned in 0.10 seconds

Example6: Scan the machine and give as much details as possible.

nmap -v hostname

Example:

nmap -v 192.168.1.1

Starting Nmap 5.21 ( http://nmap.org )
Initiating Ping Scan at 13:31
Scanning 192.168.1.1 [2 ports]
Completed Ping Scan at 13:31, 0.00s elapsed (1 total hosts)
Initiating Parallel DNS resolution of 1 host. at 13:31
Completed Parallel DNS resolution of 1 host. at 13:31, 0.00s elapsed
Initiating Connect Scan at 13:31
Scanning 192.168.1.1 [1000 ports]
Discovered open port 53/tcp on 192.168.1.1
Discovered open port 80/tcp on 192.168.1.1
Discovered open port 23/tcp on 192.168.1.1
Discovered open port 5000/tcp on 192.168.1.1
Completed Connect Scan at 13:31, 0.21s elapsed (1000 total ports)
Nmap scan report for 192.168.1.1
Host is up (0.014s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
23/tcp open telnet
53/tcp open domain
80/tcp open http
5000/tcp open upnp

Read data files from: /usr/share/nmap
Nmap done: 1 IP address (1 host up) scanned in 0.26 seconds

 Example7 : Scan a machine for TCP open ports

nmap –sT hostname

Here s stands for scanning and T is for scanning TCP ports only

Example:

nmap -sT 192.168.1.1

Starting Nmap 5.21 ( http://nmap.org )
Nmap scan report for 192.168.1.1
Host is up (0.022s latency).
Not shown: 996 closed ports
PORT STATE SERVICE
23/tcp open telnet
53/tcp open domain
80/tcp open http
5000/tcp open upnp

Nmap done: 1 IP address (1 host up) scanned in 0.28 seconds

Example8 : Scan a machine for UDP open ports.

nmap –sU hostname

Here U indicates UDP port scanning. This scanning requires root permissions.

Exmaple9 : Scanning for ports and to get what is the version of different services running on that machine

nmap –sV hostname

s stands for scaning and V indicates version of each network service running on that host

Example:

nmap -sV 192.168.1.1

Starting Nmap 5.21 ( http://nmap.org )
Stats: 0:00:06 elapsed; 0 hosts completed (1 up), 1 undergoing Service Scan
Service scan Timing: About 0.00% done
Nmap scan report for localhost (192.168.1.1)
Host is up (0.000010s latency).
Not shown: 998 closed ports
PORT STATE SERVICE VERSION
53/tcp open domain dnsmasq 2.59
631/tcp open ipp CUPS 1.5

Service detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 6.38 seconds

Example10 : To check which protocol(not port) such as TCP, UDP, ICMP etc is supported by the remote machine. This -sO will give you the protocol supported and its open status.

#nmap –sO hostname

Example:

nmap -sO localhost

Starting Nmap 5.21 ( http://nmap.org )
Nmap scan report for localhost (127.0.0.1)
Host is up (0.14s latency).
Not shown: 249 closed protocols
PROTOCOL STATE SERVICE
1 open icmp
2 open igmp
6 open tcp
17 open udp
103 open|filtered pim
136 open|filtered udplite
255 open|filtered unknown

Nmap done: 1 IP address (1 host up) scanned in 2.57 seconds

 Example11 : To scan a system for operating system and uptime details

nmap -O hostname

-O is for operating system scan along with default port scan

Example:

nmap -O google.com

Starting Nmap 5.21 ( http://nmap.org ) 
Nmap scan report for google.com (74.125.236.168)
Host is up (0.021s latency).
Hostname google.com resolves to 11 IPs. Only scanned 74.125.236.168
rDNS record for 74.125.236.168: maa03s16-in-f8.1e100.net
Not shown: 997 filtered ports
PORT STATE SERVICE
80/tcp open http
113/tcp closed auth
443/tcp open https
Device type: general purpose|WAP
Running (JUST GUESSING) : FreeBSD 6.X (91%), Apple embedded (85%)
Aggressive OS guesses: FreeBSD 6.2-RELEASE (91%), Apple AirPort Extreme WAP v7.3.2 (85%)
No exact OS matches for host (test conditions non-ideal).

OS detection performed. Please report any incorrect results at http://nmap.org/submit/ .
Nmap done: 1 IP address (1 host up) scanned in 16.23 seconds

Some sites to refer (not for practical examples, but for to get good concept):

nmap.org : official site for ourNMAP
en.wikipedia.org/wiki/Nmap

sysbench is a benchmark suite which allows you to quickly get an impression about system performance which is important if you plan to run a database under intensive load. This article explains how to benchmark your CPU, file IO, and MySQL performance with sysbench.

I do not issue any guarantee that this will work for you!

1 Installing sysbench

On Debian/Ubuntu, sysbench can be installed as follows:

apt-get install sysbench

Take a look at

man sysbench

to learn more about its parameters.

I will now perform simple three basic tests which tell you more about your CPU performance, file IO performance, and MySQL performance.

2 CPU Benchmark

You can benchmark your CPU performance as follows:

sysbench –test=cpu –cpu-max-prime=20000 run

root@server1:~# sysbench –test=cpu –cpu-max-prime=20000 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 20000

Test execution summary:
total time:                          23.8724s
total number of events:              10000
total time taken by event execution: 23.8716
per-request statistics:
min:                                  2.31ms
avg:                                  2.39ms
max:                                  6.39ms
approx.  95 percentile:               2.44ms

Threads fairness:
events (avg/stddev):           10000.0000/0.00
execution time (avg/stddev):   23.8716/0.00

root@server1:~#

You see a lot of numbers, the most important of it is the total time:

total time:                          23.8724s

Of course, you must compare benchmarks across multiple systems to know what these numbers are worth.

3 File IO Benchmark

To measure file IO performance, we first need to create a test file that is much bigger than your RAM (because otherwise the system will use RAM for caching which tampers with the benchmark results) – 150GB is a good value:

sysbench –test=fileio –file-total-size=150G prepare

Afterwards, we can run the benchmark:

sysbench –test=fileio –file-total-size=150G –file-test-mode=rndrw –init-rng=on –max-time=300 –max-requests=0 run

root@server1:~# sysbench –test=fileio –file-total-size=150G –file-test-mode=rndrw –init-rng=on –max-time=300 –max-requests=0 run
sysbench: /usr/lib/libmysqlclient.so.18: no version information available (required by sysbench)
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 1
Initializing random number generator from timer.

Extra file open flags: 0
128 files, 1.1719Gb each
150Gb total file size
Block size 16Kb
Number of random requests for random IO: 0
Read/Write ratio for combined random IO test: 1.50
Periodic FSYNC enabled, calling fsync() each 100 requests.
Calling fsync() at the end of test, Enabled.
Using synchronous I/O mode
Doing random r/w test
Threads started!
Time limit exceeded, exiting…
Done.

Operations performed:  600 Read, 400 Write, 1186 Other = 2186 Total
Read 9.375Mb  Written 6.25Mb  Total transferred 15.625Mb  (53.316Kb/sec)
3.33 Requests/sec executed

Test execution summary:
total time:                          300.0975s
total number of events:              1000
total time taken by event execution: 158.7611
per-request statistics:
min:                                  0.01ms
avg:                                158.76ms
max:                               2596.96ms
approx.  95 percentile:             482.29ms

Threads fairness:
events (avg/stddev):           1000.0000/0.00
execution time (avg/stddev):   158.7611/0.00

root@server1:~#

The important number is the Kb/sec value:

Read 9.375Mb  Written 6.25Mb  Total transferred 15.625Mb  (53.316Kb/sec)

After the benchmark, you can delete the 150GB test file from the system:

sysbench –test=fileio –file-total-size=150G cleanup

4 MySQL Benchmark

To measure MySQL performance, we first create a test table in the database test with 1,000,000 rows of data:

sysbench –test=oltp –oltp-table-size=1000000 –mysql-db=test –mysql-user=root –mysql-password=yourrootsqlpassword prepare

root@server1:~# sysbench –test=oltp –oltp-table-size=1000000 –mysql-db=test –mysql-user=root –mysql-password=yourrootsqlpassword prepare
sysbench 0.4.12: multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Creating table ‘sbtest’…
Creating 1000000 records in table ‘sbtest’…
root@server1:~#

Afterwards, you can run the MySQL benchmark as follows:

sysbench –test=oltp –oltp-table-size=1000000 –mysql-db=test –mysql-user=root –mysql-password=yourrootsqlpassword –max-time=60 –oltp-read-only=on –max-requests=0 –num-threads=8 run

root@server1:~# sysbench –test=oltp –oltp-table-size=1000000 –mysql-db=test –mysql-user=root –mysql-password=yourrootsqlpassword –max-time=60 –oltp-read-only=on –max-requests=0 –num-threads=8 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

No DB drivers specified, using mysql
Running the test with following options:
Number of threads: 8

Doing OLTP test.
Running mixed OLTP test
Doing read-only test
Using Special distribution (12 iterations,  1 pct of values are returned in 75 pct cases)
Using ”BEGIN” for starting transactions
Using auto_inc on the id column
Threads started!
Time limit exceeded, exiting…
(last message repeated 7 times)
Done.

OLTP test statistics:
queries performed:
read:                            2253860
write:                           0
other:                           321980
total:                           2575840
transactions:                        160990 (2683.06 per sec.)
deadlocks:                           0      (0.00 per sec.)
read/write requests:                 2253860 (37562.81 per sec.)
other operations:                    321980 (5366.12 per sec.)

Test execution summary:
total time:                          60.0024s
total number of events:              160990
total time taken by event execution: 479.3419
per-request statistics:
min:                                  0.81ms
avg:                                  2.98ms
max:                               3283.40ms
approx.  95 percentile:               4.62ms

Threads fairness:
events (avg/stddev):           20123.7500/63.52
execution time (avg/stddev):   59.9177/0.00

root@server1:~#

The important number is the transactions per second value:

transactions:                        160990 (2683.06 per sec.)

To clean up the system afterwards (i.e., remove the test table), run:

sysbench –test=oltp –mysql-db=test –mysql-user=root –mysql-password=yourrootsqlpassword cleanup

5 Links

• sysbench: http://sysbench.sourceforge.net/

Using Rex To Automate Your Datacentre

In my day to day work I come across a lot of tasks that I actually “hate” to be honest. Most of these tasks consist of things that are repetitive and take up a lot of time. A co-worker of mine once told me, “if you have to do a task more than once, automate it!”. This is one piece of advice that intrigued me. Ever since that day I have been thinking about automation and how to make processes flow smoother than they did before. But the main question always is, what tool are you going to use for that?

Puppet, Chef, CFEngine etc. etc.

As a system or hosting engineer or whatever normally does is first type in “datacentre automation” in Google and what you get is a lot of commercial companies trying to sell you automation frameworks. Things like Puppet, Chef etc. come along and when you look at their documentation, well, it’s huge and takes a lot of time to install and configure properly. Not to mention that you need to install a daemon on every server to get the framework working (imagine trying to do that on 40-50 servers, and imagine if you have like 100-200 servers. So that wasn’t actually an option for me at the time of my quest to find an easy framework to automate my tasks.

Rex

Rex is a small and lightweight framework I came across visiting a website (I forgot which one it was). At first I wasn’t to sure what to make up out of it, but while browsing trough the documentation I got excited and saw the possibilities of using it. Within minutes I got it up and running (thanks to their handy-dandy YUM repository (I use CentOS)), and the cool thing is, it’s written in Perl so in theory you could run it on any platform (Windows included). The big advantage to Rex is that you can use your ssh keys (or user-pass combo) from your “master” server where you have Rex installed and connect using ssh so you don’t have to install daemons on your target servers!

How Rex is structured

When you first create a Rex project (easy after you install it check the install docs: http://rexify.org/get/index.html) by using “rexify”. It will create a folder for you with a very clean structure:

• A file called a Rexfile, here you define all your hosts and usernames and passwords
• A lib folder with a perl module (.pm file) in it. It automatically creates a task for the default server group called uptime so you can test if your connection with your server group is working

Give me an example of what I could use?!

Well, everyone knows how important it is to keep all your servers up to date. So what I did was that I created a task that updates all my servers (using yum update) and checks if it succeeded, and reports to me by email, see an example below with the standard uptime task:

package task;
use Rex -base;
desc "Get uptime of server";
task "uptime", group => 'all', sub {
   say run "uptime";
};
desc "Run updates";
task "runupdate", group => 'all', sub {
        # Variables
        my $EMAIL='<your e-mail>';
        my $hostname = connection->server;
        # Execute yum update
        say "Executing yum update on $hostname";
        sudo "yum -y -e 0 -d 0 update";
        if($? == 0) {
             run 'echo "Update was successful" | mail -s "Update was completed on `hostname -s`" ' . $EMAIL;
        } else {
             run 'echo "Update was not successful" | mail -s "Update did not work on `hostname -s`" ' . $EMAIL;
        };
};

MORE?

I think to best understand the full potential you’re just going to have to try it out for yourself, install it and start experimenting. Check the rexify.org website for full documentation (it’s pretty well documented and if you don’t understand something just go to the #rex IRC channel on FreeNode, the developer is available there aswell). I would just like to take some time and thank jfried, the developer of this marvelous framework! Support and help out, and make this framework even bigger!

tmux and GNU Screen are well-known utilities which allow multiplexing of virtual consoles. Using either, it is possible to start a session, detach, move to a different machine and resume the session in uninterrupted progress. It’s also possible to use these tools to share a single session between more than one user at the same time.

Basic Sharing with a Single Account

If an account is held jointly between two or more users, then the sharing of the terminal console is very simple. Neither tmux nor screen require anything out of the ordinary for basic sharing between a single account logged in multiple times. Basic sharing is very easy if you are both logged in as the same user.

Basic sharing with screen

In one terminal create a new session for screen, where foobar is the name of your screen session:

screen -S foobar

Then in the other terminal, attach to that session.

screen -x foobar

That’s it, there were just two steps.

Basic sharing with tmux

Again, there are only two steps. In the first terminal, start tmux where shared is the session name:

tmux new-session -s shared

Then in the second terminal attach to the shared session.

tmux attach-session -t shared

That’s it.

Sharing Between Two Different Accounts

Sharing between two different accounts requires some additional steps to grant the privileges necessary for one account to access another’s session. In some cases, it will require help from the system administrator to prepare the setup.

Sharing between two different accounts with tmux

For different users, you have to set the permissions on the tmux socket so that both users can read and write it. There is only one prerequiste, that there be a group in common between the two users. If such a group does not exist it will be necessary to create one.

In the first terminal, start tmux where shared is the session name and shareds is the name of the socket:

tmux -S /tmp/shareds new -s shared

Then chgrp the socket to a group that both users share in common. In this example, joint is the group that both users share. If there are other users in the group, then they also have access. So it might be recommended that the group have only the two members.

chgrp joint /tmp/shareds

In the second terminal attach using that socket and session.

tmux -S /tmp/shareds attach -t shared

That’s it. The session can be made read-only for the second user, but only on a voluntary basis. The decision to work read-only is made when the second user attaches to the session.

tmux -S /tmp/shareds attach -t shared -r

Sharing between two different accounts with screen

If you are logged in as two different users, there are three prerequisites to using screen. First, screen must be set SUID and it is necessary to remove group write access from /var/run/screen. The safety of using SUID in this context is something to consider. Then you must use screen’s ACLs to grant permission to the second user.

sudo chmod u+s /usr/bin/screen
sudo chmod 755 /var/run/screen

In the first user’s terminal, start screen as in the basic sharing above, where foobar is the name of the screen session. Then turn on multiuser mode and add user2 to the ACL, where user2 is the second account to be sharing the session.

screen -S foobar
^A:multiuser on
^A:acladd user2

The session can be made read-only for the second user by entering the following ACL change: ^A:aclchg user2 -w “#?”

Then in the other terminal, attach to the first user’s session.

screen -x user1/foobar

It is also possible to put multiuser on and acladd user2 into .screenrc to have it take effect automatically upon starting screen. If the changes are not desired in all screen sessions, then a separate .screenrc configuration file can be specified by using the -c option when starting screen.

Afterthought

Be careful when exiting. If you just exit the shell, it will end the terminal session for all parties. If you instead detach the session then the other user can continue working uninterrupted. In tmux that is ^B-d and in screen that is ^A-d