System Notes Org

Here are a few notes to add to the previous article on ping. This time, we look at some bash one-liner tips and tricks. Combine multiple commands Return Values Ping Multiple Hosts Using Bash Nmap and Fping Ping, and Command Line Variables Start from the beginning. We want to see what happens when attempting to ping a host that resolves an IP address from DNS, but is not reachable from our network at the moment. This is to show how we might build a simple monitoring tool from scratch, and also to see what kind of fun we can have with the command line. To ping a host only one time use the count option, which in Windows is -n, and in Linux is -c. Windows C:\>set host=google.com C:\>ping -n 1 %host% Pinging google.com [216.58.219.46] with 32 bytes of data: Request timed out. Ping statistics for 216.58.219.46: Packets: Sent = 1, Received = 0, Lost = 1 (100% loss), Bash: Linux, Mac, Unix Note that in bash, we don't use the set command to assign a varia...

Here is an example of how to use awk to extract a column from a text file. The default field separator is a space, so this will work with some text files:    cat somefile.txt | awk '{print $2}' But, if the fields are separated by a delimiter, such as tab, or comma, just specify the field separator by using "-F" before the print statement    cat somefile.txt | awk -F"\t" '{print $2}'   Unix and linux distributions generally come with sed and awk, among other things. The problem. You want to get a list of companies to research before investing in them, and paste only the stock symbols into a Yahoo finance portfolio.   You run a stock screen on a site such as  magicformulainvesting.com , or investors.com but if you try to paste the text into a spreadsheet it may show up as a single row, with no way to extract the column. Here is the process 1.)  Run a Stock Screen Get output similar to this: Company Name (in alphabetical...

At SystemNotes, we love free software. Especially when it is Open Source (source code is readable and may be freely modified by the end users), and available on different platforms (Linux, Mac, Windows, iPad, Android, etc.). When preparing tax returns, many people turn to paid software, or tax professionals. There's nothing wrong with paying for a useful service or software, but linux geeks often turn to shell scripts, or whatever open source software they can find before shelling out any cash -- no pun intended. Here are some useful tools for preparing the US 1040 tax return and most common schedules: Open Tax Solver If your needs are simple, Open tax solver is quick and easy. There are even options for some state taxes. Open Tax Solver http://opentaxsolver.sourceforge.net There is a simple interface that takes inputs, and when finished, just click "Compute Tax." for the results. Microsoft Excel Spreadsheet Income Tax Calculator If you prefer to ha...

Simple ways to loop in bash shell scripts, or one-liners. Note that the variable in bash is declared without a "$", and called with it, but in perl the variable is declared with and called with a "$" at the beginning of the name. The Bash "for" Loop Loop a number of times for i in {1..10}; do echo -n "$i ";done;echo Output: 1 2 3 4 5 6 7 8 9 10 For short lists, the items can be entered manually. for num in 1 2 3;do echo -n $num;done;echo Output: 1 2 3 Loop Through Items on a List for host in `cat all.txt `;do ping -c 1 $host; done See also: How to Run a Bash Command on All Items in a List and: Ping Multiple Hosts Using Bash Nmap and Fping Loop using Perl This syntax should be familiar to C programmers. In Perl. the scalar variable does not have to be declared as it would in C (int $counter). Note that the variable name starts with "$" to indicate a scalar. #!/usr/bin/perl for($counter = 1; $counter <= 10; $counter++){ print ...

I wrote some shell scripts to generate Nagios configuration files, and I finally decided to publish them on Freshmeat . The initial version is crude, and has some hard-coded information, but it is still better than trying to edit the .cfg files by hand. Here is a brief description nagios-config-build is a group of shell scripts that generate Nagios configuration files from a list of hostnames. It automatically resolves IP addresses from DNS, and provides a quick way to manage a large number of hosts without having to set up a complicated system. Nagios is more than just a server monitoring tool. From the Nagios website: " Nagios is a powerful monitoring system that enables organizations to identify and resolve IT infrastructure problems before they affect critical business processes." Systemnotes.org download: http://www.systemnotes.org/download/shell/nagios-config-build.html Freshmeat project: http://freshmeat.net/projects/nagios-config-build Please feel free to use thi...

I mentioned previously that you can reset a shell by typing "reset." There is another problem that can cause a shell or xterm to freeze. If you ssh to a remote host, and then logout, sometimes the session will freeze. In that case, all you need to do is type "Enter", "~", "." (Enter, Tilde, Dot with no spaces) to get the console back. That's the Enter key, immediately followed by tilde "~", and then a period "." [Enter]~. Also, a convenient way to exit a shell, xterm, or remote ssh terminal is to use Ctrl-D rather than taking the time to type the word "exit."

The idea to write a shell script is often the result of having to do repeated tasks. Imagine having to check hundreds of domain names by typing "whois domain.com" each time. It would take much longer than writing a simple script. Using whois to Check for Available Domain Names Suppose you want to find a good domain name, but you want to see if all of the domain suffixes are available? For example, you want a good .com domain, but you also want the .net, and .org. Not only that, but you might want to use blogspot.com, or wordpress.com for your blog. You can always check a domain availability by using the whois command, e.g. $ whois google.com Obviously, google.com is not available, so whois will display information about the company that registered the domain name. If the domain is not taken, then whois will usually return "No match", or "NOT FOUND". You have to keep trying a domain name until you find one that is available. This can be time-consumi...

Here is a very simple way to backup and compress subdirectories. Normally you would want to copy them to a remote location in case something happens to the original drive, but in this case the concern is that disk space is filling up. Suppose you have a program that is generating log files every day, let's say they are key performance indicators (kpi) for some system that you are testing. The log files are pure text, but after a while, they take up a lot of space. One way to compress text is to use gzip, but since tar has an option (-z) to tar and gzip all at once, it may be one of the easiest solutions. #!/bin/sh # compress logs in /opt/multicast_kpi/record # by ScottM, 20080613 # Set variables TODAY = " `date +%m%d%Y` " ARCHIVE_DIR =/opt/multicast_kpi/record # move to working directory cd $ARCHIVE_DIR # For each subdirectory, make a .tgz file of it's contents, and then delete the originals. for dir in `ls -1 $ARCHIVE_DIR | grep -v " .tgz " | gre...

To copy all files from /usr/local/source to /usr/local/destination, there are several options. Tar cd /usr/local/source tar -cvf - . | (cd /usr/local /destination ; tar -xvf -) This method uses tar, but rather than create a file, it writes to a stream "-" in std output. The stream is piped to another tar command, after changing to another directory. cp -a cp -a /usr/local/src/* /usr/local/destination Other options: cpio dd

Would you like an easy way to send files to someone outside of your local network, or quickly test a mail server? Well the old "mail" command is still available, and it comes in handy for sending config files, or using within a shell script. This is all you need: $ mail -s test me@company.com That's the shortcut. If you want more details, you can read the rest of the post... Using the mail command 1) Use Mail Interactively to read mail a) type Mail b) enter the number of the message to read, press enter c) press space to page down, n for next message d) ? for help e) q to quit 2) Use mail Interactively to send mail a) type mail command followed by email address b) Enter subject, press enter c) Enter text of message d) press Ctrl-d on a line by itself when finished e) Enter CC: if desired, or press Ctrl-d again $ mail user@company.com Subject: test Here is my test message <Ctrl-d> Cc: <Ctrl-d>$ For a little less work, add the subject to the command prompt. ...

I explained how to get a list of hosts using nmap -- using-nmap-to-generate-host-lists.html , but here is another look at the subject. The question is, how do I ping multiple hosts to find out which ones are down? Sure this could be considered a topic of system monitoring, but maybe you just want to reboot a bunch of machines, and make sure they all come back online. This quick check will tell you whether there is a problem or not. Here are three methods for pinging a list of hosts: 1.) for host in `cat all.txt `;do ping -c 1 $host; done 2.) nmap -sP -R -iL all.txt 3.) sudo fping -u < xyz/all.txt First, we assume that you have a text file named all.txt that contains a list of hostnames, one per line. Obviously, the examples here contain fake hostnames, domain names and IPs, as described in another article about vi: vim-tips-search-and-replace.html . --- all.txt --- xyz-1 xyz-2 xyz-3 xyz-4 xyz-5 xyz-6 xyz-7 xyz-8 --- end all.txt --- 1.) Use a for loop in ...

Did you ever cat a file that caused the terminal to display all kinds of strange characters? Pressing Ctrl-C and Enter, or typing clear doesn't get the console back to normal. The characters are there, but they are unreadable. A quick way to fix this is to use "reset" -- type: $ reset That should bring the console back to normal. Of course, if you just want to clear the screen, you can type "clear" ("cls" in DOS, or a Windows command prompt).

The command history is stored in ~/.bash_history. The history typically contains the last 500 commands typed at the shell prompt. You can view the history by using the history command. history | less Then you can search for a command. e.g. what was the name of that directory I created yesterday? Oh, yeah, I can search history for the mkdir command by using the slash. /mkdir You could also use grep, but with less, you can see the surrounding commands, as you often want to see the sequence of what was done. History Shortcuts !! - Executes the last command !902 - Executes command number 902 (from the history list) !service - Executes the command whose string matches the most recent history entry Up / Down - The most common shortcut is to simply use the up and down arrows to scroll through the bash history. Example If you recently restarted the network service, and then wanted to restart it again, you might do something like this; service network restart ping somehost.example.c...

For Linux / Unix Follow as instructed For Windows Download cygwin from http://cygwin.com For instructions, see this article: http://systemnotesorg.blogspot.com/2007/04/use-cygwin-to-run-linux-on-windows.html Two Easy Steps for One-Liners e.g., you have a list of servers, and would like to do something, such as ping each one, or check their IP address. Here is a quick two-step process, with a one-line shell script that can be run from the command line. 1) Create a Text file, with one hostname per line 2) Run a for loop on the file $ cat servers.txt server01 server02 server03 $ for host in `cat servers.txt`;do host $host;done server01.example.com has address 10.10.10.10 server02.example.com has address 10.10.10.11 server03.example.com has address 10.10.10.12 $ If you want different output, you can use awk, but that's another topic. Notice a few things about this one line script. 1) Each command is separated by a semicolon ";" 2) The ba...

This shell script demonstrates how to write to a text file, and then modify the contents. #!/bin/sh # modfile.sh # by ScottM, 04/12/2007 # demonstrates writing text to a file, and then using sed to modify it. TESTFILE=test.txt FRUIT=banana # add some content to the file (note: file will be overwritten) echo "apple" > $TESTFILE # modify the content sed -e "s/apple/& $FRUIT/g" -i $TESTFILE # sed uses the "s" option, which uses regular expressions to search and replace text # "s/apple/" means search for any lines that contain the characters "apple" # "& " means use the results of whatever was found # "/& $FRUIT/g" -- replace "apple" with "apple banana", # the g is for global, or all lines containing the pattern # output: # $ cat test.txt # apple banana # $ Sed One Liner This is really only a one line script, commonly referred to as a on-liner, so we don't really need a...

An easy way to get a list of hosts from a single domain that you are a part of, is to query DNS host -l mydomain.com But that is not always practical. Sometimes you have machines that are in different domains, but they all are part of a network you manage. Rather than trying write a script that pings hosts and reports the output, just use nmap for a very fast scan. To scan all hosts in a list of subnets 1) Create a subnets.dat file with one subnet on each line: $ cat subnets.dat 192.168.0.* 192.168.1.* 2) Run nmap with the subnets.dat file as input $ nmap -sP -R -iL subnets.dat Reading target specifications from FILE: subnets.dat Starting nmap V. 3.00 ( www.insecure.org/nmap/ ) Host (192.168.0.0) appears to be down. Host box1.mydomain.com (192.168.0.1) appears to be up. Host box2.mydomain.com (192.168.0.2) appears to be down. Host box3.mydomain.com (192.168.0.3) appears to be up. Host (192.168.0.4) appears to be up. ... Notice how names are resolved ...

How do you get the current date to use in a Linux / Unix shell script? Take a look at this simple backup script: ---------------------------------------------------- #!/bin/sh export   FILE_DATE = `date +%Y%m%d` export   backup_file =backup_ ${ FILE_DATE } .tgz [   !   -d  /opt/backups ]   &&  mkdir -p  /opt/backups tar -zcvf  /opt/backups/ ${ backup_file }  /var/named ---------------------------------------------------- What does that FILE_DATE variable do for me? Well, let's try it on the command line. bash-2.05b$ export FILE_DATE=`date +%Y%m%d` bash-2.05b$ echo $FILE_DATE 20070323 Ah, so that's how we get the current date in a usable format. This method makes the files sortable later. Of course, if you need more than one backup per day, you can always add more date options. Notice a few things about this method of scripting. 1) export, is optional, but it makes the environmet variable available to other s...