Wednesday, November 17, 2010

bash loops


Simple ways to loop in bash shell scripts, or one-liners.

Note that the variable in bash is declared without a "$", and called with it, but in perl the variable is declared with and called with a "$" at the beginning of the name.

The Bash "for" Loop

Loop a number of times

for i in {1..10}; do echo -n "$i ";done;echo

Output:

1 2 3 4 5 6 7 8 9 10


For short lists, the items can be entered manually.

for num in 1 2 3;do echo -n $num;done;echo

Output:

1 2 3


Loop Through Items on a List

for host in `cat all.txt `;do ping -c 1 $host; done
See also: How to Run a Bash Command on All Items in a List
and: Ping Multiple Hosts Using Bash Nmap and Fping


Loop using Perl

This syntax should be familiar to C programmers. In Perl. the scalar variable does not have to be declared as it would in C (int $counter). Note that the variable name starts with "$" to indicate a scalar.


#!/usr/bin/perl

for($counter = 1; $counter <= 10; $counter++){
print "for loop #$counter\n";
}

Wait a minute. Isn't this article about bash and one-liners?

O.K., you can run perl for loops from a one-liner at a bash prompt, or within a shell script.

perl -e for($counter = 1; $counter <= 10; $counter++){
print "for loop #$counter\n";
}


For more detailed scripting ideas, check out the Linux Documentation Project's "Advanced Bash Scripting Guide" -- http://www.tldp.org/LDP/abs/html

Friday, October 8, 2010

Yum Repositories for RHEL 5

Sometimes we need a RHEL5 Repository that includes software not released by Redhat. It may be that we need a newer version, or simply something that was not included in our installation, and is not part of the standard redhat repositories.

Missing RPM

Have you ever tried to install something with yum, only to find that the package is not available?

$sudo yum -y install vlc
Password:
Server | 1.3 kB 00:00
Updates | 951 B 00:00
Workstation | 1.1 kB 00:00
addons | 951 B 00:00
base | 2.1 kB 00:00
extras | 2.1 kB 00:00
update | 1.9 kB 00:00
Excluding Packages in global exclude list
Finished
Setting up Install Process
No package vlc available.
Nothing to do
$

Install New Repos

The solution is simple, if you add the repo files for repositories that contain the missing package.

Repo files are located in the /etc/yum.repos.d directory.

$ pwd
/etc/yum.repos.d

Here are some of my favorites:


Quick Install

A quick way to install the repoforge settings, is to download the package for your distro, and install the it.  That will place the required files in the /etc/yum.repos.d/ directory.

  http://repoforge.org/use/
  rpm -Uvh rpmforge-relese-x.x.x-x....

e.g., for RHEL5, i686:
$ wget http://pkgs.repoforge.org/rpmforge-release/rpmforge-release-0.5.2-2.el5.rf.i386.rpm
$ sudo rpm -Uvh rpmforge-release-0.5.2-2.el5.rf.x86_64.rpm

Dag Wieers' repository

Much of the software you need can be found here.

$ cat r5.dag.repo

[dag]
name=Red Hat Enterprise Linux 5 Dag Wieers' repository
baseurl=http://apt.sw.be/redhat/el5/en/i386/dag/
http://ftp.heanet.ie/pub/freshrpms/pub/dag/redhat/el5/en/i386/dag/

CentOS-5

Very useful for missing libraries. Be aware that this may replace some redhat libraries, but it should be binary compatible.

$ cat CentOS-Base.repo
[base]
name=CentOS-5 - Base
#mirrorlist=http://mirrorlist.centos.org/?release=$releasever5&arch=$basearch&
repo=os
#baseurl=http://mirror.centos.org/centos/$releasever/os/$basearch/
baseurl=http://mirrors.163.com/centos/5/os/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

#released updates
[update]
name=CentOS-5 - Updates
#mirrorlist=http://mirrorlist.centos.org/?release=4&arch=$basearch&repo=updates
baseurl=http://mirrors.163.com/centos/5/updates/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

#packages used/produced in the build but not released
[addons]
name=CentOS-5 - Addons
#mirrorlist=http://mirrorlist.centos.org/?release=4&arch=$basearch&repo=addons

baseurl=http://mirrors.163.com/centos/5/addons/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

#additional packages that may be useful
[extras]
name=CentOS-5 - Extras
#mirrorlist=http://mirrorlist.centos.org/?release=4&arch=$basearch&repo=extras

baseurl=http://mirrors.163.com/centos/5/extras/$basearch/
gpgcheck=1
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

#additional packages that extend functionality of existing packages
[centosplus]
name=CentOS-5 - Plus
#mirrorlist=http://mirrorlist.centos.org/?release=4&arch=$basearch&repo=centosplus
baseurl=http://mirrors.163.com/centos/5/centosplus/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

#contrib - packages by Centos Users
[contrib]
name=CentOS-5 - Contrib
#mirrorlist=http://mirrorlist.centos.org/?release=4&arch=$basearch&repo=contrib
baseurl=http://mirrors.163.com/centos/5/contrib/$basearch/
gpgcheck=1
enabled=0
gpgkey=http://mirror.centos.org/centos/RPM-GPG-KEY-CentOS-5

ATrpms

$ cat atrpm.repo
[atrpms]
name=Red Hat Enterprise Linux 5 $releasever - $basearch - ATrpms
baseurl=http://dl.atrpms.net/el$releasever-$basearch/atrpms/stable
gpgkey=http://ATrpms.net/RPM-GPG-KEY.atrpms
gpgcheck=1

RPMforge

$ cat rpmforge.repo
### Name: RPMforge RPM Repository for Red Hat Enterprise 5 - dag
### URL: http://rpmforge.net/
[rpmforge]
name = Red Hat Enterprise $releasever - RPMforge.net - dag
baseurl = http://apt.sw.be/redhat/el5/en/$basearch/rpmforge
mirrorlist = http://apt.sw.be/redhat/el5/en/mirrors-rpmforge
#mirrorlist = file:///etc/yum.repos.d/mirrors-rpmforge
enabled = 1
protect = 0
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rpmforge-dag
gpgcheck = 1

$ cat rpmforge-testing.repo
### Name: RPMforge RPM Repository for Red Hat Enterprise 5 - dag
### URL: http://rpmforge.net/
[rpmforge-testing]
name = Red Hat Enterprise $releasever - RPMforge.net - test
baseurl = http://apt.sw.be/redhat/el5/en/$basearch/test
enabled = 0
protect = 0
gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-rpmforge-dag
gpgcheck = 0

How to Add Repositories

You may be wondering how to make use of these repos. Simply create the files listed above with a text editor, and place them in the /etc/yum.repos.d directory.

Now when you use yum to install, there should be more rpms available, and less dependency issues. For example, the following should now work on a RHEL5 system with no errors. (This was tested on RHEL5U2 and RHEL5U4).

$sudo yum -y install vlc

Monday, September 27, 2010

Cisco CCNA Exam Notes - Network Simulator

When studying for a Cisco certification such as CCNA, CCNP, CCIP, or CCIE, there is nothing like having real routers and switches to play with, but it can get expensive to build a lab to study with.

There is an excellent simulator that is free, but the users must supply their own Cisco IOS software. Once it is set up, the simulated routers, switches, and firewalls act as if they were real hardware. That gives the user flexibility to play with the real Cisco IOS from command prompts on the simulated hardware.


What is GNS3?

GNS3 is a graphical network simulator that allows you to design complex network topologies. You may run simulations or configure devices ranging from simple workstations to powerful Cisco routers. It is based on Dynamips, an IOS emulator which allows users to run IOS binary images from Cisco Systems and Pemu, a PIXOS emulator to emulate PIX firewalls.


Where Can I Download GNS3?

http://www.gns3.net


How To Configure GNS3

This installation assumes you have a cisco router image (c3640), and a pix firewall image pix722-10.bin. You're on your own for finding these, but check with http://cisco.com, or ask you local network administrator where you can get them.

This is how I set it up on Windows XP:

1.) Install GNS3.
c:\cisco\GNS

2.) Put the two files in a subdirectory
c:\cisco\GNS\images

3.) Run the GNS3 gui, and configure the router.
- IOS images and hypervisors
-- Select the image file (browse to c:\cisco\GNS\images for the c3640 image)
Leave defaults for RAM, etc. Set working directory, if desired.

4.) Set up PIX (not in the IOS images screen)
Go to Edit / Preferences
Select Pemu
Browse to PIX image file (C:\cisco\GNS\images\pix722-10.bin)
Select Apply or OK
Set working directory, if desired

5.) Restart GNS3


Running GNS3

  1. Drag a Router c3600 icon to the middle screen
  2. Right-click on the R0 Image, and select start
  3. Right-click on the R0 Image, and select console
  4. Click inside the console, and press enter
You should be at a router prompt.

After going through some setup prompts that come from the configure command, there should be a prompt such as:
 Press RETURN to get started!

Router>en
Password:
Router#sh run
Building configuration...

Current configuration : 498 bytes
!
version 12.1
no service single-slot-reload-enable
service timestamps debug uptime
service timestamps log uptime
no service password-encryption
no service dhcp
!
hostname Router
!
enable secret 5 $1$q1UW$mtKU87u8.A8a8BLxsNGyB/
enable password enable
!
!
!
!
!
ip subnet-zero
!
!
!
--More--
!
!
!
ip local pool setup_pool 10.10.10.1 10.10.10.100
ip classless
ip http server
!
dialer-list 1 protocol ip permit
dialer-list 1 protocol ipx permit
!
line con 0
line aux 0
line vty 0 4
password vt
login
!
end

Router#

There you have it. A virtual Cisco router to configure as you like.

Friday, June 18, 2010

Data Recovery: 5 Things You Must Know

Hard drive data recovery is one of the most difficult tasks a sysadmin can attempt to perform, so it should be considered a last resort that, after many frustrating hours, may not even work at all. Data Recovery or Disaster Recovery can be thought of in several categories, and it is important to have a plan in place, and know the options before resorting to one of your local hard disk data recovery services. Many people fail to consider the value of their data until it is too late. This article discusses general principles, with some specific examples from experience with Novel, Microsoft, and Linux operating systems. Detailed tool discussions will be reserved for separate posts.

Here is a short list of categories, roughly in order of increasing level of difficulty to recover.

  • Backups and off-site storage
  • RAID
  • Documents
  • File Systems
  • Data Recovery from Hard Drives

There is software available that can recover from several types of failures, but it does not always work when there is corruption, or failure of disk drives, media or other hardware. Not only that, but recovery can cost hundreds of dollars for each disk, and is not guaranteed to recover any data. This is why backups are essential. If you are not a sysadmin, and just have a single computer to work on, there is probably some data that you consider important. If nothing else, copy it to another location such as a USB drive, another computer, a CD, or an ftp site.

Reliable Backups

Obviously, when there are reliable backups, there should be little difficulty in restoring the data, but it may not be as easy as it sounds. Some of the problems with backup and restore procedures could be:

  • The data may be outdated.
  • The missing data may not have been backed up yet.
  • The tape or other media may fail.
  • The tape, or media may not be found because it was misfiled, mislabeled, or too old to keep.
  • The time lost in restoring data may be important.
  • The backup software may have failed, or not been setup properly.
  • Directories, drives, or systems may not have been selected to be backed up.
  • The backed up files may have been corrupted, or infected with a virus.
An important part of disaster recovery is reliable backups. How can the backups be verified? Try a restore once in a while to make sure it is working properly. I have seen restores fail when backed up on one file system, and restored to another. The file was restored, but the data was corrupt. It took a while to figure out, but when an Excel file was backed up from Netware and restored to NT some of the cells were corrupted. Another time a problem was noticed when attempting to restore. Some new directories had not been selected to be backed up. Sometimes software will allow you to select directories, but will not automatically include subdirectories. These are the types of things that should be tested by attempting to restore once in a while.


RAID Systems

One of the best inventions for sysadmins was the RAID system. RAID, or a Redundant Array of Inexpensive Disks provides a means to quickly recover data in the event of a drive failure. In normal operation, systems will use RAID 5, which includs three or more disks in an array. When any one drive fails, the system continues to function normally, and the drive can be replaced at the leisure of the system administrator. If it was on good hardware, there is usually no downtime. The drive can be "hot-swapped", meaning that the defective drive can be removed, the replacement drive inserted, and the array rebuilt -- all without taking down the computer system.

Here is an example of how RAID arrays saved time when moving servers. We had a couple dozen servers to move across town, and were concerned about the data on the drives as the servers were moved. There were reliable weekly and daily backups, but the current day had not been backed up. The standard tape backup process would take several hours. The solution was to remove a drive from each system, and replace it with a spare. How did that work? Each system had a RAID 1 array, which consists of two mirrored SCSI drives. Removing one drive gave us an instant backup that could be used to restore the array in case the other drive failed while being transported. Replacing the drive with a spare caused the array to rebuild in a matter of minutes, so that there was even less chance of failure. Of course we were careful in moving the hardware, and none of the drives failed, but we saved several hours of time-wasting tape backups. All we had to do was reconnect the servers, and assign new IP's of course, and then let the nightly tape backup continue as scheduled.

Document Autosave

Documents are in a little different category of data recovery, since we are discussing recovery of documents that have not been backed up by the daily backup process. It is common to be working on a document during the day, and have backups run at night. If there is a corruption, or crash while working on a new document, there may be a way to recover the most recent work. We will look at vi as an example, but other text editing, and word processing software often have a similar feature.

Using Autosave in vi.

Documents are automatically saved in vi, but it is still a good idea to save your work as you go. Simply type Esc:w! to save the current file.

Try this. Open a file, and type some text.

$vi test.txt
i
This is a test
Esc:wq!

Now look at the directory.

$ ls -a
. .. .test.txt.swp test.txt test.txt~

Notice that there is a hidden swap file, your original file, and a backup file. Vi will automatically delete the file on exit, or use the file, if it still exists the next time you open the original file. The file ending with a tilde ~ is the auto-save version of the file, which may or may not have the latest data in case of a crash.


File Systems

File system integrity can be handled with several utilities, each of which could have a discussion of their own.

One of the most commonly used Linux utilities is fsck. The fsck utility will run automatically at boot up if it detects that it has not been run in a while, or if the system was not shut down properly. More advanced users will need to learn this and other utilities which can be run on unmounted file systems, often from a boot disk. Be careful about running fsck on mounted drives, or RAID arrays.

Data Recovery Software and Services

Before resorting to a data recovery service, it may be worth the time to look for some software that can recover the data you need. A data recovery service can be helpful when there is no other way to retrieve the data. They are able to open drives that do not spin, and read them with a microscope, but the process can cost thousands of dollars. If the data can be recovered by using software, the recovery service may be able to restore the data for a few hundred dollars. Wait a minute. If they are using software, can't I do the same thing? The short answer is - maybe. Some software is free, and some costs several hundred to several thousand dollars. As our readers know, we love free open source software. The first thing to try is some of the recovery software, and boot disks listed on freshmeat.net. Then search for data recovery software on search engines such as google, bing, or yahoo. If the free ones don't work, some of the other might do the trick, but the free software should at least show that a partition exists on the drive, and that there might be something to recover. Remember, if the hardware is noisy, or unreliable, it is best to take it to a data recovery service, and not keep it powered on.