Create a Symlink as a Workaround for Partition Full Error

Sometimes, a mounted partition does not have enough space, and it may take time to expand the LVM volume, if there is not currently enough physical space available, and a separate team handles storage.  Of course making a separate partition is a more proper fix, and LVM is flexible enough to accommodate the changes, but sometimes space is filling up, and something has to be done right away.  Maybe /var/log, or /opt is filling up, and you did not think ahead to make it a separate partition.

As a quick workaround, it is fairly common to use a symbolic link, or as sysadmins affectionately call it, a symlimk.

Let's say an application wants to install under /opt, and you are out of space in /opt, but have plenty of space in /app.  You can use a symlink to trick the system into thinking that it is using /opt, when in fact it is using /app/opt.

I like to perform a directory move and symlink creation at the same time.  Note that both /opt and /app already exist, but we are moving the entire /opt directory to be under /app, as /app/opt.

-->

mv /opt /app/ && ln -s /app/opt /opt

Note the use of double ampersand "&&", which tells the shell to run the next command, only if the result of the first command was successful.

Results of ls -l, should give something like this:

     lrwxrwxrwx   1 root   root       8 Oct  1 18:01 opt -> /app/opt


Sometimes this will fail, if there is not enough space at the target location, or if files are in use.  You can use fuser -c /opt to see which files are open, and if necessary, close them with fuser -ck /opt.  It may also be necessary to temporarily close any apps that are using the old partition.

Note that there can also be occasional problems with symlinks, for example mounting across different volume types or NAS shares, but a symlink can be a very convenient solution in many cases.

Career Resources

I thought it might be helpful to point out some career resources for those who may be interested, or actively involved in job hunting.  I realize we all  need to update our resume from time to time, and some of you may not have a LinkedIn profile.

Here are some learning resources, and some useful links for career research.  Some of you know me as someone who is always promoting self-learning.

Learning Resources

Paid Learning Sites are often made available by your employer, so check to see if you have a subscription, or if you can expense it (get reimbursed for the expense).  It doesn't hurt to ask.

http://www.safaribooksonline.com -- Huge selection of technical books, videos, seminars, etc.

http://www.pluralsight.com -- High Quality, from overview to advanced topics for programmers

http://www.lynda.com -- High Quality, some topics are not very advanced, but courses are very well done

https://skillport.com -- Books, courses, videos

You can find career related courses by searching for “job search”, or “resume”,  "interview", or “career.”  For example, here are some courses I recommend in 2018:

• Learning LinkedIn  https://www.lynda.com/LinkedIn-tutorials/Learning-LinkedIn/595944-2.html
• "Discover how to get the most out of using LinkedIn. Learn how to construct a stellar profile, find and add connections, share content, and more."

• My LinkedIn profile may give you some ideas for how to format yours:  https://www.linkedin.com/in/scottmcclelland1

• Job Search Strategies by @VS_Career 
• https://www.lynda.com/Business-Skills-tutorials/Using-exercise-files/97580/111198-4.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3ajob+search%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a2

• Resume Makeover by @@jobjenny 
• https://www.lynda.com/Business-Skills-tutorials/Why-resume-writing-so-hard/706909/716831-4.html?srchtrk=index%3a1%0alinktypeid%3a2%0aq%3aresume%0apage%3a1%0as%3arelevance%0asa%3atrue%0aproducttypeid%3a

You may find it useful to look at job postings when updating your resume to see what kind of requirements are currently out there.  Then do some research:

Also see this article for additional learning resources:  
http://systemnotesorg.blogspot.com/2015/11/online-learning-resources.html

Career Resources

Connect and Organize

http://linkedin.com – connect, research, apply

http://www.jibberjobber.com - Organize and Manage your Career, job search, interviews, etc. - Free site at the time I started using it.

Job Search

http://linkup.com – job postings directly from companies, without 3^rd party recruiters

http://indeed.com -- Popular job search site

https://www.usajobs.gov – US federal government jobs

https://www.zoominfo.com -- for startups

Company and Pay Info, also good for company research

http://payscale.com

http://salary.com

http://glassdoor.com – Jobs, Company Reviews, Salaries

Paid Service

https://www.righteverywhere.com – Excellent career management resources, including company research

Some of these notes are also available in the public forum for job seekers.  This is something I quickly threw together in 2015 to help my coworkers who were impacted by a massive layoff:

http://www.lifeisnojoke.com/upward

Use ssh instead of ping to check server status

Server Availability cannot always be checked by using ping, since often a secured subnet will not allow traffic back.  That common practice of dropping packets is an attempt to hide servers and provide additional security, but it can make troubleshooting more difficult.

Since ssh is typcially running and responding on any linux servers that need to be managed, check_ssh seems to be more practical than the default check_ping as the check_command for each host definition in Nagios.

This simple shell script uses a Nagios plugin /usr/local/nagios/libexec/check_ssh to quickly check the status of a server.  One thing to note is that a Nagios installation is not required, just the plugins have to be compiled so the binary can be available to the wrapper script.

#!/bin/sh

# checkssh.sh
#
# Author: Scott McClelland
# 2017-03-29
#
# checkssh, uses Nagios plugin check_ssh
# /usr/local/nagios/libexec/check_ssh
#
# nagios-plugins are downloadable from:
# https://www.nagios.org/downloads/nagios-plugins/

if [ "$1" = "" ]
then
echo enter host name
 exit 1
fi
host=$1

checkssh()
{
chkssh=/usr/local/nagios/libexec/check_ssh

$chkssh $host  2>&1 >/dev/null
RESULT=$?

if [ "$RESULT" -eq "0" ]

then
        echo $host : OK
else
        echo $host : DOWN
fi

        return $RESULT
}

Run the command with:

   checkssh $host

To run the command on a group of servers, use a for loop.

$ for s in web0{1..3} db0{1,2} xyz01; do ./chkssh.sh $s;done
web01 : OK
web02 : OK
web03 : DOWN
db01 : OK
db02 : OK
xyz01 : DOWN

$

Related article:  http://systemnotesorg.blogspot.com/2007/12/ping-multiple-hosts-using-bash-nmap-and.html

Useful nmap Tips

OS Fingerprint Scanning

Sometimes there is a question about what OS is running on a server. There are advanced options from –help, but usually running nmap with -A is enough to tell you exactly what OS is running on the server in question.

Simple nmap scan, which includes OS detection:

nmap -v -A $s

Find an Available IP

Note that your company may have another system of record, so check there for the final authoritative answer.

A quick way to find an IP that is not in DNS, and not responding to ping, or ssh, is to run an nmap scan of the subnet.

Use nmap to find available IPs

 net=10.1.1.0;p=24;nmap -v -R -sn ${net}/${p} -o /tmp/subnet-${net}-${p}.txt

Note: This may scan a large number of IPs, so it may be more convenient to write to a file to analyze later, rather tthan running the scan multiple times.

An available IP would be one that shows an IP address with no DNS name, and also “host down.”