Replacing the Red Hat-supplied OpenSSH package with a newer version built from sourceEdit

Background

For many years now I have used the excellent rssh shell to lockdown access to third-parties who require upload (SFTP) access to the server.

For basic security, rssh enables users to connect to the server and upload using SFTP without needing full-fledged shell access.

For additional security, rssh can operate within a chroot "jail" which restricts users to a designated area (usually their home directory) and prevents them from seeing other parts of the filesystem.

In one of the recent updates supplied by Red Hat I noticed that I could no longer create working jails. Old jails continued to work because all the necessary files for the jail had been copied inside them at the time of their creation, but new jails didn’t work.

The script that I use to create these jails (shown below) was specifically designed to handle the process automatically (to eliminate human error) and to be easily re-runnable on an existing jail to "refresh" it; nevertheless it didn’t work.

I spent a little while investigating the problem and couldn’t see any obvious errors in the set-up of the new jails. Running the script using sh -x to see exactly what it was doing revealed no obvious problems, and manual inspection of the required libraries with ldd didn’t reveal any obvious omissions.

This wasn’t a very satisfactory situation because it shows how brittle the system is: at any given time it is possible that Red Hat might make a change that breaks the jail-creation script, even though it was carefully crafted to be flexible and re-runnable. And the fact that processes running inside the jail can’t write to system log files, at least without special configuration of the syslog daemon, makes the problem difficult to diagnose. This was not a process which I wanted to face repeatedly in the future.

#!/bin/sh -e
# vhost-jail
# Copyright 2006-2008 Wincent Colaiuta
# Version 1.0 (20 January 2006): RHEL 3
# Version 2.0 (27 February 2008): RHEL 5.1

#
# Constants
#

# tools/executables:
RSSHHELPER=/usr/local/libexec/rssh_chroot_helper
SFTPSERVER=/usr/libexec/openssh/sftp-server

# support files:
LDCACHE=/etc/ld.so.cache
LDCONF=/etc/ld.so.conf
PASSWD=/etc/passwd
RSSHCONF=/usr/local/etc/rssh.conf

#
# Functions
#

die()
{
  SCRIPT=$(basename $0)
  echo "$SCRIPT: error: $1" >&2
  exit 1
}

# Returns the true name of a regular file or symlink (not a directory).
# For example, given a symbolic link like this:
#  /lib/libresolv.so.2 -> libresolv-2.5.so
# Then:
#   truename /lib/libresolv.so.2
# Returns:
#   /lib/libresolv-2.5.so
truename()
{
  FILE=$1
  DIR=$(dirname $FILE)
  NAME=$(basename $FILE)
  if [ ! -f $FILE ]; then
    die "truename(): $FILE is not a regular file"
  fi
  if [ -L $FILE ]; then
    TARGET=$(find $DIR -maxdepth 1 -name $NAME -printf "%l\n")
    TRUENAME="$DIR/$TARGET"
  else
    TRUENAME=$FILE
  fi
  echo $TRUENAME
}

lib_list()
{
  # Will process output from ldd that looks like this:
  #   linux-gate.so.1 =>  (0x00a5c000)
  #   libc.so.6 => /lib/libc.so.6 (0x00318000)
  #   /lib/ld-linux.so.2 (0x002fb000)
  # grep filters out lines which don't contain an absolute path
  # perl then strips off everything except the (space-delimited) path
  ldd $1 | grep / | perl -pe "s/.*?(\/[^ ]+).*/\1/"
}

createdir()
{
  if [ $# -ne 1 ]; then
    die "createdir() requires exactly one argument"
  fi
  if [ ! -d "$1" ]; then
   if [ -e "$1" ]; then
     die "createdir(): $1 already exists and is not a directory"
   else
    mkdir -p -v "$1" || die "createdir() failed for path: $1"
   fi
  fi
}

checkbase()
{
  if [ $# -ne 1 ]; then
    die "checkbase() requires exactly one argument"
  fi
  BASE="$1"
  COUNT=`echo "${BASE}" | egrep -c "\/$"`
  if [ $COUNT -ne 0 ]; then
    die "checkbase() requires base directory to have no trailing slash"
  fi
}

# Previously this method set up hard links and symbolic links.
# On this RHEL 5.1 install most hard links won't work (because they'd be cross-device links);
# so we copy instead of hard-linking (uses about 5MB per jail).
link()
{
  if [ $# -ne 2 ]; then
    die "link() requires exactly two arguments"
  fi
  FILE="$1"
  BASE="$2"
  DIR=$(dirname "$FILE")
  checkbase "${BASE}"
  createdir "${BASE}${DIR}"
  if [ ! -f "${FILE}" ]; then
    die "link(): ${FILE} is not a regular file"
  fi
  if [ -L "$1" ]; then
    NAME=$(basename "${FILE}")
    TARGET=$(find "$DIR" -maxdepth 1 -name "${NAME}" -printf "%l\n")
    TRUENAME="${DIR}/${TARGET}"
    echo "${FILE} is a symbolic link with target ${TRUENAME}"
    cp -v -f "$TRUENAME" "$BASE$TRUENAME" || die "cp failed for file: $TRUENAME"
    pushd "${BASE}${DIR}"
    ln -v -f -s "${TARGET}" "${NAME}" || die "link() (symbolic) failed for file: ${BASE}${FILE}"
    popd
  else
    cp -v -f "$FILE" "$BASE/$FILE" || die "cp failed for file: $FILE"
  fi
}

#
# Main
#

if [ $(whoami) != "root" ]; then
  die "this tool must be run as root"
fi

if [ $# -ne 2 ]; then
  die "exactly two arguments required ('path to chroot' and 'username')"
fi

BASE="$1"
USER="$2"
checkbase "${BASE}"

# create relevant /etc/passwd entry
DIR=$(dirname "$PASSWD")
createdir "${BASE}${DIR}"
cat "$PASSWD" | egrep "^${USER}:" > "$BASE$PASSWD"
COUNT=$(wc -l "$BASE$PASSWD" | awk '{print $1}')
if [ $COUNT -ne 1 ]; then
  die "error creating $BASE$PASSWD from $PASSWD"
fi

# Get an up-to-date list of libraries linked to by sftp-server:
LIBS=$(lib_list $SFTPSERVER)
for LIB in ${LIBS}; do
  link "${LIB}" "${BASE}"
done

LIBS=$(lib_list $RSSHHELPER)
for LIB in ${LIBS}; do
  link "${LIB}" "${BASE}"
done

# copy/link support files
link "${LDCACHE}" "${BASE}"
link "${LDCONF}" "${BASE}"

# copy/link tools
link "${RSSHHELPER}" "${BASE}"
link "${SFTPSERVER}" "${BASE}"

# set up /dev/null if required
test -d "$BASE/dev" || mkdir "$BASE/dev"
test -e "$BASE/dev/null" || cp -a /dev/null "$BASE/dev/null"

# append to rssh config
CONFIG="user=${USER}:022:00010:\"${BASE}\""
COUNT=$(cat "${RSSHCONF}" | egrep -c "${CONFIG}")
if [ $COUNT -eq 0 ]; then
  echo "Appending line to ${RSSHCONF}"
  echo "${CONFIG}"
  echo "${CONFIG}" >> "${RSSHCONF}"
fi

# tighten ownership on chroot jail dir
chown -v root:root "${BASE}"

exit 0

The decision to update

At this time I became aware of a new option added to OpenSSH around version 4.9, the ChrootDirectory setting which largely renders rssh unnecessary. Red Hat actually backported this feature to the older version that they currently ship with RHEL 5.4 (they are currently shipping OpenSSH version 4.3; the latest version available is 5.3), but it is basically useless without the Match directive which they didn’t backport and which is necessary in order to selectively apply the jail only to certain users.

As such, jailing SFTP users isn’t really practical under the Red Hat-supplied OpenSSH because doing so would effectively jail all users, even system administrators, rendering remote administration impossible. (The possible workaround of setting the home directory of administrators to the root of the filesystem, /, seems like a hideous kludge and I immediately dismissed it.)

I like sticking to the stock-standard Red Hat packages wherever it is practical to do so because of the automated, quality-controlled security fixes, but in this case I felt like the advantages of switching to a hand-compiled, newer version of OpenSSH (robust, easy-to-configure chroot jails for SFTP users) outweighed the costs (having to monitor another mailing list for security advisories).

Another option I considered but discarded was the possibility of running a custom OpenSSH build out of /usr/local and listening on a different port, but given that these jails exist in part because they are for giving access to pseudo-trusted third parties, it might be confusing for them to set up their file transfer clients adequately (as it is, standard SFTP access proves to be tricky for many users, many of which are accustomed only to vanilla FTP clients).

The question was, how to replace the yum-installed Red Hat version of OpenSSH with a hand-built one? This was all to be done remotely so I wanted to be very careful about the steps I took so as not to get locked out.

Making the transition

Grab the latest version from http://www.openssh.org/portable.html.

$ wget ftp://ftp.OpenBSD.org/pub/OpenBSD/OpenSSH/portable/openssh-5.3p1.tar.gz
$ tar xzvf openssh-5.3p1.tar.gz
$ cd openssh-5.3p1

The page linked to above provides instructions on how to verify the signatures on the downloaded archive.

Now do a test build to see if we can do a warning and error-free build:

$ ./configure
$ make
$ make tests

Inspect what packages are currently installed:

$ rpm -qa|grep ssh
openssh-4.3p2-36.el5_4.2
openssh-clients-4.3p2-36.el5_4.2
openssh-server-4.3p2-36.el5_4.2

See what files on disk are associated with those packages:

$ rpm -ql openssh
$ rpm -ql openssh-clients
$ rpm -ql openssh-server

Do a test install under /usr/local:

$ sudo make install

Edit the test install’s configuration file to listen on a different port:

# vi /usr/local/etc/sshd_config

Specifically, I wanted the test install to listen on port 2222:

Port 2222

Instead of the default port 22:

Port 22

On my system I have a restrictive firewall which rejects all traffic except for some specifically whitelisted things, so I had to allow access to port 2222. First step is to find out the rule number corresponding to the rejection rule:

# iptables -L --line-numbers|grep REJECT
33   REJECT     all  --  anywhere             anywhere            reject-with icmp-host-prohibited

So I need to insert a new rule at rule 33 which whitelists traffic to port 2222:

# iptables -I INPUT 33 -m state --state NEW -m tcp -p tcp --dport 2222 -j ACCEPT

Now we fire up the test server:

# /usr/local/sbin/sshd

After testing connectivity to the test server (with ssh -p 2222 user@example.com and sftp -oPort=2222 user@example.com; it worked), time to do a real install over the top of the existing system install. Any existing sessions will survive here which is a nice safety measure.

$ make clean
$ ./configure --prefix=/usr --sysconfdir=/etc/ssh
$ make
$ sudo make install

Kill the old master process and start a new one:

# service sshd restart

After confirming that it works, we are free to set up our jails. Set up a new group, sftp, which will contain all users who are to have SFTP-only access.

# groupadd sftp

Add a test user to the group to see if it works:

# usermod -G sftp exampleuser

My test user already had its home directory owned by root:root, with no write permissions of others, so I proceeded to edit the end of my /etc/ssh/sshd_config:

# override default of no subsystems
[/tags/Subsystem #Subsystem]	sftp	/usr/libexec/openssh/sftp-server
Subsystem	sftp	internal-sftp

Match Group sftp
	ChrootDirectory	%h
	ForceCommand	internal-sftp
	X11Forwarding	no
	AllowTcpForwarding	no

Test, it works as intended.

Change the user’s shell:

chsh -s /sbin/nologin exampleuser

Test again; still works, so proceed to add all other SFTP users to the sftp group with usermod.

I also have a couple of users on the server for Rails apps, and these had a slightly different set-up so I needed to tweak the permissions slightly. These home directories weren’t owned by root:root but by the Rails user itself so I had to tweak like this:

# chown root rails_app
# chmod 750 rails_app

As a finally step had to clean up the RPM database (as yum/RPM still thinks other package is installed):

# rpm -e --test --justdb openssh-clients openssh-server openssh
# rpm -e --justdb openssh-clients openssh-server openssh

Confirm that yum and rpm are happy:

# rpm -qa | grep ssh
# yum list|grep ssh
openssh.i386                            4.3p2-36.el5_4.2      rhel-i386-server-5
openssh-askpass.i386                    4.3p2-36.el5_4.2      rhel-i386-server-5
openssh-clients.i386                    4.3p2-36.el5_4.2      rhel-i386-server-5
openssh-server.i386                     4.3p2-36.el5_4.2      rhel-i386-server-5

And get rid of the temporary firewall rule:

# iptables -L --line-numbers

Just check the rule number we want to target:

33   ACCEPT     tcp  --  anywhere             anywhere            state NEW tcp dpt:rockwell-csp2

Go ahead with the deletion:

# iptables -D INPUT 33

And confirm that it worked:

# iptables -L

We can kill the test process (find it with ps auxwww|grep ssh and look for /usr/local/sbin/sshd, and kill using kill) and make sure that the PID file at /var/run/sshd.pid corresponds to the new master process that we launched earlier so that service sshd restart (and similar) will work.

Finally, subscribe to https://lists.mindrot.org/mailman/listinfo/openssh-unix-announce to keep on top of security and release announcements.

Follow-ups

git push

I later discovered that my attempts to git push to the server were broken by the update. The message in the logs was:

User git_user not allowed because account is locked

This is because the Git user had !! as a password in /etc/shadow, indicating that the account was locked. The old version of OpenSSH had allowed public key authentication anyway, but the new version does not.

So the required modification was to change the !! (user cannot log in to the system) to a * (user cannot log in by password) by manually editing /etc/shadow.

This has to be done manually because the usermod utility doesn’t provide a means of swapping from !! to *, only of prepending a ! (locking with -L or --lock) or removing a prepended ! (unlocking with -U or --unlock).

# chmod +w /etc/shadow
# vim /etc/shadow
# chmod -w /etc/shadow

SSH PATH

Another discovery: the old SSH had a broader PATH setting, so commands like this:

$ ssh user@example.com "sh -c 'which gem git'"

Would work and find /usr/bin/gem and /usr/local/bin/git.

On the new install only the former is found but the later is not because /usr/local/bin is not in the PATH.

The solution is to set up a ~/.profile file containing:

PATH=/usr/local/bin:$PATH
export PATH

And cause Bash to consult it by invoking sh with the -l switch:

$ ssh user@example.com "sh -l -c 'which gem git'"

Similarly, commands like:

$ ssh user@example.com sudo monit restart all

Used to work because monit is at /usr/local/bin/monit and /usr/local/bin was in the PATH, but now need to be wrapped inside sh -l -c ... in order to flesh out the PATH appropriately:

$ ssh user@example.com "sh -l -c 'sudo monit restart all'"