FUSE & SSHFS - Overview and example in STAR online environment

FUSE (Filesystem in Userspace)


FUSE is a kernel module that acts as a bridge between the kernel’s built-in filesystem functions and user-space code that “understands” the (arbitrary) structure of the mounted content.  It allows non-root users to add filesystems to a running system.

Typically, FUSE-mounted filesystems are (nearly) indistinguishable from any other mounted filesystem to the user.

Some examples of FUSE in action:

  • WikipediaFS - viewing and editing Wikipedia articles as if they are local files.
  • Archive access - accessing and in some cases manipulating files in tarballs, zip archives, cpio archives, etc.
  • Encrypted filesystems
  • Union of filesystems (as is done in many live Linux boot disks and Linux installation routines to merge the read-only CD-rom filesystem with read-write space on disk)
  • Event Triggering - FUSE implementations can have triggered events.  Some possible uses might be:
    • automatically restarting a service if its configuration file is altered
    • automatically re-compiling code whenever a source file is changed
    • making a back-up after a file is changed
  • Arbitrary hardware interface
  • ... and the one we will focus on here:  SSHFS

The Fuse project FileSystems page has a more complete list and links to individual software projects that use FUSE.

 
SSHFS (Secure Shell Filesystem)


SSHFS allows a user (not necessarily root) on host A (the "client") to mount a directory on host B (the "server") using the (almost) ubiquitous SSH client-server communication protocols.  Generally, no configuration changes or software installations are required on host B.

The directory on host B then looks like a local directory on host A, at a location in host A's directory structure chosen by the user (in a location where user A has adequate privileges of course).

Unlike NFS, the user on host A must authenticate as a known user on host B, and the operations performed on the mounted filesystem are performed as known user on host B.  This avoids the "classic" NFS problem of UID/GID clashes between the client and server.

Here is a sample session with some explanatory comments:

In this example, host A is "stargw1" and host B is "staruser01".  The user name is wbetts on both hosts, but the user on host B could be any account that the user can access via SSH.
 
First, create a directory that will serve as the mountpoint:

[wbetts@stargw1 ~]$ mkdir /tmp/wbssh
[wbetts@stargw1 ~]$ ls -ld /tmp/wbssh
drwxrwxr-x  2 wbetts wbetts 4096 Oct 13 10:52 /tmp/wbssh

Second, mount the remote directory using the sshfs command:

[wbetts@stargw1 ~]$ sshfs staruser01.star.bnl.gov: /tmp/wbssh


In this example, no remote username or directory is specified, so the remote username is assumed to match the local username and the user’s home directory is selected by default.  So the command above is equivalent to:

% sshfs wbetts@staruser01.star.bnl.gov:/home/wbetts /tmp/wbssh

That’s it!  (No password or passphrase is required in this case, because wbetts uses SSH key agent forwarding) 

Now use the remote files just like local files:

[wbetts@stargw1 ~]$ ls -l /tmp/wbssh |head -n 3
total 16000
-rw-rw-r--  1 1003 1003    6412 Oct 19  2005 2005_Performance_Self_Appraisal.sxw
-rw-rw-r--  1 1003 1003   10880 Oct 19  2005 60_subnet_PLUS_SUBSYS.sxc
[wbetts@stargw1 ~]$ ls -ld /tmp/wbssh
drwx------  1 1003 1003 4096 Oct 11 15:56 /tmp/wbssh


The permissions on our mount point have been altered -- now the remote UID is shown (a source of possible confusion) and the permissions have morphed to the permissions on the remote side, but this is potentially misleading too…

[root@stargw1 ~]# ls /tmp/wbssh
ls: /tmp/wbssh: Permission denied

Even root on the local host can’t access this mount point, though root can see it in the list of mounts.
 
In addition to the ACL confusion, there can be some quirks in behaviour, where sshfs doesn't translate perfectly:

[wbetts@stargw1 ~]$ df /tmp/wbssh
Filesystem                                       1K-blocks       Used     Available        Use%     Mounted on
sshfs#staruser01.star.bnl.gov:    1048576000         0     1048576000       0%     /tmp/wbssh


Ideally the user unmounts it once finished, else it sits there indefinitely (it is probably subject to the same timeouts (TCP, firewall conduit, SSH config, etc.) as an ordinary ssh connection, but in limited testing so far, the connection has been long term)  Here is the unmount command:

[wbetts@stargw1 ~]$ fusermount -u /tmp/wbssh/
[wbetts@stargw1 ~]$ ls /tmp/wbssh
[wbetts@stargw1 ~]$

Some additional details:

By default, users other than the user who initiated the mount are not permitted access to the local mountpoint (not even root), but that can be changed by the user, IF it is permitted by the FUSE configuration (as decided by the admin of the client node).  The options though are not very granular.  The three possible options are:

  1. access for the user who mounted it (and no one else)
  2. the mounter plus root
  3. everybody

In any case, whoever accesses the mount point will act as (and have the permissions of) the user on host B specified by the mounter.  This requires careful evaluation of the options permitted and user education on the possibilities of allowing inappropriate or unnecessary access to other users.

The mount is not tied to the specific shell it is started in.  It lasts indefinitely it seems – the user can log out of host A, kill remote agents, etc. and the mount remains accessible on future logins.  (Interpretation: an agent of some sort is maintained on the client (host A) on the user’s behalf.  (If multiple users have access to the user account on A, this could be worrisome, in the same manner as the allowance of others to access the mount point mentioned above.)) 

 

Here are some potential advantages and benefits of using SSHFS, some of which are mentioned above:

  • User-initiated
  • Encrypted communications over the network
  • Authenticated (at first order) – somewhat better user tracing than NFS
  • SSH keys/forwarding can make it relatively painless (no pass{words,phrases} required for mounting)
  • Networking/firewalling is simple – if ssh works between the two nodes, then so will sshfs (unlike NFS, where port configuration and firewalls are a pain)
  • “Passthrough” mounting works -- an sshfs mount point can be mounted from another node (if host B mounts a directory on C, then A can mount B's mountpoint and have access to C's filesystem.  In this case, B acts as both a client (to C) and a server (to A).)
  • No server-side configuration is needed.
  • These mounts can be automounted by the user somewhat like autofs using afuser ( http://afuse.sourceforge.net/ ), though this is primarily for interactive use based on SSH agents.

 

And some drawbacks:

  • User initiated (they are unlikely to clean up after themselves)
  • Access controls are either very strict (by default), or very lax in the hands of users (-o allow_other or -o allow_root) -- nothing else
  • Cross-system UID overlap and ACLs can be confusing
  • Availability of FUSE for RHEL/SL 3 and other clients?
  • Use of SSHFS in scripts could entice users to create SSH keys without passphrases -- a real no-no!

And some final details about the configuration of the online gatekeepers that presumably are prime candidates for the use of SSHFS:

The standard installation of FUSE for Scientific Linux 4 seems to not be quite complete.  A little help is required to make it work:

In /etc/rc.d/rc.local:

/etc/init.d/fuse start
/bin/chown root.fuse /dev/fuse
/bin/chmod 660 /dev/fuse


“fuse” group created – each user who will use SSHFS needs to be a member of this group (must be kept in mind if we use NIS or LDAP for user management on the gateways)

 

Server Logging

The default openssh packages  from Scientific Linux 3, 4 and 5 (~openssh 3.6, 3.9 and 4.3 respectively) do not support sftp-subsystem logging.  Later versions of openssh do (starting at version ~4.4).  This provides the ability to log file accesses and trace them to individual (authenticated) users. 

I grabbed the latest openssh source (version 5.1) and built it on an SL4 machine with no trouble:

% ./configure --prefix=/opt/openssh5.1p1 --without-zlib-version-check --with-tcp-wrappers
% make
% make install

 

Then in the sshd_config file, append "-f AUTHPRIV -l INFO" to sftp-subsystem line.  This activates the logging level (INFO) and causes the logs to be sent to /var/log/secure.  (To be tried: VERBOSE log level).

Even at the INFO level, the logs are fairly detailed.  Shown below is a sample session, with the client commands on the left and the resulting log entries from the server (carradine, using port 2222 for testing) on the right.  For brevity, the time stamps from the log have been removed after the first entry.

 

SFTP LOGGING at the INFO level
CLIENT COMMANDS SERVER LOG (/var/log/secure)
   
sshfs -p 2222 wbetts@carradine.star.bnl.gov:/home/wbetts/ carradine_home Nov 20 14:30:29 carradine sshd[29120]: Accepted publickey for wbetts from 130.199.60.84 port 41746 ssh2
carradine sshd[29122]: subsystem request for sftp
carradine sftp-server[29123]: session opened for local user wbetts from [130.199.60.84]
ls carradine_home carradine sftp-server[29123]: opendir "/home/wbetts/."
carradine sftp-server[29123]: closedir "/home/wbetts/."
touch carradine_home/test.txt carradine sftp-server[29123]: sent status No such file
carradine sftp-server[29123]: open "/home/wbetts/test.txt" flags WRITE,CREATE,EXCL mode 0100664
carradine sftp-server[29123]: close "/home/wbetts/test.txt" bytes read 0 written 0
carradine sftp-server[29123]: open "/home/wbetts/test.txt" flags WRITE mode 00
carradine sftp-server[29123]: close "/home/wbetts/test.txt" bytes read 0 written 0
carradine sftp-server[29123]: set "/home/wbetts/test.txt" modtime 20081120-14:36:36
cat /etc/DOE_banner >> carradine_home/test.txt carradine sftp-server[29123]: open "/home/wbetts/test.txt" flags WRITE mode 00
carradine sftp-server[29123]: close "/home/wbetts/test.txt" bytes read 0 written 1119
rm carradine_home/test.txt carradine sftp-server[29123]: remove name "/home/wbetts/test.txt"
fusermount -u carradine_home/ carradine sftp-server[29123]: session closed for local user wbetts from [130.199.60.84]

 

From these logs, we would appear to have a good record of the who/what/when of sshfs usage.  But the need to build our own openssh packages puts a burden on us to track and install updated openssh versions in a timely fashion, rather than relying on the distribution maintainer and the OS's native update manager(s).  The log files on a heavily utilised server may also become unwieldy and cause a performance degredation, but I've not made any estimates or tests of these issues.

 


Here are the specific relevant packages installed on the client test nodes (stargw1 and stargw2):


fuse-2.7.3-1.SL
fuse-libs-2.7.3-1.SL
fuse-devel-2.7.3-1.SL
fuse-sshfs-2.1-1.SL
kernel-module-fuse-2.6.9-78.0.1.ELsmp-2.7.3-1.SL

(Exact versions should not be terribly important, but it appears that fuse-2.5.3 included up to SL4.6 requires more tweaking after installation than fuse 2.7.3 included in SL4.7).