HOWTO: setup avfs / fuse on debian (etch)

Linux nowadays comes with integrated support for FUSE which allows all kinds of interesting user level file systems. One of the most interesting might well be avfs which allows you to treat any archive (like zip, tar, etc) as if it was a directory.

This is particularly handy, if you want to browse archives through apache. In our test automation environment we generate and receive many megabytes of log files every day. Log files from several clients are zipped together and send to our "visulization center". We extract some common information out of the log files which is displayed in a web-app and we keep the log files for reference (in their compressed format). On the test results page we have links that link right into the log file archives. Obviously this "zip browsing" could be implemented in the web-app itself, but why bother apache with avfs has all you need.

The only problem is, that for the current stable release of debian (etch) avfs is not available and can not be compiled against stable versions of libfuse2/libfuse-dev. However there is a simple solution to this, you can get updated fuse packages from backports.org.

Before you follow this guide: Make sure that you really (still) need etch.

This howto does not strive to be a complete help guide! Please read the information available on the backports.org page very carefully.

CHALLENGES:

INSTALLING AVFS

PRE-SETUP:

You may want to start with a system update, in other words

apt-get update
apt-get install kernel-image-2.6-686
apt-get dist-upgrade
update-grub
sync
reboot

You need to add etch-backports to your /etc/apt/sources.lst. The following is a very short description, for details please see backports.org. Add the following lines

deb http://www.backports.org/debian etch-backports main contrib non-free
deb-src http://www.backports.org/debian etch-backports main contrib non-free
You also need to get the backports pgp keys
wget -O - http://backports.org/debian/archive.key | apt-key add -
Once this is done update your package lists
apt-get update

PKG INSTALL:

The packages we need for this to work are:

apt-get -t etch-backports install libfuse-dev
apt-get -t etch-backports install libfuse2
apt-get -t etch-backports install fuse-utils

GET AND COMPILE AVFS:

We will get the lenny (testing) packages of avfs which compile fine against the fuse version from backports. You may want to check with debian if there are later versions available by now and update the following commands accordingly.

curl -O http://ftp.de.debian.org/debian/pool/main/a/avfs/avfs_0.9.8.orig.tar.gz
curl -O http://ftp.de.debian.org/debian/pool/main/a/avfs/avfs_0.9.8-6.diff.gz
tar -zxf avfs_0.9.8.orig.tar.gz
gunzip avfs_0.9.8-6.diff.gz
mv avfs-0.9.8.orig avfs-0.9.8-6 && cd avfs-0.9.8-6 && patch < ../avfs_0.9.8-6.diff

Let's compile and install.

./configure --enable-fuse
make clean all
make install
If the configure shows you that it doesn't like your version of fuse (which it shouldnt with the backports packages), you can try the option --disable-fuse-test.

SETUP AND START AVFS:

avfs works on a per user basis. If you just want to use it as a regular user, you need to create a .avfs subdirectory in your homedir and start the daemon (each user on its own). You also need to be able to access /dev/fuse which is possible if you are in group fuse.

As user root you will have to edit /etc/group or use

usermod -G fuse -a <user>
to add the users that should be able to use avfs.

Each user that wants to use acfs has to execute:

mkdir ~/.avfs
mountavfs

TEST AVFS:

Well that's it for avfs. You are still reading, so it seems, things are going well so far. Let's give a quick try.

(cd /tmp && echo "Hide me up" > bla && tar -zcf bla.tar.gz bla && rm bla)
cd ~/.avfs/tmp/bla.tar.gz#/
ls
cat bla

AVFS AND APACHE

Combining avfs and apache can be very useful if you want or need to store large amounts of data in a compressed format and still want to be able to browse it.

AVFS STARTUP:

We want to run avfsd for the "apache user" and we want it to mount to a "global" mountpoint.

sudo -u www-data avfsd -o modules=subdir -o subdir=/projects/fasthosting/ -o rellinks /var/www/archiveroot/
will mount the directory /projects/fasthosting with avfs at /var/www/archiveroot accessible to the user www-data. It will also transform any absolute links to relative links so that you stay within /var/www/archiveroot.

You may want to try option -f -d to get an idea what is going on in case things don't work.

APACHE CONFIGURATION:

I will just provide an example here which works in our particular case - all of our archive files are .zip files, but you should get the idea. The trick is to change the URL such that you add a # after the zip filename (which is a bit tricky because # has a special meaning in urls...)

Alias "/browse" "/var/www/archiveroot"
<Directory /var/www/archiveroot>
    Allow from All
    Options Indexes SymLinksIfOwnerMatch
    AllowOverride None
    RewriteEngine On
    RewriteBase /browse
    RewriteCond %{REQUEST_FILENAME} -f
    RewriteRule ^(.*?.zip)$ $1/BROWSE/ [R,L]
    RewriteRule ^(.*)/BROWSE/(.*)$ $1\%23/$2 [NS,L,NE]
</Directory>

This should work out of the box, give it a try - don't forget to update the RewriteBase if you change the Alias directive.

AVFS AUTOSTART:

If you want to use this in a productive environment you will need to start avfsd at startup. Here is my init.d script based on the debian skeleton, yours may need some adaptation.

#! /bin/bash
### BEGIN INIT INFO
# Provides:          fast_avfs
# Required-Start:    $local_fs $remote_fs
# Required-Stop:     $local_fs $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Start avfsd for user www-data
# Description:       Starter for the fuse plugin avfs which allows browsing of zip files.
#                    Used by the fasthosting apache configurations to view inside uploaded zip file.
### END INIT INFO

# Author: Joachim Buechse 

# Do NOT "set -e"

USER=www-data
GROUP=www-data
MOUNT_POINT=/var/www/archiveroot/
BASE_DIR=/projects/fasthosting/

# PATH should only include /usr/* if it runs after the mountnfs.sh script
#PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="archive browser for fasthosting (user www-data)"
NAME=fast_avfs
DAEMON=/usr/bin/avfsd
DAEMON_ARGS="-o modules=subdir -o subdir=$BASE_DIR -o rellinks $MOUNT_POINT"
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME



# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Read configuration variable file if it is present
#[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
    # Return
    #   0 if daemon has been started
    #   1 if daemon was already running
    #   2 if daemon could not be started
    start-stop-daemon --start --quiet --pidfile $PIDFILE --exec $DAEMON --test > /dev/null \
        || return 1
    start-stop-daemon --start --quiet --make-pidfile --pidfile $PIDFILE --background --chuid $USER:$GROUP --exec $DAEMON -- \
        -f $DAEMON_ARGS \
        || return 2
    # we start avfsd in foreground mode a create PIDFILE with the correct PID
}

#
# Function that stops the daemon/service
#
do_stop()
{
    # Return
    #   0 if daemon has been stopped
    #   1 if daemon was already stopped
    #   2 if daemon could not be stopped
    #   other if a failure occurred
    
    # check if the mountpoint exists, if not (directory) we are done
    [ -d $MOUNT_POINT ] && return 1

    #echo "unmounting with fusermount"
    /usr/bin/fusermount -u $MOUNT_POINT
    RETVAL="$?"
    [ "$RETVAL" = 0 ] && [ -d $MOUNT_POINT ] && rm -f $PIDFILE && return 0
    
    #echo "killing process"
    start-stop-daemon --stop --quiet --retry=TERM/30 --pidfile $PIDFILE --name $NAME
    RETVAL="$?"
    [ "$RETVAL" = 2 ] && return 2

    rm -f $PIDFILE
    return "$RETVAL"
}

case "$1" in
  start)
    [ "$VERBOSE" != no ] && log_daemon_msg "Starting $DESC" "$NAME"
    do_start
    case "$?" in
        0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
        2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  stop)
    [ "$VERBOSE" != no ] && log_daemon_msg "Stopping $DESC" "$NAME"
    do_stop
    case "$?" in
        0|1) [ "$VERBOSE" != no ] && log_end_msg 0 ;;
        2) [ "$VERBOSE" != no ] && log_end_msg 1 ;;
    esac
    ;;
  restart|force-reload)
    #
    # If the "reload" option is implemented then remove the
    # 'force-reload' alias
    #
    log_daemon_msg "Restarting $DESC" "$NAME"
    do_stop
    case "$?" in
      0|1)
        do_start
        case "$?" in
            0) log_end_msg 0 ;;
            1) log_end_msg 1 ;; # Old process is still running
            *) log_end_msg 1 ;; # Failed to start
        esac
        ;;
      *)
          # Failed to stop
        log_end_msg 1
        ;;
    esac
    ;;
  *)
    echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
    exit 3
    ;;
esac

If this works for you, don't forget to activate it. I'm starting and stopping it at the same time as apache which is S91 and K09 on our installation.

update-rc.d fast_avfs defaults 91 09

LINKS:

(c) joachim(et)buechse.ch. This description is in the public domain.