monitor files (à la tail -f) in an entire directory (even new ones)

I normally watch many logs in a directory doing tail -f directory/*.
The problem is that a new log is created after that, it will not show in the screen (because * was expanded already).

Is there a way to monitor every file in a directory, even those that are created after the process has started?

Answers:

Thank you for visiting the Q&A section on Magenaut. Please note that all the answers may not help you solve the issue immediately. So please treat them as advisements. If you found the post helpful (or not), leave a comment & I’ll get back to you as soon as possible.

Method 1

You can tail multiple files with… multitail.

multitail -Q 1 'directory/*'

-Q 1 PATTERN means to check for new content in existing or new files matching PATTERN every 1 second. Lines from all files are shown in the same window, use -q instead of -Q to have separate windows.

Method 2

xtail is also an alternative. Its man page describes it as:

Xtail monitors one or more files, and displays all data written to a
file since command invocation. It is very useful for monitoring
multiple logfiles simultaneously. If an entry given on the command
line is a directory, all files in that directory will be monitored,
including those created after the xtail invocation. If an entry given
on the command line doesn’t exist, xtail will watch for it and monitor
it once created. When switching files in the display, a banner showing
the pathname of the file is printed.

An interrupt character (usually CTRL/C or DEL) will display a list of
the most recently modified files being watched. Send a quit signal
(usually CTRL/backslash) to stop xtail.

Method 3

No idea about a shell solution, but (assuming Linux¹) inotify could be the way to go… see this example imitating tail -F (using pyinotify), maybe it can be used as a basis for following an entire directory.

In general, inotify can monitor directories (citing man 7 inotify)

The following bits can be specified in mask
when calling inotify_add_watch(2) and may be returned in the mask field
returned by read(2):

IN_ACCESS         File was accessed (read) (*).
IN_ATTRIB         Metadata changed, e.g., permissions, timestamps,
                    extended attributes, link count (since Linux 2.6.25),
                    UID, GID, etc. (*).
IN_CLOSE_WRITE    File opened for writing was closed (*).
IN_CLOSE_NOWRITE  File not opened for writing was closed (*).
IN_CREATE         File/directory created in watched directory (*).
IN_DELETE         File/directory deleted from watched directory (*).
IN_DELETE_SELF    Watched file/directory was itself deleted.
IN_MODIFY         File was modified (*).
IN_MOVE_SELF      Watched file/directory was itself moved.
IN_MOVED_FROM     File moved out of watched directory (*).
IN_MOVED_TO       File moved into watched directory (*).
IN_OPEN           File was opened (*).

When monitoring a directory, the events marked with an asterisk (*) above can
occur for files in the directory, in which case the name field in the returned
inotify_event structure identifies the name of the file within the directory.

(…and pyinotify closely follows theses options)

1: BSDs have a similar thing, kqueue. Maybe a cross-platform solution is achievable using GIO (Python bindings) as abstraction layer since it can, beside inotify, also use kqueue

Method 4

Also you can watch directory with watch

watch -n0,1 "ls -lrt /directory/ | tail"

Method 5

I wrote a quick one that fulfils the need.

#!/bin/bash
LOG_PATTERN=$1
BASE_DIR=$(dirname $LOG_PATTERN* | head -1)

run_thread (){
    echo Running thread
    tail -F $LOG_PATTERN* &
    THREAD_PID=$!
}

# When someone decides to stop the script - killall children
cleanup () {
    pgrep -P $$ | xargs -i kill {}
    exit
}

trap cleanup SIGHUP SIGINT SIGTERM

if [ $# -ne 1 ]; then
    echo "usage: $0 <directory/pattern without * in the end>"
    exit 1
fi

# Wait for the directory to be created
if [ ! -d $BASE_DIR ] ; then
    echo DIR $BASE_DIR does not exist, waiting for it...
    while [ ! -d $BASE_DIR ] ; do
        sleep 2
    done
    echo DIR $BASE_DIR is now online
fi

# count current number of files
OLD_NUM_OF_FILES=$(ls -l $LOG_PATTERN* 2>/dev/null | wc -l)

# Start Tailing
run_thread

while [ 1 ]; do
    # If files are added - retail
    NUM_FILES=$(ls -l $LOG_PATTERN* 2>/dev/null | wc -l)
    if [ $NUM_FILES -ne $OLD_NUM_OF_FILES ]; then
        OLD_NUM_OF_FILES=$NUM_FILES
        kill $THREAD_PID
        run_thread
    fi
    sleep 1
done

Method 6

This works recursively.

It makes use of inotify (apt-get install inotify-utils on Debian) and just restarts the tail commands whenever new files/directories are discovered.

#!/bin/bash
logDir=/var/log/
while true; do
        ( find "$logDir" -type f -print0 | xargs -0 tail --verbose --follow --lines=0 )&
        tailPid=$!
        inotifywait --recursive --quiet --event create --format=%w%f "$logDir" | tr \n \0 | xargs -0 tail --verbose
        kill $tailPid
done

My use case was redirecting all logs to /dev/stderr inside a docker container to have them show up in docker-compose logs -f.

Method 7

Here’s a bash script that also uses inotifywait. It watches for any new or changed file and runs tail --verbose -n {NN} on that file, which prints the filename and last NN lines.

#!/usr/bin/bash
NLINES=75
function usage() {
  echo "Usage: $0 {folder} [#lines]
  Monitor {folder} and execute 'tail -n [#lines]'
  for any new or modified files.
  Default [#lines] is $NLINES.
"
  exit 1
}
[[ ! -z $1 ]] || usage
[[ -d $1 ]] || {
  echo "ERROR: $1 is not a a folder."
  usage
}
[[ -z $2 ]] || {
  [[ $2 =~ ^[0-9]+$ ]] && NLINES=$2 || {
    echo "ERROR: [#lines] must be an integer"
    usage
  }
}
inotifywait -m -e modify -e create --format=%w%f $1 | xargs -d 'n' -n 1 tail --verbose -n $NLINES

All methods was sourced from stackoverflow.com or stackexchange.com, is licensed under cc by-sa 2.5, cc by-sa 3.0 and cc by-sa 4.0