We’re going to write a script that watches a directory and acts on new files that are added. Each file is gzipped and moved to another directory, as soon as it is detected. The script uses the inotify subsystem, through a utility called inotify-tools. But first, let’s install the tool and experiment.
Installing inotify-tools and gzip
Use apt-get to install this package onto your system if you’re using Ubuntu or another Debian-based distribution. On other Linux distributions, use your Linux distribution’s package management tool instead.
Experimenting with inotify-tools
Let’s begin by watching a directory and seeing what events initiate when new files arrive. We will use a tool called inotifywatch , which is part of inotify-tools. Create a new directory called “incoming”:
Start watching this directory by executing the following command:
This will instruct inotify to watch for all filesystem events in the “incoming” directory. The -v option makes the tool print out extra information about what it’s doing. We haven’t specified a timeout option (-t ), and the command will keep gathering events until we exit with CTRL+C. At this point, our terminal should look something like this:
Open a new terminal window (or tab) and change to the incoming directory. Use the touch command to create a new file named “newfile.”
Now go back to the first terminal window and stop inotifywatch by hitting CTRL+C.
A table of events will be served to the console, indicating one instance of “create,” “open,” “attrib,” and “close_write.” These four events occurred when we used touch to create a new file, set its file access attributes, opened it to write a null terminating character, and then closed it afterward. These are just a few of the multitude of events that can be monitored on a filesystem with inotify-tools. You can see the full list on the main page for inotifywatch.
For our purposes we’re only interested in two events:
“create” – when a file is created in the target directory. “moved_to” – when a file is moved from another location into the target directory.
Let’s try inotifywatch again, but this time instructing it to monitor these two events only. Run this command in the first terminal window:
In the second terminal window or tab, let’s try creating a new file, changing its contents, and then moving the file from another location to the target directory. All these commands are run from the home directory.
Go back to the first terminal window and stop inotifywatch by hitting CTRL+C. We’ll see the following output:
Only two events were counted: creating a file called “created.txt” and moving an existing file called “created2.txt”. Everything else, such as modifying “created.txt,” was ignored.
Watching a Directory and Executing a Task
Now that we know what events to follow, we can use another tool called inotifywait to block execution until a file is created in or moved to our target directory. We’ll use the same arguments as we did with inotifywatch and also specify how we want the filename to be formatted for use in our task.
Before we begin, we need a directory to hold files that have already been processed. Create a directory called “processed”:
Next, create a new script called “watch-incoming.sh” and add the contents listed below:
The script executes the inotifywait command with the -m option. This makes the command monitor changes indefinitely. Each time a new event is detected, the filename is passed to the read command and injected into the “FILENAME” variable. The block under the while loop is executed, in which the file is first moved to the “processed” directory and then gzipped. The original file is replaced with the compressed file, and the filename will end in “.gz”.
Let’s grant execute permissions on this script and run it from our home directory.
Open the second terminal window and create a new file in the “incoming” directory. List the contents of both the “incoming” and “processed” directory to see the results of the event being detected:
The raw text file that we copied into the “incoming” directory was detected by the script, copied into “processed” and then compressed using gzip.
