Having got lsyncd set up to automatically mirror my music, videos and pictures onto the Synology Diskstation NAS (see previous post) with the intention of using the DiskStation’s built in media server to stream them to other connected devices, there was one more hurdle to overcome.
When transferring files to the DiskStation through Windows shares or uploading through the Photo Station or File Station software that comes with it, the files are automatically indexed for use by the media server. However, the DiskStation doesn’t automatically detect and index files transferred via NFS or rsync, or files that are moved or modified directly through SSH.
After searching around a bit, I found that it’s possible to index, unindex or reindex individual files or directories on the DiskStation using the “synoindex” command. I don’t remember where I first found that, but google it – it’s mentioned in a few different places. Running “synoindex -h” gives the usage, e.g. “synoindex -a filename” will add a file to the index. So, if I could find a way to monitor the filesystem and detect which files change, I could then run synoindex with the appropriate parameters to automatically index them.
There were a lot of false starts getting this to happen. If you just want to see how I did it, skip down a few paragraphs…
I first tried modifying the default “rsyncssh” module used by lsync on my desktop computer, to have it ssh to the DiskStation and run synoindex after remotely modifying files. But that turned out to be fraught with difficulty, and I thought there must be an easier and more general way that will also cover files transferred through NFS or any other means.
Next, I tried cross-compiling lsyncd to run on the DiskStation itself. The idea was to use lsyncd to monitor files on the DiskStation, and rather than copying them somewhere (the usual use case for lsyncd), run a custom command to trigger synoindex. The model of DiskStation I’m using uses an ARM V5 CPU. Cross compilers for that platform are readily available, and after some hitches, I got it to compile. However, I won’t bore you with the details because as soon as I the ARM version of lsyncd on the DiskStation, it segfaulted.
Rather than waste any more time trying to get that working, I dug a bit deeper. lsyncd monitors files using the Linux kernel’s “inotify” features. There are a set of tools called “inotifytools” that provide user level access to inotify, and one in particular, the “inotifywait” command seemed like it could be useful. inotifywait will monitor a file or directory, recursively if desired, and output a line of text whenever something changes. Feeding that output into another script, it should be possible to trigger synoindex whenever a file changes or is added or removed. Others on the internet had come up with similar ideas, for example, this post, where the author had used inotifywait to build a directory of the latest video files. Not quite what I wanted to do, but close. There’s one slight hitch: inotifytools are not installed by default on the DiskStation. To install them, you have to bootstrap the DiskStation and install optware. Details are in the previous link. After doing that, I got a perl script set up to take a feed from inotifywait and update the index as required. Again, I won’t bore you with the details because that too had a problem. Whenever the DiskStation “woke up” from sleep (not really sleep, but a kind of low power mode), the next time a file changed and the script was triggered, it would stop. Not crash, just stop, as if it had reached the end of input. I never quite got to the bottom of it, but I suspect that somehow the pipe between inotifywait and my script was being closed in low power mode.
So, no good there. Besides, having to bootstrap the box and add optware is not really idea. I wanted a more portable solution.
At last, something that works
Here’s what I came up with in the end (and this one does work – it’s been running continuously for 2 or 3 weeks now), using Python.
Step by step:
Install Python from the DiskStatio npackage manager.
Here’s my Python script to monitor files and run synoindex as appropriate when anything changes:
import pyinotify import sys import os.path from subprocess import call import signal log_file = open("/var/log/monitor.log", "a") def log(text): log_file.write(text + "\n") log_file.flush() def signal_handler(signal, frame): log("Exiting") sys.exit(0) log("Starting") signal.signal(signal.SIGTERM, signal_handler) allowed_exts = ["jpg", "jpeg", "png", "tga", "gif", "bmp", "mp3", "flac", "aac", "wma", "ogg", "ogv", "mp4", "avi"] wm = pyinotify.WatchManager() # Watch Manager mask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_DELETE | pyinotify.IN_CREATE | pyinotify.IN_MOVED_TO | pyinotify.IN_MOVED_FROM # watched events class EventHandler(pyinotify.ProcessEvent): def __init__(self): self.modified_files = set() def process_IN_CREATE(self, event): self.process_create(event) def process_IN_MOVED_TO(self, event): self.process_create(event) def process_IN_MOVED_FROM(self, event): self.process_delete(event) def process_IN_DELETE(self, event): self.process_delete(event) def process_create(self, event): arg = '' if event.dir: arg = "-A" else: arg = "-a" self.do_index_command(event, arg) def process_delete(self, event): arg = '' if event.dir: arg = "-D" else: arg = "-d" self.do_index_command(event, arg) def process_IN_MODIFY(self, event): if self.is_allowed_path(event.pathname, event.dir): self.modified_files.add(event.pathname) def process_IN_CLOSE_WRITE(self, event): # ignore close_write unlesss the file has previously been modified. if (event.pathname in self.modified_files): self.do_index_command(event, "-a") def do_index_command(self, event, index_argument): if self.is_allowed_path(event.pathname, event.dir): log("synoindex %s %s" % (index_argument, event.pathname)) call(["synoindex", index_argument, event.pathname]) # Remove from list of modified files. try: self.modified_files.remove(event.pathname) except KeyError, e: # Don't care. pass else: log("%s is not an allowed path" % event.pathname) def is_allowed_path(self, filename, is_dir): # Don't check the extension for directories if not is_dir: ext = os.path.splitext(filename)[1:].lower() if ext not in allowed_exts: return False if filename.find("@eaDir") > 0: return False return True handler = EventHandler() notifier = pyinotify.Notifier(wm, handler) wdd = wm.add_watch(["/volume1/music", "/volume1/photo", "/volume1/video"], mask, rec=True, auto_add=True) try: notifier.loop(daemonize=True, pid_file='/var/run/monitor.pid') except pyinotify.NotifierError, err: print >> sys.stderr, err
Put that script somewhere (I put mine in /volume1/Extensions/scripts/monitory.py).
Hopefully, most of this is pretty straightforward, though I may come back and add more explanation when I have more time. In lines 78-80, I set up inotify watchers and tell pyinotify to use my custom EventHandler class to handle events. EventHandler defines several methods that will be called in response to various inotify events. I’m only interested in certain events – file creation (or moved to), deletion (or moved from) and close write. Note, I’m not responding to every write or modify event, only close write, because otherwise there would be several write events resulting in several reindexes in some cases when a file is modified. Method is_allowed_path is used to exclude some files from indexing. I don’t want to attempt to index every file, only media files. Also, when copying files with rsync, this ensures that the temporary files that rsync creates won’t be indexed. I also exclude directories named “@eaDir”. These are created internally by the DiskStation to store thumbnails of images. Attempting to index those separately would not be a good idea.
Now you need a script to live in /usr/syno/etc/rc.d (equivalent of /etc/init.d on most systems) and start the python program automatically when the box is started. Here’s mine:
#!/bin/sh #S99latestfileswatch.sh case "$1" in start|"") #start the monitoring daemon python /volume1/Extensions/scripts/monitor.py ;; restart|reload|force-reload) echo "Error: argument '$1' not supported" >&2 exit 3 ;; stop) kill `cat /var/run/monitor.pid` ;; *) echo "Usage: S99latestfileswatch.sh [start|stop]" >&2 exit 3 ;; esac
Save that as something in /usr/syno/etc/rc.d, e.g. “S99latestfileswatch.sh” (the “S99” prefix will ensure that it starts up at or near the end of the startup process). Start the script using “/usr/syno/etc/rc.d start”. After that, it should stay running in the background, and start automatically on every boot.