More on the Synology NAS – automatically indexing new files

Having got lsyncd set up to automatically mirror my music, videos and pictures onto the Synology Diskstation NAS (see previous post) with the intention of using the DiskStation’s built in media server to stream them to other connected devices, there was one more hurdle to overcome.

When transferring files to the DiskStation through Windows shares or uploading through the Photo Station or File Station software that comes with it, the files are automatically indexed for use by the media server. However, the DiskStation doesn’t automatically detect and index files transferred via NFS or rsync, or files that are moved or modified directly through SSH.

synoindex

After searching around a bit, I found that it’s possible to index, unindex or reindex individual files or directories on the DiskStation using the “synoindex” command. I don’t remember where I first found that, but google it – it’s mentioned in a few different places. Running “synoindex -h” gives the usage, e.g. “synoindex -a filename” will add a file to the index. So, if I could find a way to monitor the filesystem and detect which files change, I could then run synoindex with the appropriate parameters to automatically index them.

False starts

There were a lot of false starts getting this to happen. If you just want to see how I did it, skip down a few paragraphs…

I first tried modifying the default “rsyncssh” module used by lsync on my desktop computer, to have it ssh to the DiskStation and run synoindex after remotely modifying files. But that turned out to be fraught with difficulty, and I thought there must be an easier and more general way that will also cover files transferred through NFS or any other means.

Next, I tried cross-compiling lsyncd to run on the DiskStation itself. The idea was to use lsyncd to monitor files on the DiskStation, and rather than copying them somewhere (the usual use case for lsyncd), run a custom command to trigger synoindex. The model of DiskStation I’m using uses an ARM V5 CPU. Cross compilers for that platform are readily available, and after some hitches, I got it to compile. However, I won’t bore you with the details because as soon as I the ARM version of lsyncd on the DiskStation, it segfaulted.

Rather than waste any more time trying to get that working, I dug a bit deeper. lsyncd monitors files using the Linux kernel’s “inotify” features. There are a set of tools called “inotifytools” that provide user level access to inotify, and one in particular, the “inotifywait” command seemed like it could be useful. inotifywait will monitor a file or directory, recursively if desired, and output a line of text whenever something changes. Feeding that output into another script, it should be possible to trigger synoindex whenever a file changes or is added or removed. Others on the internet had come up with similar ideas, for example, this post, where the author had used inotifywait to build a directory of the latest video files. Not quite what I wanted to do, but close. There’s one slight hitch: inotifytools are not installed by default on the DiskStation. To install them, you have to bootstrap the DiskStation and install optware. Details are in the previous link. After doing that, I got a perl script set up to take a feed from inotifywait and update the index as required. Again, I won’t bore you with the details because that too had a problem. Whenever the DiskStation “woke up” from sleep (not really sleep, but a kind of low power mode), the next time a file changed and the script was triggered, it would stop. Not crash, just stop, as if it had reached the end of input. I never quite got to the bottom of it, but I suspect that somehow the pipe between inotifywait and my script was being closed in low power mode.

So, no good there. Besides, having to bootstrap the box and add optware is not really idea. I wanted a more portable solution.

At last, something that works

Here’s what I came up with in the end (and this one does work – it’s been running continuously for 2 or 3 weeks now), using Python.

Step by step:

Install Python from the DiskStatio npackage manager.

You need a python package called “pyinotify” for Python to interface with inotify. You can get it here. Download the package somewhere on the DiskStation and follow the install instructions here.

Here’s my Python script to monitor files and run synoindex as appropriate when anything changes:

import pyinotify
import sys
import os.path
from subprocess import call
import signal

log_file = open("/var/log/monitor.log", "a")
	
def log(text):
	log_file.write(text + "\n")
	log_file.flush()
	
def signal_handler(signal, frame):
	log("Exiting")
	sys.exit(0)
	

log("Starting")

signal.signal(signal.SIGTERM, signal_handler)

allowed_exts = ["jpg", "jpeg", "png", "tga", "gif", "bmp", "mp3", "flac", "aac", "wma", "ogg", "ogv", "mp4", "avi"]

wm = pyinotify.WatchManager()  # Watch Manager
mask = pyinotify.IN_CLOSE_WRITE | pyinotify.IN_DELETE | pyinotify.IN_CREATE | pyinotify.IN_MOVED_TO | pyinotify.IN_MOVED_FROM  # watched events


class EventHandler(pyinotify.ProcessEvent):
	def __init__(self):
		self.modified_files = set()
		
	def process_IN_CREATE(self, event):
		self.process_create(event)
	
	def process_IN_MOVED_TO(self, event):
		self.process_create(event)
		
	def process_IN_MOVED_FROM(self, event):
		self.process_delete(event)
		
	def process_IN_DELETE(self, event):
		self.process_delete(event)
	
	def process_create(self, event):
		arg = ''
		if event.dir:
			arg = "-A"
		else:
			arg = "-a"
		self.do_index_command(event, arg)
	
	def process_delete(self, event):
		arg = ''
		if event.dir:
			arg = "-D"
		else:
			arg = "-d"
		self.do_index_command(event, arg)
		
	def process_IN_MODIFY(self, event):
		if self.is_allowed_path(event.pathname, event.dir):
			self.modified_files.add(event.pathname)
				
	def process_IN_CLOSE_WRITE(self, event):
		# ignore close_write unlesss the file has previously been modified.
		if (event.pathname in self.modified_files):
			self.do_index_command(event, "-a")
		
	def do_index_command(self, event, index_argument):
		if self.is_allowed_path(event.pathname, event.dir):
			log("synoindex %s %s" % (index_argument, event.pathname))
			call(["synoindex", index_argument, event.pathname])
			
			# Remove from list of modified files.
			try:
				self.modified_files.remove(event.pathname)
			except KeyError, e:
				# Don't care.
				pass
		else:
			log("%s is not an allowed path" % event.pathname)
		
			
	def is_allowed_path(self, filename, is_dir):
		# Don't check the extension for directories
		if not is_dir:
			ext = os.path.splitext(filename)[1][1:].lower()
			if ext not in allowed_exts:
				return False
		if filename.find("@eaDir") > 0:
			return False
		return True

handler = EventHandler()
notifier = pyinotify.Notifier(wm, handler)
wdd = wm.add_watch(["/volume1/music", "/volume1/photo", "/volume1/video"], mask, rec=True, auto_add=True)

try:
    notifier.loop(daemonize=True, pid_file='/var/run/monitor.pid')
except pyinotify.NotifierError, err:
    print >> sys.stderr, err

Put that script somewhere (I put mine in /volume1/Extensions/scripts/monitory.py).

Hopefully, most of this is pretty straightforward, though I may come back and add more explanation when I have more time. In lines 78-80, I set up inotify watchers and tell pyinotify to use my custom EventHandler class to handle events. EventHandler defines several methods that will be called in response to various inotify events. I’m only interested in certain events – file creation (or moved to), deletion (or moved from) and close write. Note, I’m not responding to every write or modify event, only close write, because otherwise there would be several write events resulting in several reindexes in some cases when a file is modified. Method is_allowed_path is used to exclude some files from indexing. I don’t want to attempt to index every file, only media files. Also, when copying files with rsync, this ensures that the temporary files that rsync creates won’t be indexed. I also exclude directories named “@eaDir”. These are created internally by the DiskStation to store thumbnails of images. Attempting to index those separately would not be a good idea.

Now you need a script to live in /usr/syno/etc/rc.d (equivalent of /etc/init.d on most systems) and start the python program automatically when the box is started. Here’s mine:

#!/bin/sh
#S99latestfileswatch.sh

case "$1" in
  start|"")

  #start the monitoring daemon
  python /volume1/Extensions/scripts/monitor.py

   ;;
  restart|reload|force-reload)
   echo "Error: argument '$1' not supported" >&2
   exit 3
   ;;
  stop)
   kill `cat /var/run/monitor.pid`
   ;;
  *)
   echo "Usage: S99latestfileswatch.sh [start|stop]" >&2
   exit 3
   ;;
esac

Save that as something in /usr/syno/etc/rc.d, e.g. “S99latestfileswatch.sh” (the “S99” prefix will ensure that it starts up at or near the end of the startup process). Start the script using “/usr/syno/etc/rc.d start”. After that, it should stay running in the background, and start automatically on every boot.

Advertisements

55 Comments

Filed under Uncategorized

55 responses to “More on the Synology NAS – automatically indexing new files

  1. Edit: I’ve changed the script so that the close write event now results in “synoindex -a” instead of “synoindex -R”. I thought R would reindex just the specified file, but instead it seems to reindex everything (which is very slow if you have a lot of files). The -a option seems to result in reindexing if the file already exists in the index.

  2. Tobias Beck

    Hi Mark,
    this looks very interesting to me. I am synchronizing two DiskStations with unison and the problem is that the media server is not recognizing the changes made to the music folder. Synoindex -R is time consuming, therfore I tried to setup pyinotify. While installing pyinotify with “sudo easy_install pyinotify” this message came up: “unable to execute /usr/local/i686-linux-gnu/bin/i686-linux-gnu-ccache-gcc: No such file or directory
    error: Setup script exited with error: command ‘/usr/local/i686-linux-gnu/bin/i686-linux-gnu-ccache-gcc’ failed with exit status 1”
    I don’t know how to go on. I am using a 1512+ with a cedarview architecture.
    Your help is highly appreciated. Thank you.
    Regards
    Tobias

  3. Hi Tobias,

    Thanks for your comment. I’m not sure I can be much help, because I didn’t encounter that error. But it looks like it’s trying to compile something. Because I had been fiddling about with other things before installing that python package, I already had GCC installed, which may be why it worked for me. Try installing GCC first and try again. Unfortunately, installing GCC on the diskstation is not entirely straightforward. I think you have to install optware first and go from there. Instructions are on the web somewhere, but I’ll try to dig out any notes on what I did.

    Mark

    • Tobias Beck

      Hi Mark,
      thanks for your reply. GCC is already installed. I already compiled other software like unison without problems. It seems that there is only a special part missing. Installing optware-devel didn’t fix it. I installed gcc with ipkg, this was straightforward. I’ll check about installing GCC.

      Tobias

  4. vagelis

    Hi,
    first of all i want to thank you for the script.
    I have installed it on a DS213+ and everything is running fine.
    The problem is that every time i select (highlight) a file on windows explorer (win7 Utlimate x64) a _IN_CLOSE_WRITE event is triggered. I did the same from another windows 7 machine and the same event is triggered (so it’s not just my pc). When i browse and select files from from windows XP or Linux everything is working fine, no extra events are triggered.
    So it’s not a problem with your script or inotify but with windows 7.
    Any solutions, explanations?
    Thanks in advance

    • Hi vagelis,

      Your comment reminded me that the script pasted here is not actually the most up to date version. I’ve made some changes since I wrote this page, and I’ve now updated it with the most recent version of the script.

      I don’t have a Windows 7 system to test with, but I have found occasionally when coping things with lsyncd, there can be a CLOSE_WRITE event when the file wasn’t modified. The solution is to keep track of MODIFY events too, and only respond to CLOSE_WRITE if there has been a previous MODIFY for the same file. Hopefully, this should help with your problem too.

      Mark

      • vagelis

        Hi Mark,
        thanks for your answer and thanks once again for sharing your script with us. I will try your solution (when i find some free time), and post back.

  5. Yann C.

    Brilliant ! Really !
    Just a little typo in your rc script, it should be :
    kill `cat /var/run/monitor.pid`
    and not :
    kill `cat /var/run/monitor.pl`

    Thank you so much for this script !

  6. Wello

    Hi, thanks for the effort to share this with us. It is strange Synology has not repaired this flaw themselves, even though it has been around for a long time (like 2010 already) and people have complained about it. I tried to follow your setup, but failed when I had to install Easy-install for Python and gave up. Another solution that does work for me is this one with a perl script to be added as a cronjob:

    http://www.naschenweng.info/2010/02/13/synology-automatic-indexing-via-synoindex/

    Any way, maybe this is of help to others who do not get past some of the steps you described above ๐Ÿ˜‰

  7. Edit: I’ve modified the script slightly from the original version, so that now it doesn’t respond to every CLOSE_WRITE event. Instead, there is a check that the file has been modified first. This is because sometimes I get phantom CLOSE_WRITE events without the file actually being modified, and vagelis’ comments suggest the same thing.

    I should really start keeping track of my code changes in a repository instead of pasting them into a blog.

    • Yann C.

      Oh great, I was coming back to say I had some problems with IN_CLOSE_WRITE looping when adding music (guess the indexing by/for audio station is firing those events), and the solution is already there !

      Thanks again.

  8. Karel

    Hi!
    This sounds like a great tool but I have the same issue as Tobias… unable to execute /home/spksrc/spksrc/toolchains/syno-cedarview/work/x86_64-linux-gnu/bin/x86_64-linux-gnu-gcc: No such file or directory
    Tobias did you manage to get it working?
    Anyone else with an idea?
    THANKS!!

    • I tried many ways to get this to work, and in the end, I just had to use this hack: ln -s /opt/bin/gcc /usr/local/i686-linux-gnu/bin/i686-linux-gnu-ccache-gcc

      You may also need to use this command to install pyinotify (so that the compiler will find the right Python.h):

      CC=/opt/bin/gcc CPPFLAGS=-I/opt/include/python2.7/ pip install pyinotify

  9. Jorman Franzini

    Hi, I’m trying to modify this script, to watch recursively a dir for media files, and then launch an external script. I almost done but I don’t figure out how to don’t watch a dir creation. Because if I create a new dir, that one will sended to the external script.

    Anyone can help me?

    J

  10. Because @eaDir[s] are created for each media file (containing the thumbnails and other metadata), it’s important to use the exclude_dir parameter in wm.add_watch to not even add inotify to those directories in order to save memory and startup time.

    For example:

    def exclude_filter(filename):
    return filename.find(“@eaDir”) != -1

    wdd = wm.add_watch([“/volume1/music/”, “/volume1/photo/”], mask, rec=True, auto_add=True, exclude_filter=exclude_filter)

  11. Matthias

    Works great on my DS412+! Thank you for sharing!

  12. Matthias

    One more thing – I think you’ve missed to add IN_MODIFY to mask. So line 25 would be like:

    mask = pyinotify.IN_MODIFY | pyinotify.IN_CLOSE_WRITE | pyinotify.IN_DELETE | pyinotify.IN_CREATE | pyinotify.IN_MOVED_TO | pyinotify.IN_MOVED_FROM # watched events

  13. I really like it when folks come together and share thoughts.
    Great blog, keep it up!

  14. I’m a Unix newbie, and I’m having problems with step 1, downloading easy_install. I have Python installed fine. But when I try to execute sudo through the terminal, I get a “not found”. wget does appear to work, but then I get ez_setup.py: Permission denied.

    What account should I be using? admin? root? What directory should I be in?

  15. bartux

    I manage to install and have everything runnig exept that Iย cannot see anything working. It there a way to check if the python script is running properly ?

    Thanks for the great job by the way.

  16. This is great, thanks so much for posting it!

    I think the whole easy_install thing is unnecessary, and skipping it would eliminate some of the problems people are having. Here is the simplest sequence of commands to install pyinotify – worked fine for me on DSM 5.0 without having gcc or anything else (besides Python) installed:

    wget https://pypi.python.org/packages/source/p/pyinotify/pyinotify-0.9.4.tar.gz

    tar -xzvf pyinotify-0.9.4.tar.gz

    cd pyinotify-0.9.4

    sudo python setup.py install

    (If a newer version of pyinotify is released, you can get the package download URL from the files list at https://pypi.python.org/pypi/pyinotify )

    Also, I noticed a few typos in the blog post: “monitory.py” where you meant “monitor.py”, and at the end where you show the command to start the service, you left off the script name itself, I believe it should be “/usr/syno/etc/rc.d/S99latestfileswatch.sh start” instead of just “/usr/syno/etc/rc.d start”

    Lastly, I think it would make sense to have the name of the startup script match the name of the Python file, to be less confusing when you’re looking at it again in a year or two. For my setup, I named the Python script ‘mediamonitor.py’ and the start script ‘S99mediamonitor.sh’.

    Oh, and ‘.m4v’ might be a reasonable addition to the file extensions list.

    • Thanks for your comment and corrections. As for me, I am no longer using this script because the latest version of DSM now includes a Linux CloudStation client. I found that using that to sync my files to the Diskstation, I no longer need to coax it into indexing the files – it seems to happen automatically.

  17. bshor

    Carl,

    Can you do a blog post with the start-to-finish instructions? I don’t want to get things wrong piecing together a several year old blog post and comments together ๐Ÿ™‚

  18. Morten

    Hi,

    I have used your script, but it seems like the synoindex do nothing on my photostation. The script is loading, and I can see in the log that it runs the command synoindex -a filename as it is supposed. However the file do not show up in the photostation.

    Can anybody help me?

  19. Minh

    Hi houghtonmark, Im having an error running this script:

    Traceback (most recent call last):
    File “/volume1/homes/admin/pyinotify2.py”, line 1, in
    import pyinotify
    File “/volume1/homes/admin/pyinotify.py”, line 25, in
    wm = pyinotify.WatchManager()
    AttributeError: ‘module’ object has no attribute ‘WatchManager’

    • Minh

      Did I not install pyinotify properly? I’m totally newbie at this, hope you can help.

      • Koci72

        Hi,
        I had a similar problem. I copied the pyinotify directories and files from /usr/lib/python2.7/site-packages to python2.7 install directory site-packages and its works fine!

  20. Hi all,

    If it helps anyone, I’ve posted my complete setup, with instructions, in a git repo at https://github.com/carljm/synology-mediamon

  21. tomterl

    So Cool! Thanks for sharing!

    I stumbled over this post accidentally. This was the last, missing brick in my setup — now it’s finished and perfect for my needs — automatic upload of my photos resulting in automatic indexing for the media server, tracks usable as soon as ripped music files are present (this will speed up my progress in converting my whole cd collection — instant gratification is always a plus ๐Ÿ˜€ ).

  22. r1z8

    This works like a charm. Can’t thank you guys enough, this is the only setup I’ve found that does the job of properly indexing files that are downloaded by Sabnzbd / Sickbeard. Thought I’d never get it working!

  23. wessel

    Thank you so much for this script. One question though: I almost never restart my box. is it possible to run the script every so often?

  24. You generally don’t need to restart to use this script – it just runs in the background once it’s set up. For what it’s worth though, I don’t personally use this any more. When I originally wrote this, there wasn’t a stable Linux Cloudstation client. There is now, so I just use that to sync files, and I find that files synced through CloudStation get indexed automatically without my script.

  25. TironBanes

    I came over this script because Synology media indexing is still not recognizing media that are linked via hard link (to a encrypted share).
    Though I think I have made all the steps it won’t run ๐Ÿ˜ฆ

    When I start S99mediamon.sh nothing seems to happen. When I CTRL-C than I get this and I don’t have a clue what’s wrong…

    MyDiskStation> /usr/local/etc/rc.d/S99mediamon.sh start
    ^CTraceback (most recent call last):
    File “/root/mediamon.py”, line 126, in
    auto_add=True,
    File “/usr/local/lib/python2.7/site-packages/pyinotify-0.9.6-py2.7.egg/pyinotify.py”, line 1934, in add_watch
    for rpath in self.__walk_rec(apath, rec):
    File “/usr/local/lib/python2.7/site-packages/pyinotify-0.9.6-py2.7.egg/pyinotify.py”, line 2118, in __walk_rec
    for root, dirs, files in os.walk(top):
    File “/usr/lib/python2.7/os.py”, line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
    File “/usr/lib/python2.7/os.py”, line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
    File “/usr/lib/python2.7/os.py”, line 296, in walk
    for x in walk(new_path, topdown, onerror, followlinks):
    File “/usr/lib/python2.7/os.py”, line 286, in walk
    if isdir(join(top, name)):
    File “/usr/lib/python2.7/genericpath.py”, line 41, in isdir
    st = os.stat(s)
    KeyboardInterrupt

    Someone any idea?

  26. Pingback: Synology DiskStation: Medien-Indizierung (synoindex) per Script/Kommandozeile - indiBit

  27. Richard Bignell

    Hi,

    This didn’t work for me. I couldn’t get python2 to work with pyinotify as i got this error:

    bash-4.3# python -m pyinotify -v /tmp
    Traceback (most recent call last):
    File “/usr/lib/python2.7/runpy.py”, line 162, in _run_module_as_main
    “__main__”, fname, loader, pkg_name)
    File “/usr/lib/python2.7/runpy.py”, line 72, in _run_code
    exec code in run_globals
    File “build/bdist.linux-armv7l/egg/pyinotify.py”, line 2386, in
    File “build/bdist.linux-armv7l/egg/pyinotify.py”, line 2340, in command_line
    File “build/bdist.linux-armv7l/egg/pyinotify.py”, line 1776, in __init__
    File “build/bdist.linux-armv7l/egg/pyinotify.py”, line 128, in create
    File “build/bdist.linux-armv7l/egg/pyinotify.py”, line 212, in init
    File “/usr/lib/python2.7/ctypes/util.py”, line 247, in find_library
    raise RuntimeError(“can not find library %s” % name)
    RuntimeError: can not find library c

    I can get pyinotify to run with python3 but i get this error:

    bash-4.3# python3 extensions/scripts/monitory.py
    File “extensions/scripts/monitory.py”, line 77
    except KeyError, e:
    ^
    SyntaxError: invalid syntax

  28. roland65

    Yes, the trick is to use python3 instead of python in the file S99latestfileswatch.sh with the following fix:
    In monitor.py, replace line 77 with:
    except KeyError as e:
    and line 100 with:
    except pyinotify.NotifierError as err:

    Also, the file S99latestfileswatch.sh has to be copied in /usr/syno/etc/rc.sysv instead of /usr/syno/etc/rc.d that seems to not exist anymore.

    With these fixes, indexation works as expected.

  29. Richard Bignell

    Thanks so much for the reply roland. I was worried no one would be subscribed to this post anymore due to its age.

    I tried your fix but i’ve hit another issue. I think I’m close now.

    bash-4.3# ./S99latestfileswatch.sh start
    Traceback (most recent call last):
    File “/volume1/extensions/scripts/monitor.py”, line 96, in
    wdd = wm.add_watch([“/volume1/music”, “/volume1/photo”, “/volume1/video”], mask, rec=True, auto_add=True)
    File “/volume1/@appstore/py3k/usr/local/lib/python3.5/site-packages/pyinotify.py”, line 1919, in add_watch
    exclude_filter)
    File “/volume1/@appstore/py3k/usr/local/lib/python3.5/site-packages/pyinotify.py”, line 1836, in __add_watch
    wd = self._inotify_wrapper.inotify_add_watch(self._fd, path, mask)
    File “/volume1/@appstore/py3k/usr/local/lib/python3.5/site-packages/pyinotify.py”, line 153, in inotify_add_watch
    return self._inotify_add_watch(fd, pathname, mask)
    File “/volume1/@appstore/py3k/usr/local/lib/python3.5/site-packages/pyinotify.py”, line 246, in _inotify_add_watch
    pathname = pathname.encode(sys.getfilesystemencoding())
    UnicodeEncodeError: ‘utf-8’ codec can’t encode character ‘\udc8f’ in position 69: surrogates not allowed

  30. Richard Bignell

    Trying to google for the answer but all i’ve found is a suggestion that add_watch should be encoded to string before use:

    def inotify_add_watch(self, fd, pathname, mask):
    # Unicode strings must be encoded to string prior to calling this
    # method.
    assert isinstance(pathname, str)
    return self._inotify_add_watch(fd, pathname, mask)

    • Richard Bignell

      Hi everyone,

      I’m still struggling. I must be really close to getting this working. Can anyone help?

      Rik

  31. Barbe

    Hi. I have same issue with “surrogates not allowed”. Do anyone have an answer to this issue ?

  32. Vincent Barberet

    Well, I’ve found a dirty workaround.
    The issue is that I have some filenames that are not UTF-8 encoded, but my filesystem is UTF-8 (use python command “sys.getfilesystemencoding()” to get your filesystem encoding).
    So, the real issue is that I have bad filenames. As I cannot fix easily those filenames, I have to live with them.

    Here is what I did:
    in /volume1/@appstore/py3k/usr/local/lib/python3.5/site-packages/pyinotify.py, line 246:
    I have put:
    pathname = pathname.encode(sys.getfilesystemencoding(), ‘surrogateescape’)
    instead of
    pathname = pathname.encode(sys.getfilesystemencoding())

    By the way, do any of you know why this live indexing function is not part of synology basic functions ?

    • Jean Francois LEON

      HI I have a different question:
      Do you know which linux app I could use to remotely search my indexed synology folder mounted as nfs share on my laptop?
      I note that spotlight can search these indexes.. but I dont own a mac
      Thanks
      JF

  33. JohnD

    How about simply running something like this every 15 minutes (as a cronjob):

    find /targetfolder -cmin -20 | xargs -I file synoindex -a file

    Advantages:
    1) simpler than either this perl script or some python alternatives using some notify libraries.
    2) does not require the synoindex -g option which apparently no longer works according to comments below

    Disadvantages – ?? TO BE CONFIRMED – can you please help ??
    1) If the file happens to already be there, this would just cause the same entry to be overwritten?
    2) I guess it consumes more to run this every 15′ rather than working with triggers from the notify library, but how bad is that really?
    3) Anything else I am not understanding?

    Unfortunately I was not able to eliminate the need for manual reindexing. In my case, the problem seems to be that I add new files from another DSM over a mount point, which does not trigger indexing automatically. Replacing the mount approach with the new Cloud Sync does not allow me to specify specific folders to replicate, only top-level DSM shared folders.

    • JohnD

      I got this to work now as a simple batch script, no Python or Perl required. I have scheduled this script for crontab execution every 10 minutes. I had to handprogram a loop because piping it to xargs seemed to fail when there is an apostrophe in the filenames – no doubt some unix wizard can compress the below into one line but it works for me ๐Ÿ™‚

      #!/bin/sh -x
      logfile=”/volume1/homes/admin/etc/IndexDownloads/indexdownloads.log”
      recentfile=”/volume1/homes/admin/etc/IndexDownloads/recentfile.txt”

      echo “———————–” >>$logfile
      echo ‘Starting indexing of new downloads at: ‘`date` >>$logfile
      echo >> $logfile

      find /volume1/NBU/Downloads -cmin -15 >$recentfile

      while read aline
      do
      echo “Processing: $aline” >> $logfile 2>&1
      synoindex -a “$aline” >>$logfile 2>&1
      done > $logfile

  34. Richard Bignell

    Hi JohnD,

    I need some help with your code. im not sure the characters are pasting correctly into vi so i had to correct some of the questionmarks and backticks but now cant get it to run.

    I’ve pasted into codedump so you can see what i’ve attempted.

    The log files are touched and writable and the script is executable but when i run i just get the following:

    bash-4.3# ./synoscript.sh
    + logfile=/volume1/share/indexdownloads.log
    + recentfile=/volume1/share/recentfile.txt
    + echo ____________
    ++ date
    + echo ‘Starting indexing of new downloads at: Sun’ Sep 18 11:27:08 IST 2016
    + echo
    + find /volume1/video -cmin -15
    + read aline

    + echo ‘Processing: ‘
    + synoindex -a ”
    + read aline

    + echo ‘Processing: ‘
    + synoindex -a ”
    + read aline
    ^C

    https://codedump.io/share/PWrA4YXRqn6p/1

    • JohnD

      Seems to be some weird copy paste error on my part in my post from August 26. Here is the correct script which is definitely working for me:

      #!/bin/sh -x
      logfile=”/volume1/homes/admin/etc/IndexDownloads/indexdownloads.log”
      recentfile=”/volume1/homes/admin/etc/IndexDownloads/recentfile.txt”

      echo “———————–” >>$logfile
      echo ‘Starting indexing of new downloads at: ‘`date` >>$logfile
      echo >> $logfile

      find /volume1/NBU/Downloads -cmin -15 >$recentfile

      #find /volume1/NBU/Downloads -cmin -50 | sed ‘s/\x27/\\\x27/’ | xargs -t -I file synoindex -a \’file\’ >>$logfile 2>&1

      while read aline
      do
      echo “Processing: $aline” >> $logfile 2>&1
      synoindex -a “$aline” >>$logfile 2>&1
      done > $logfile
      echo ‘Ending indexing of new downloads at: ‘`date` >>$logfile

  35. JohnD

    OK, that is funky. The same copy-paste error again. Somehow parts of my script are taken as format codes or something when I hit post. Let me see if I can edit the original and delete the later mails.

  36. Richard Bignell

    Thanks John for such a fast reply and upload of the corrected script.

    Its working great. Just adding it to cron now:

    MAILTO=””
    PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/syno/sbin:/usr/syno/bin:/usr/local/sbin:/usr/local/bin
    #minute hour mday month wday who command
    */10 * * * * /volume1/homes/admin/synoscript.sh

    synoservice -restart crond

    I got the cron instructions here:

    http://www.multigesture.net/articles/how-to-use-cron-on-a-synology-nas/

    Does anyone know how to make this cron change persistent after a reboot?

  37. lzap

    John, no need to do a shell loop at all, here is the xargs version:

    find /volume1/video/ -type f -cmin -15 -print0 | xargs -0n1 synoindex -a

    Take care!

    • lzap

      I bet you had problems with special characters, that’s why I use -print0 option to avoid these. (I just saw the commented version).

  38. Brilliant work. Stumbling through instructions as not a proper dev but I think I have it running:

    24928 adrianl+ 20 0 4.5m 1.0m 0.6 0.2 0:00.04 S grep –color=auto mediamon

    Few questions as it it doesn’t seem to have picked up a file yet successfully:

    1) Will all sub folders be automatically monitored to those listed in the script (e.g. video)?

    2) Do I need to disable the existing Sync /index thing, if so how? Is it the ‘conversion process’ that you can see from the top of the Synology gui ?

    3) Please could you do a noob description of what the high level process is, e.g.

    Mediamon monitors for updates, moves etc etc of FILES/FOLDERS into specified (monitored) Linux directories, and when it sees one it…..

    I assume it is or will work, but am keen to more granularly interrogate and am a bit out of my depth.

    Thanks very much!

    Adrian

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s