Sync Dev: Automating Task Execution Among Servers With Sync

Developing on the BitTorrent Sync API? Our developer evangelist is here to work with you.

This is a guest post by David Muñoz – Director at OK HOSTING, a Mexican company that offers corporate software development, web design and hosting services.

We have always had the need to move a lot of data from one computer to another, in order to backup our customer’s info and also to move hosting accounts from one server to another. For this purpose, we had always relied on FTP, which is not perfect. Most of the issues we got from FTP were lost connections, issues when moving files that contain special characters (like quotes) in the file name, disconnects and timeouts with big files, and the need to have a dedicated public IP on the destination server.

Then we saw BitTorrent Sync come out, and we instantly fell in love with the concept of unlimited, free and private data transfers. On top of that we had an API and a portable, multi-platform executable, perfect for automating it.

So we decided to try it, and liked it very much. Some weeks later we decided to use it as a substitution for FTP data transfers in our backups and migration processes. We came up with some code that can be used as a general messaging system between servers, using files as broadcast messages.

We use a C# + EntityFramework + MySQL very simple system that you can download here:

https://github.com/okhosting/OKHOSTING.Data.BittorrentSync

To automate Sync folder creation and destruction among all the servers, on demand.

Here’s how we do it:

  1. We install and execute our code and btsync on all the computers/servers that will conform the network.
  2. We have a shared btsync folder among all the computers, called CommandDirectory. This is used to send a broadcast signal to all computers all at once. Any computer on the network can simply create a file inside this folder and automatically sync with all of the other computers. This sync’ed folder is automatically created on setup using a predefined key.
  3. One of the computers creates a Task and saves it on the database, the trigger for this can be another parent-task, an event, or user input. When a new Task is created, a file is saved inside the CommandDirectory, containing the TaskId in the file name.
  4. All computers get a copy of the file in a few seconds thanks to btsync. A Task in our model represents a call to a particular method in a set of C# objects. It contains a System.Type name, a method name, optional arguments if the method requires them, an optional filter criteria that will be used to know in which objects the method will be executed, and an filter criteria for which computers should execute the task.
  5. For example, a task could contain this:

    Type: "OKHOSTING.ERP.Hosting.Vendors.Parallels.Plesk.PleskDomain, OKHOSTING.ERP.Hosting" (type's full name + assembly name)
    Method: "Backup(System.Boolean fullBackup, System.Int32 destinationServerId)"
    Arguments: "true, 7"
    Object Filter: "PleskDomain.ServerId = 3"
    Computer Filter: "Computer.ComputerId = 20"

    This would result in reading from database all objects of type PleskDomain that match the filter “PleskDomain.ServerId= 3”, then executing the method Backup(true, 7) in each of the retrieved objects, which will result in backing up all PleskDomains that are hosted in Server the server with id=5 and the resulting backup will be sent to the destination Server.Id=7 (these are the method arguments). In this case, the Task will only be executed on the computer with id=20. A lot of reflection here, since this is really a general purpose code for executing any kind of method of any kind of class in any computer.

  6. Every server is actively listening for new files being created in the CommandDirectory, so when a file is sync’ed our code reads the file and get the Task.Id. If the task’s Computer Filter includes the local computer where the code is running, then we try to execute the task and store the result in database. And the process repeats.
  7. Another task could be like this:

    Type: "OKHOSTING.Data.BittorrentSync.SharedDyrectory, OKHOSTING.Data.BittorrentSync"
    Method: "SynchOnceAndDestroy()"
    Arguments: ""
    Object Filter: "SharedDyrectory.Id = 170"
    Computer Filter: "Computer.ComputerId = 20"

    And this would result in the computer with id=20 executing the task, then reading the SharedDirectory with id=170 from database, retrieving the secrets and path where it should be configured, and create the shared directory using the btsync API in the local computer. In this case, the method “SynchOnceAndDestroy()” creates the folder, waits until it’s fully synched with all peers, and then destroys the folder.

We have found this implementation to be far more reliable and faster than FTP, and it turned out to be a very general tool that we can use to trigger task execution of any kind, on any server. Since we can call any method on any class, we can trigger all kinds of tasks, from creating and synching new btsync folders, to send an email or delete some files.

Thanks to the BitTorrent Sync team for creating such a great product, and allowing us to share our experiences.

Photo: Flickr/churkinms

Please follow and like us: