In addition, there are tools like Lsyncd<\/a> to help with this task. Unfortunately, one of the problems you\u2019ll encounter when you do this is that your data sets will never be fully in sync. <\/p>\n\n\n\n You\u2019ll also be in charge of building, scripting, troubleshooting, and supporting Rsync, Lsyncd, and any other tool you use to brute-force your way to parallel Rsyncs (not something you want to scale for enterprise business needs). Syncing multiple servers concurrently as fast as possible to keep them all in sync is enough to keep any admin busy, not considering the inevitable Rsync troubleshooting<\/a> when things break.<\/p>\n\n\n\n If you do get everything working, there are still other hurdles to overcome. Rsync is a single-threaded command-line sync solution; it can\u2019t capture incremental file changes in real time, especially if you have a larger file system with millions of files. It also falls short if you need multi-directional synchronization over WANs.<\/p>\n\n\n\n As a result, syncing large files or large numbers of files with Rsync<\/a> across systems can take too long to make it a feasible solution (one user found<\/a> it took almost 15 minutes to transfer just 8GB of data). This problem is increased exponentially when transferring to multiple endpoints. There\u2019s just no way to speed up Rsync enough to solve these problems.<\/p>\n\n\n\n If you\u2019re just syncing files and folders between two offices, consolidating files into one location, or pushing files out (one way only) to a handful of other remote servers, you can use Rsync to keep them reasonably well synced as long as you have a good network. But, for any more complex use cases (or when you need native security features \u2014 which Rsync lacks), you\u2019ll need an alternative solution.<\/p>\n\n\n\n Resilio Active Everywhere<\/em><\/a>\u00a0(formerly Resilio Connect) is a powerful alternative to\u00a0Rsync. To see for yourself how quickly Resilio syncs files of any size\/number across multiple endpoints,\u00a0<\/em>schedule a demo<\/em><\/a>.<\/em><\/p>\n\n\n\n Resilio Platform is designed to handle large replication environments and use cases. It uses a P2P (peer-to-peer) replication architecture which lets every device<\/em> in your replication environment share data with and receive data from other devices in the network. This enables blazing-fast replication speeds (we\u2019ve seen 100+ Gbps<\/a>), syncing in any direction, and organic scalability to sync files of any size and number to any number of endpoints. Plus, Resilio\u2019s WAN optimization protocol enables it to quickly and reliably sync files over any network \u2014 including high-latency, loss-prone WANs and edge networks.<\/p>\n\n\n\n In contrast to Rsync, Resilio Platform is:<\/p>\n\n\n\n 1. Reliable<\/strong> \u2014 There\u2019s no single point of failure and no data corruption. Files always sync reliably and on time.<\/p>\n\n\n\n 2. Fast<\/strong> \u2014 Sync files of any size and type across any number of servers in any direction throughout the world thanks to P2P replication and WAN optimization.<\/p>\n\n\n\n 3. Fluidly scalable<\/strong> \u2014 Simply add servers to a job to sync files across them.<\/p>\n\n\n\n 4. Easy to use<\/strong> \u2014 You can automate everything via scripts, APIs, or in the UI. Once it\u2019s set up, it \u201cjust works\u201d in the words of Deutsche Aircraft\u2019s IT manager<\/a>. <\/p>\n\n\n\n 5. Proven<\/strong> \u2014 Well-respected businesses such as Blizzard, Larian Studios<\/a>, Mercedes-Benz<\/a>, Cisco, Skywalker Sound<\/a>, Turner Sports, Lindblad Expeditions<\/a>, Exxon, and more rely on Resilio. Read more on our case studies page<\/a>. <\/p>\n\n\n\n 6. Well-supported<\/strong> \u2014 You don’t need to fuss with managing scripts and supporting them thanks to Resilio\u2019s world-class engineering and support. <\/p>\n\n\n\n 7. Predictable<\/strong> \u2014 You know when the job will complete. This consistency is huge when you’re synchronizing hundreds of servers for QA, development, and production. <\/p>\n\n\n\n 8. Transparent<\/strong> \u2014 There are no batch files or actions that users need to take. They just work in files and directories as they normally would. Apps use the files as files; it’s easy to integrate with apps and workflows via an API.<\/p>\n\n\n\n 9. Secure<\/strong> \u2014 Resilio comes with end-to-end AES-256-bit encryption, cryptographic data integrity validation, and more.<\/p>\n\n\n\n <\/p>\n\n\n\n In this article, we\u2019ll cover four different ways you can create parallel Rsyncs and share resources you can use to learn more about them. <\/strong>We\u2019ll also take a deep dive into how Resilio Platform replaces the need for parallel Rsyncs and what it can do for you instead.<\/p>\n\n\n\n Organizations such as Bungie, MixHits Radio, Match.com, Deutsche Aircraft, Mercedes-Benz, and more use Resilio Platform to achieve blazing-fast synchronization speeds that can reach <\/em>over 100 Gbps<\/em><\/a>. For a full demonstration of how Resilio\u2019s capabilities can optimize your application, <\/em>schedule a demo<\/em><\/a>.<\/em><\/p>\n\n\n\n We scoured blogs and forums to find four of the best tools and Rsync scripting workarounds to create parallel Rsync processes. We\u2019ll describe how each method works and share resources you can use to learn more and employ them.<\/p>\n\n\n\n NOTE<\/strong>: Before actually deploying these (or other) Rsyncs, you should always perform a dry-run to see which files will be affected and ensure your file system copies as intended.<\/p>\n\n\n\n GNU Parallel is a shell tool for Unix and Unix-like operating systems that executes jobs in parallel by splitting the input (which can be lists of files, hosts, users, URLs, and tables) and piping it into commands in parallel.<\/p>\n\n\n\n Put more simply, GNU Parallel acts as a manager, splitting your full list of files into separate smaller Rsync jobs.<\/p>\n\n\n\n You can find and install the latest release of GNU Parallel here<\/a>.<\/p>\n\n\n\n GNU Parallel commands are split into two parts:<\/p>\n\n\n\n The following is an example of a GNU Parallel command:<\/p>\n\n\n\n <\/p>\n\n\n\n The -j command is used to set the maximum number of Rsync jobs \u2014 in the case above, seven.<\/p>\n\n\n\n Parallel Rsync (aka msrsync) is an Rsync wrapper that splits transfers into multiple buckets and transfers them in multiple threads. You can download the wrapper for parallel Rsync on GitHub<\/a>.<\/p>\n\n\n\n One of the downsides of Parallel Rsync is that it can only be used over LAN connections. But, if you only need to transfer over LANs, Parallel Rsync is a great solution for maximizing bandwidth utilization and copying multiple pieces of data simultaneously.<\/p>\n\n\n\n The following is an example of a Parallel Rsync command:<\/p>\n\n\n\n <\/p>\n\n\n\n The -p command sets the maximum number of parallel rsync commands to run between the source directory (sender) and destination directory (destination). <\/p>\n\n\n\n Parasyncfp is a Perl script that wraps Rsync and enables parallel threading. It was designed specifically for transferring very large data sets over fast network connections and enables you to limit the bandwidth utilized, if necessary.<\/p>\n\n\n\n The following is an example of a parasync command:<\/p>\n\n\n\n <\/p>\n\n\n\n The breakdown of the key parts of this command is as follows:<\/p>\n\n\n\n <\/p>\n\n\n\n The command ends with source and destination directories.<\/p>\n\n\n\n More information and a link to download Parasyncfp can be found here<\/a>.<\/p>\n\n\n\n A commonly used solution to create parallel processes in Rsync is to launch multiple Rsync instances that run concurrently with different inputs. <\/p>\n\n\n\n While this can work, it can be complicated. You have to split your directory structure into smaller, equal parts that transfer separately. And you\u2019ll be forced to keep track of each Rsync instance, which can be cumbersome when deployed in large numbers.<\/p>\n\n\n\n One workaround to this is to use the find command<\/a> and split command<\/a> to divide your file list into multiple parts for your Rsync \u2014 a great description of how to execute this process using bash scripts can be found here<\/a>.<\/p>\n\n\n\n RSync may be a free tool. But the time spent trying to get Rsync to do what you want \u2014 i.e., researching Rsync scripts, applying and testing those scripts, trying to figure out what went wrong, and resolving scripting issues \u2014 comes at the cost of man hours and frustration. If you\u2019re syncing large files and\/or large numbers of files (and especially, if you\u2019re syncing to many endpoints), then Rsync is simply not worth the trouble.<\/p>\n\n\n\n Resilio Platform is a real-time, file synchronization software system that you can use to quickly sync files of any size, type, and number to any number of endpoints concurrently. It uses a P2P replication architecture \u2014 where every endpoint in your system can engage in the sync process simultaneously \u2014 to sync files 3-10x faster than legacy sync solutions, sync in any direction (one-way, two-way, one-to-many, many-to-one, and N-way), and eliminate single points of failure.<\/p>\n\n\n\n Resilio utilizes a proprietary WAN optimization protocol to enhance transfer over any network, including high-latency WANs, consumer-grade networks, and edge networks.<\/p>\n\n\n\n In this section, we\u2019ll discuss the features and capabilities that make Resilio Platform a superior solution for quickly synchronizing large files and environments, such as:<\/p>\n\n\n\n <\/p>\n\n\n\n Both Resilio Platform and Rsync engage in differential sync \u2014 i.e., syncing only the changed portions of files.<\/p>\n\n\n\n But Rsync uses a point-to-point replication architecture. In point-to-point replication, files can only be transferred between two devices at a time in one of three models:<\/p>\n\n\n\n <\/p>\n\n\n\n Point-to-point sync is slowed by the fact that file transfers are limited to two devices at a time. Achieving full synchronization of your environment will take a long time when syncing large files or large numbers of files across many devices. It also introduces single points-of-failure. If any one of the devices or networks goes down, it will impede the sync process as every other device must wait to receive their files.<\/p>\n\n\n\n Resilio Active Everywhere’s peer-to-peer sync architecture overcomes all of these limitations. In a P2P topology, every device can share files with and receive files from every other device. When syncing files, every device in your application can work together, allowing you to utilize the full bandwidth of your sync environment and achieve:<\/p>\n\n\n\n When synchronizing files, Resilio Platform uses a process known as file chunking to split files into several chunks. Each chunk can be transferred independently from the others. As soon as it receives a file chunk, each device can immediately share that chunk even before it receives the rest of the file.<\/p>\n\n\n\n Imagine you want to sync a file across five devices. Resilio Platform will split that file into five chunks. Device 1 can share the first chunk with Device 2. Device 2 can immediately share that first chunk with Device 3 while it waits to receive the other four chunks. With every device working together, you can sync your entire environment 3-10x faster than with traditional solutions like <\/strong>Rsync<\/strong>.<\/p>\n\n\n Unlike Rsync, Resilio Platform can also sync files in real time. Resilio utilizes optimized checksum calculations (identification markers assigned to each file that change whenever the file changes) and notification events from the host operating system to immediately detect and transfer file changes. Resilio can also perform manual or scheduled syncs.<\/p>\n\n\n\n While it\u2019s possible to run Rsync twice in two directories to approximate a two-way Rsync<\/a>, Rsync is a tool designed for one-way synchronization. This limits Rsync\u2019s utility for certain use cases, such as disaster recovery.<\/p>\n\n\n\n But Resilio Platform can sync in any direction, such as one-way, two-way, one-to-many, many-to-one, and N-way sync.<\/p>\n\n\n\n N-way sync is especially useful for use cases that require fast synchronization of many endpoints or high availability.<\/p>\n\n\n\nFour Ways to Run Rsync in Parallel for Faster Synchronization<\/h2>\n\n\n\n
1. GNU Parallel<\/h3>\n\n\n\n
\n
parallel -j 7 rsync A\/{} \/dest-dir\/<\/code><\/pre>\n\n\n\n2. Multi-Stream Rsync (msrsync)<\/h3>\n\n\n\n
$ msrsync -p 4 \/sender \/destination<\/code><\/pre>\n\n\n\n3. Parasyncfp<\/h3>\n\n\n\n
% parsyncfp --maxload=3.5 --NP=5 --chunksize=2154202 \\--startdir='[source directory name]' [destination directory name]<\/code><\/pre>\n\n\n\n\n
4. Launch Multiple Rsync Sessions<\/h3>\n\n\n\n
Resilio Active Everywhere: The Best Solution for Fast, Scalable Synchronization<\/h2>\n\n\n\n
\n
High-Performance, Real-Time Sync: The Speed and Scalability of P2P Replication<\/h3>\n\n\n\n
\n
A. Blazing Fast Replication Speeds (100+ Gbps per server)<\/h4>\n\n\n\n
![\"GIF](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2022\/09\/image2.gif\")
<\/figure>\n<\/div>\n\n\nB. Sync Files in Any Direction<\/h4>\n\n\n\n
![\"Hot\/Live](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/04\/hot-dr-warm-dr-cold-dr-and-offsite-copy-699x1024.png\")
<\/figure>\n<\/div>\n\n\n![\"Traditional](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/08\/aspera-vs-signiant-vs-resilio-3-1024x601.png\")
<\/figure>\n<\/div>\n\n\n![\"\"](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2024\/03\/dfs-replication-error-recovery-office.jpg\")
<\/figure>\n<\/div>\n\n\n![\"\"](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2024\/03\/finnacial-services-dr-1024x980.jpg\")
<\/figure>\n<\/div>\n\n\n