Rsync is designed for one-way synchronization, so setting up an Rsync two-way sync isn’t actually possible. However, you can use Rsync to set up a one-way sync in two directions — from A to B and from B to A. While this may suffice for some simple, homogenous sync configurations, there are issues with how Rsync executes such a sync setup that you’ll need to be aware of and address. (For example, how Rsync resolves file conflicts on your source and destination directories.)
In this article, we’ll discuss how to set up a two-way sync with Rsync — i.e., a one-way sync in both directions, instead of a true two-way sync — plus some of the issues you may run into and how to overcome them. Overall, we don’t recommend Rsync for businesses that need to sync large files, many files, or files in remote locations, because:
- Setting up and managing synchronization (particularly synchronizing one way in both directions) with Rsync can be complex and time-consuming.
- Synchronization with Rsync can be slow and unreliable (particularly for large file systems).
- Rsync does not use WAN optimized transfer protocols, making it a poor solution for syncing files over high-latency, lossy networks.
- Rsync offers no encryption, leaving your data exposed to interception by malicious parties.
We’ll also discuss our own alternative to Rsync , Resilio Connect, and how it provides true bidirectional synchronization and superior file synchronization.
Resilio is a high-performance file sync solution that uses a peer-to-peer file transfer architecture, which enables it to sync 3-10x faster than traditional sync solutions and scale organically in order to grow with your sync needs.
A one-way sync in two directions (which Rsync can perform) synchronizes two systems sequentially, but not simultaneously (i.e., first syncing system A to system B, then B to A after).
A true bidirectional sync occurs when two systems are synced simultaneously. Resilio’s P2P architecture not only enables it to perform true bidirectional synchronization, it can also sync multiple systems, across multiple sites, in multiple directions simultaneously (i.e., one-way, two-way, one-to-many, many-to-one, and N-way sync).
Resilio Connect also uses proprietary WAN optimization technology to optimize file transfer over any network, including long-distance, high-latency networks and in areas with poor network coverage. It can also be deployed on your existing infrastructure (in as little as 2 hours) and easily managed with a user-friendly Management Console (as well as GUI and command-line interface).
Organizations in gaming, tech, and media — such as Larian Studios, Wargaming.net, and Turner Sports — use Rsync to sync files across their replication environment and enhance business workflows.
To learn more about using Resilio Connect for file synchronization, schedule a demo.
How to Perform Approximate 2-Way Sync with Rsync
As stated earlier, Rsync can’t perform true bidirectional sync. Instead, you’ll need to run Rsync twice in both directories — i.e., sync from source to destination and from destination to source.
Rsync -[sync parameters] [source dir] [destination dir]
Rsync -[sync parameters] [destination dir] [source dir]For example, if you wanted to sync a folder on your desktop with a Dropbox folder (while also enabling archive mode, -a, and the verbose command displaying detailed info of the transfer, -v), you could use the following command:
Rsync -av [desktop folder name] [Dropbox folder name]
Rsync -av [Dropbox folder name] [desktop folder name]Potential Issues to Address When Using Rsync in Two Directions
Since Rsync is designed for one-way sync, two-way sync deployments are imperfect and require some modifications and fine-tuning (and even then, you may still run into issues). We’ve compiled a list of several common problems you may run into when using Rsync to sync in two directions and potential solutions you can use to address them.
1. Double Check Your Sync
You can (and should) use the -n Rsync command to perform a dry run and double check the syntax of your bidirectional sync set up.
Rsync -n[sync parameters] [source] [destination]
Rsync -n[sync parameters] [destination] [source]Doing so will help you spot any issues with your sync set up that you need to resolve.
2. Syncing File Deletions from Source to Destination
Imagine you’ve created a bidirectional sync of source file systems and destination file systems. When you delete a file on the source, you notice that the file does not delete on the destination.
In order to specify how file deletions are synchronized, you’ll need to use the following delete commands:
| –delete | Rsync deletes extraneous files from the destination directory |
| –delete-before | The receiver deletes files before transfer |
| –delete-during | The receiver deletes files during transfer, not before |
| –delete-delay | The receiver finds deletions during transfer, then deletes them after |
| –delete-after | The receiver deletes files after the transfer |
3. How to Address File Conflicts During Sync
If you’re running a simple sync of a small, homogenous environment, you likely won’t have much to worry about. But in more complex situations, you may run into file conflicts that need to be addressed, such as:
- When Rsync attempts to sync a file that exists on the source but not the destination (or vice versa).
- When the same file is changed (in different ways) on the source and the destination, which version should be synced.
Instead of creating complex Rsync commands to address every possible issue, you can use similar file sync solutions that give you a greater degree of control over specifying copy parameters, such as:
- Rclone: Rclone is a feature-rich, command-line file sync solution that gives you greater control over how file conflicts are resolved.
- Bsync (Github link): Bsync is a bidirectional file sync solution that uses Rsync for file transfers and provides a more reliable alternative to Unison.
How Resilio Connect Provides Superior File Sync
While a multi-directional Rsync may be suitable for simple, homogenous sync jobs, its performance suffers as your sync needs grow and become more complex. And even if you can successfully set up a sync in two directions, monitoring Rsync progress, fine-tuning your sync commands, and resolving the potential sync issues that can arise takes time and effort.
Resilio Connect makes performing a two-way sync simple and easy, and addresses several other Rsync problems, such as providing:
- Easier deployment and management of your sync environment
- Faster sync in any direction with P2P Transfer
- Optimized WAN transfer
- End-to-end security
Easier Deployment and Management of Your Sync Environment
Resilio Connect’s BYO storage model means that it can be easily deployed on your existing infrastructure and begin replicating in as little as 2 hours.
Resilio Connect is a flexible solution that works on:
- Multiple operating systems: Resilio works on multiple operating systems and can be configured cross-platform on Windows, Linux, Ubuntu, Unix, FreeBSD, macOS, and all major NAS solutions.
- Multiple set-ups: Resilio can be deployed on-prem, in cloud, or in cloud hybrid scenarios. It supports a variety of servers, desktops, virtual cloud servers, and cloud storage platforms. Resilio also offers an iOS and Android app for use on mobile devices.
While Rsync requires you to create and fine-tune complex Rsync commands, Resilio’s Management Console makes creating sync jobs easy. Simply add folders to a sync job and specify how the sync should occur. You get granular control over sync parameters, such as bandwidth usage, buffer size, disk I/O threads, and more.
The Management Console also makes it easy to monitor and troubleshoot sync errors. You’ll receive real-time notifications of transfer status, detailed transfer logs, and more; so you’ll always have deep insight into the state of replication in your environment.
Case Study: Voicebase
Voicebase is a software used to provide speech-to-text for audio and video transcriptions. Voicebase uses Resilio Connect to distribute large speech models files (of 50+ GB) to over 400 servers.
“Resilio Connect enables us to reliably distribute our code, specifically new language models, in a fraction of time. These copy jobs now take an hour, down from eight. Best of all, once Resilio Connect was installed, it just works: We never need to manually intervene in any way.”
Faster Sync in Any Direction with P2P Transfer
Rsync uses a point-to-point file transfer topology for sync jobs. In other words, file transfers either occur in a “follow-the-sun” model (where files are replicated from one server to another sequentially) or in a hub-and-spoke model (where files are first replicated to a hub server, which then replicates them to every other server one-by-one).
In either case, file transfer with Rsync only occurs between two servers at a time. Not only does this expose you to bottlenecks (if a transfer is interrupted or delayed between two servers, every other server must wait to receive files), but it results in slower sync speeds — particularly when Rsync is synchronizing large files, large numbers of files, or syncing to many endpoints.
Resilio Connect was built on proven peer-to-peer replication technology. In a P2P system, every server can replicate files to every other server in your environment simultaneously.
P2P replication enables Resilio Connect to:
- Eliminate bottlenecks: Synchronization won’t be impeded by slow networks or hardware failures, as any server can retrieve the necessary files or services from any other server in your environment.
- Sync in every direction: It also enables Resilio to sync files in any direction, including one-to-one, bidirectional, one-to-many, many-to-one, and N-way sync.
- Scale organically: Since every server can take part in replication, adding more servers to your sync environment only increases bandwidth and transfer speed. Resilio can sync to hundreds of servers in roughly the same time that it would take Rsync to sync two servers.
Resilio uses notifications from the host operating system and optimized checksum calculations (identification markers assigned to each file that change whenever a change is made to the file) to spot and sync file changes in real-time. And it uses a process known as file chunking — where files are broken down into multiple chunks that can transfer independently of each other — so every server can take part in synchronization simultaneously.
Imagine you need to sync a file among five servers. Resilio will break that file into (for example) five chunks. Server 1 can share the first chunk with Server 2. Once Server 2 receives the first chunk, it can immediately share it with Server 3, and so on.
Resilio’s P2P architecture, real-time synchronization capabilities, and file chunking enable it to sync files 3-10x faster than traditional, point-to-point solutions like Rsync.
Optimized WAN Transfer
Rsync uses transfer protocols that aren’t designed for transfer over high-latency, lossy WAN networks. That means that, when transferring over WANs, synchronization will occur slowly and poorly (i.e., there will be data loss).
But Resilio Connect uses a proprietary WAN optimization protocol known as Zero Gravity Transport™ (ZGT). ZGT optimizes transfer over WANs using:
- Congestion control: ZGT uses a congestion control algorithm to constantly calculate the send rate for data packets. While most protocols just send data packets at a fixed rate, ZGT constantly probes the RTT (Round Trip Time) to identify the ideal send rate and create a uniform packet distribution over time.
- Interval acknowledgements: Most transfer protocols send acknowledgements every time a packet is received. ZGT saves time and increases transfer speed by sending acknowledgements periodically for groups of packets.
- Delayed retransmission: ZGT retransmits lost packets once per RTT to decrease unnecessary retransmissions.
ZGT can also dynamically reroute around network outages and hardware failures to ensure your files always arrive at their destination. And, because it can fully utilize the bandwidth of any network and optimize transfer, ZGT enables you to synchronize files from areas with very little network coverage (such as in remote locations, out at sea, etc.).
Case Study: Shifo
Shifo provides health data solutions to over 3,000 remote communities worldwide. Shifo uses Resilio Connect to synchronize their servers across remote communities with poor network connections.
“You can’t imagine how difficult it was for us [prior to Resilio]: Delays, nothing being synchronized; now I don’t hear any of those problems. Now we can focus on solving healthcare problems [instead of moving files].”
End-to-End Security
One of the major downsides of using Rsync is that it doesn’t encrypt or protect your files, leaving them exposed to interception and malicious activity.
But Resilio Connect uses state-of-the-art security features that were reviewed by 3rd-party security experts to protect your data, such as:
- End-to-end encryption: Resilio uses AES 256 to encrypt data at rest and in transit.
- Mutual authentication: Resilio only delivers files to designated, approved endpoints.
- Cryptographic file integrity validation: Resilio uses integrity validation to ensure that your files arrive at their destination intact and uncorrupted.
- Permission controls: Resilio enables you to create role-based permissions, multi-level admins, and control access to settings, agents, and jobs.
Synchronize Files with Resilio Connect
Resilio Connect is a P2P-based file sync solution that provides faster, more reliable synchronization than traditional solutions like Rsync. It’s easier to manage and troubleshoot, syncs in any direction, optimizes transfer over high-latency WAN networks, and keeps your data secure.
Schedule a demo to learn more about using Resilio Connect for file synchronization.
