Two Ways to Check Rsync Progress While Syncing Files

Rsync is a free Linux-based file syncing tool. However, like many other command line replication solutions, it suffers from several shortcomings — such as complications when checking your Rsync progress — that may cause organizations to seek an alternative solution for Linux file sync. 

In this article, we’ll discuss several Rsync commands you can use to check your Rsync transfer status. We’ll also discuss why our file replication solution, Resilio Connect, is a superior solution for syncing files across Linux servers.

Resilio Connect is a P2P (peer-to-peer) server replication solution that uses a UDP-based WAN optimization transfer protocol (known as Zero Gravity Transport™)  to optimize file transfer across high-latency and lossy networks. It can sync large files and large numbers of files (tested at 250+ million files in a single job) faster than any other solution available (10+ Gbps per server). Resilio is used by companies in gaming (Blizzard, Wargaming.com), tech (Match.com, Kaspersky), media (Mixhits Radio, Turner Sports), and more.

Leading organizations in tech (Match.com, Kaspersky), gaming (Larian Studios, Blizzard), media (Turner Sports, Mixhits Radio) and more use Resilio Connect to synchronize data.

To learn more about syncing with Resilio Connect, schedule a demo.

How to Check Rsync Progress

There are two Rsync options to check the state of replication in your environment:

• The “rsync — progress” command
• The PV command

We’ll break down both progress options, how they work, and how you can alter them to show progress in optimal formats based on your needs.

“Rsync– progress” Command

To display the progress of file transfer between your source directory and destination directory, use the following Linux command syntax:

rsync -- progress [source] [destination]

This commands Rsync to display the following information about each individual file:

• File size (in bytes)
• Current transfer progress (as a percentage)
• Transfer speed (in bytes)
• Time until transfer is complete

The readout will look like this:

File 1
16354838  100%    82.82MB/s    0:00:00 (xfer#1, to-check=1/2)

File 2
15243526  100%    74.62MB/s    0:00:00 (xfer#2, to-check=1/2)

You can use two additional commands to alter how Rsync presents this information:

Human-Readable Numbers

You can use the “-h” command to present transfer status in human-readable numbers.

rsync -- progress -h [source] [destination]

Basically, this changes file size into megabytes rather than bytes — which is a more user-friendly format.

File 1
12.3M  100%    86.15MB/s    0:00:00 (xfer#1, to-check=1/2)

File 2
1.62G  100%    62.13MB/s    0:00:00 (xfer#2, to-check=0/2)

Display File Transfer Stats

The “– progress” command provides information about the overall progress of file transfer. For a more detailed breakdown of your transfer status, you can use the following command:

rsync -- stats -- progress -h [source] [destination]

This will return the following information:

File 1
12.3M  100%    86.15MB/s    0:00:00 (xfer#1, to-check=1/2)

File 2
1.62G  100%    62.13MB/s    0:00:00 (xfer#2, to-check=0/2)

Number of files: 2
Number of files transferred: 2 
Total file size: 1.92G bytes
Total transferred file size: 1.92G bytes 
Literal data: 1.92G bytes
Matched data: 0 bytes
File list size: 67
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 1.92G
Total bytes received: 67

PV Command

You can also install the “pv command” to get a readout of Rsync transfer progress indication. The “pv command” write-out reads as follows:

$ sudo apt install pv

Another format for installing the pv command reads as follows:

~$ rsync -vrltD --stats --h [file dir 1]/ [file dir 2]/ | pv -lep -s 5

The pv command provides you with an incremental list of files that includes information such as:

• The current throughput rate.
• Elapsed time.
• The total data that’s been transferred.
• The percentage of file transfer that’s been completed (along with a progress bar).
• The remaining time until completion.

6 Reasons Why Resilio Connect Is a Superior File Sync Solution

While Rsync is a suitable replication solution for most small file sync jobs, many users complain that Rsync breaks down and has trouble syncing large numbers of files and very large files. After a certain point, troubleshooting methods such as making sure you have the latest version of Rsync, X, and Z, will not solve your problems.

Resilio Connect is a server replication solution that uses a P2P file transfer architecture and WAN optimization technology to provide faster, more reliable file sync across any network — particularly for large applications that require synchronizing large files and large numbers of files in real-time.

Resilio is superior to Rsync in several key ways, such as:

1. Easier deployment, insight into, and management of replication
2. Easier automation of sync jobs
3. Better security features
4. Faster file synchronization
5. Superior WAN (wide area network) transfer
6. More reliable and scalable file transfer

1. Easy Deployment, Insight, and Management

Resilio is an agent-based software that can be deployed easily on your existing infrastructure and begin replicating in as little as 2 hours. It’s also more flexible than Rsync in that it can be deployed:

• In the cloud, on premises, or in a hybrid cloud model.
• Supports a variety of servers, cloud storage platforms, virtual cloud servers, desktops, and mobile devices.
• Can be configured cross-platform on Linux, Windows, Ubuntu, Unix, MacOS, iOS, Android, and more servers.

Resilio’s Management Console makes it easy to monitor and control the state of file replication in your environment. The dashboard provides detailed transfer logs and real-time notifications that provide easy insight into the state of replication. It’s easy to track replication progress and diagnose and fix replication issues.

And you can use the Management Console to configure user permissions, set up Webhooks, and adjust replication parameters (such as disk I/O threads, buffer size, bandwidth usage, and more).

2. Easy Automation of Sync Jobs

Automating sync jobs with Rsync requires learning, implementing, and adjusting complex Rsync commands and setups.

But Resilio Connect gives you all of the automation capabilities with none of the complexity. You can easily automate replication jobs using the GUI, command-line, or Resilio’s REST API.

3. Superior Security Features

One of the big problems with Rsync is that it doesn’t encrypt data, leaving your files exposed to malicious activity and interception during file transfer.

But Resilio Connect provides state-of-the-art security features that were reviewed by security experts, such as:

• One-time session encryption keys that protect data.
• In-transit and at-rest encryption using AES 256.
• Mutual authentication that ensures files are delivered only to approved devices.
• Cryptographic integrity validation that ensures data is delivered to endpoints uncorrupted.

4. Faster Synchronization with a P2P Transfer Architecture

Resilio utilizes several features that enable it to synchronize files faster, such as real-time file change detection, P2P transfer, and file chunking.

To synchronize files, Rsync uses a point-to-point topology — either client-server or “follow-the sun.” Both of these topologies can only transfer files from one server to another, making them slow and less reliable.

But Resilio uses P2P transfer and a process known as file chunking — where files are split into several blocks that can transfer independently from each other — to deliver fast, simultaneous synchronization across your entire replication environment.

For example, let’s assume you have seven devices in your replication environment, and Device 1 wants to sync files with the other six servers. It will split the file into several blocks and begin syncing with Device 2. As soon as Device 2 receives the first file block, it can begin sharing it with any other server before it receives the full file. With file chunking and P2P transfer, Resilio can transfer files 3-10x faster than point to point solutions.

While tools like Rsync use incremental syncs — i.e., syncing files every 1, 3, 5 minutes etc. — Resilio can actually perform true real-time synchronization. Resilio uses optimized checksum calculations (identification markers for each file that change whenever a change is made to the file) and real-time notifications from the host OS to detect and sync file changes as soon as they occur.

5. Superior WAN Transfer

Another problem with Rsync is that it’s not optimized for transfer over long-distance, high-latency networks. Resilio uses proprietary WAN acceleration technology known as Zero Gravity Transport™.

ZGT can provide the fastest transfer speeds over any WAN connection using:

• A congestion control algorithm that calculates the ideal send rate.
• Interval acknowledgments for groups of packets (rather than each individual packet) that also provide information about packet loss.
• Retransmission of lost packets once per RTT (Round Trip Time).

6. More Reliable and Scalable File Transfer

As your replication environment grows, Rsync begins to experience more and more issues when syncing large files, large numbers of files, or syncing to many endpoints. The more data you have to sync, the longer a sync will take. And there are multiple points of failure — i.e., replication will be delayed or stopped by any network or hardware failure between any of your endpoints.

But Resilio’s P2P architecture means: 

• Organic scalability: As your replication environment grows, performance improves. Resilio syncs 50% faster than point-to-point solutions like Rsync in a 1:2 scenario, and 500% faster in a 1:10 scenario.
• No single point of failure: Since every device can contribute to a sync, there are no single points of failure. Any device can receive the necessary services or files from any other device, and Resilio can dynamically route around outages. Replication is not dependent on any device, so it will never fail.

Resilio can also sync in any direction, including one-to-one, one-to-many, many-to-one, or N-way sync.

To learn more about syncing with Resilio Connect, schedule a demo.