A DFSR Alternative: Fast & Resilient P2P File Replication with Connect

What is DSFR?

DFSR (sometimes written DFS-R), or distributed files system replication, is a feature of Windows Server for replicating files across several servers. Microsoft introduced this feature in Windows NT4.0 as an add-on, and it became a standard feature in Windows 2000 server and on.

Basically, it was an attempt by a Microsoft to solve the complex problem of distributing files in real time so teams across several offices could collaborate effectively. Nowadays, the challenge of real time file replication and distribution is much easier to solve, with the advent of the public cloud. Instead of sharing files between offices, people work directly with the cloud. However, the cloud doesn’t solve all the issues in collaboration and you might still need to build a system where several servers need to be synchronized across different offices. Let’s see if the DFSR as a technology for 2000 is an adequate response to these needs, or if a DFSR replacement should be considered.

When to use a Distributed File Systems Replication Tool vs. When to Find an Alternative

First, let’s see why you might need to synchronize servers directly. One of the reasons might be that you have big files and would require significant time to upload to the cloud. The second case might be that due to a variety of reasons you can’t upload files to the cloud. It might be security concerns, company or government policy etc. Third and most popular reasons that we see is that moving files or synchronizing servers is just a part of your workflow. You are building a business process that is more complex than just uploading and downloading files to the cloud.

The DFSR is a standard client-server or 1:1 file synchronization approach that was popular 10-15 years ago. All of the other commercial solutions use the same model for synchronization – copy the file from point A to point B. This approach obviously has limitations, such as the availability of A – if server A is not available, then server B and C will not get the data. Resource shortage of A, if there is any resource shortage on server A (networking, memory, CPU) the load cannot be moved to server B, even if it has all the data available.

DFSR Issues & Size Limits

There are a number of other limitations to DSFR. If you need to extend your file synchronization beyond Windows servers, DSFR is not the right tool for the job. You also can’t give different roles to different servers, e.g. if you need some servers to just read information, without being able to change it.

A DFSR-based solution may be effective for some businesses, but let’s take a look into typical DFSR problems that can arise:

DFSR Over Long WAN, Connections

When you need to send data to remote locations over mobile or satellite connections that have a very long retransmission time and potential for packet losses, DSFR can run into problems.  It’s based on the TCP/IP protocol, which treats every packet loss as a network congestion and backs off speed in order to reduce the load of the connection. This approach helps applications that are TCP/IP based share networks and collectively agree on the maximum speed they can use for data transfer. In case of wide-area networks (WAN), the packet loss might represent some failure on intermediate device and the channel is often not congested. Therefore, the logic of reducing speed in case of packet loss is not appropriate for WAN connectivity.

TCP/IP Data Delivery Faults

Another problem in the WAN and use of DSFR, relates to the way TCP/IP guarantees data delivery. First, it needs a recipient to acknowledge that packet arrived at the destination. Once a recipient gets a packet, it sends a confirmation packet to the sender acknowledging that it has received the specific packet. The time during which the packet travels from sender to receiver is called retransmission time (RTT). In the local network (LAN) it is below 0.01ms, but in WAN networks it can be as high as 800ms or more. Therefore, a receiver can wait up to a second or even more before it’s able to send another packet. These TCP/IP deficiencies are inherited by DSFR. Overcoming these bottlenecks requires the consideration of a DFSR alternative.

Inability to Deliver Files to Multiple Locations

DFSR is limited in the ability to synchronize or deliver files to more than one destination. It is extremely rare that an organization needs to send or synchronize files just to one location or server. Usually, it is more than one destination server, in more than one location. It such a case, a common approach is to execute jobs one by one. DFSR sends files to one location, then to another.

Poorly Optimized for Real Time File Replication

If you’re working with tens of millions of files and real-time change detection, you’ll find DFSR is not optimized for a large number of files. DFSR speed is usually very slow when you need to synchronize folders with few millions of files since it takes forever to scan through this folder, find changes and transfer them. A better approach would be to use real-time file system monitoring to pick up changes on the fly without the need to browse through the whole directory tree like Resilio Connect.

Lack of Smart Data Routing

DFSR needs static ip:ports to establish a connection to different machines. If a machine has  a new IP:port or is not available, DSFR stops operation and needs a human to re-configure.

Inability To Incorporate Scripting Into DFSR Workflows

Simply put, there is no way to have scripting around DFSR. If you need to build workflows beyond a simple “do something after the file arrives at destination”, there is no way to do so with DFSR.

In contrast to DSFR, alternative replication & file distribution solutions Resilio Connect were developed to be faster under any network conditions and data volumes. This distributed file system replication service is specifically optimized to replicate large volumes of data, and works well with many small files, without any limits.  Besides, with a unique network protocol, it’s sensitive to bandwidth changes and is smart enough to avoid network congestions or use full bandwidth when possible. Resilio Connect lets you take control over the file replication process, see its progress and evaluate the results.

Resilio Connect data replication and synchronization is not limited to the Windows platform only, and can be easily configured as a cross-platform solution (Linux, OS X, iOS, and Android).

Solutions to Common DFSR Problems

How to make DFSR faster

Because DFSR’s basic set of technologies is limited to delta encoding and compression, it means it’s impossible to make DFSR faster. You would need to replace DFSR to get faster performance. If you are looking for a faster alternative to DFSR, consider Resilio Connect, which adds peer-to-peer data transfer, WAN line optimization, smart routing and real-time file system monitoring in order to speed up syncing for today’s enterprise.

DFSR and Transferring Large Files

Because Windows DFSR does not have an optimized way of calculating the checksum of a file, this leads to an extremely long time to calculate file differences, making large file transfers take that much longer. Unlike DFSR speed issues, our DFSR alternative, Resilio Connect, optimizes the checksum calculations quickly, allowing for files of all sizes to sync very fast.

DFSR & VPN

One of the biggest drawbacks of Windows DFSR is that it lacks embedded traffic encryption, which makes it impossible to use DFSR without additional encryption. DFSR currently requires users to install and configure an additional encryption channel – a VPN. Any good DFSR alternative should include encryption within the product so that no additional products or set up is required. Our DFSR replacement, Resilio Connect, uses AES128 in CTR mode to encrypt all traffic that is sent between clients. This also includes both data and all of the control traffic.

DFSR & Static IP Addresses

Another limitation of DFSR is that it requires a static IP and port address for a destination. This makes DFSR impossible to user in a dynamic network environment. With rsync, as soon as an IP address of a server is changed, then rsync will fail to operate. A good DFSR replacement, such as Resilio Connect, will use a dynamic routing approach that specifies when machine A and B need to exchange data. This means there’s no human intervention necessary at all — both machines will use a tracker or multicast to discover the required IP:Port address on the fly.

DFSR & WAN Network Connection

Because DFSR lacks WAN network support, it makes it impossible to use DFSR over long connections for offices located 3,000 miles or more apart. The long distance of the office creates a long travel time for the TCP packet, which increases the chances of packet loss due to equipment failure or congestion. As rsync is built on the TCP protocol, the sync speed will be slow when using these types of networks.

To prevent packet loss over large distances, Resilio Connect uses WAN network support, allowing you to utilize 100% of the available bandwidth in your network totally independent of distance, latency, or loss. To achieve this, Resilio Connect uses UDP based protocol uTP2, which uses bulk packet transfer with selective acknowledgement of lost packets in transferring large files across vast distances.

DFSR & Transferring Files to Several Destinations

Any good alternative to DFSR will use a peer-to-peer approach, which leverages networking between all available offices and significantly speeds up data transfer times. Using this approach splits each file into blocks, then sends these blocks independently. This means that each recipient can send the block to another recipient once it has been received. Using a peer-to-peer approach dramatically speeds up real time syncing operations, since we are not only transferring concurrently to several machines, but also using other network channels to help offload loads from a sender network channel. Resilio Connect uses this approach to deliver large files in real time quickly.

DFSR & NAT

NAT, which is usually a firewall that hides server internal addresses and provides a connection with external IP addresses, makes it impossible for DFSR to connect to a server. In order to do this, you will need to open ports for each incoming connection on the device so that rsync can establish a connection. Resilio Connect, our DFSR replacement, uses NAT traversal techniques that allow a direct connection between computers without the need to open ports.

Summary of Resilio Connect vs. DFSR

Here is a handy summary table of the features needed in a synchronization solution today, and how Resilio Connect stacks up against DSFR.

  DFSR Resilio Connect
Cross platform solution No Yes
Max number of servers 256 for each replication group No limits
Max number of files 70 million (2012R2)
11 million (Server 2008)
No limits
ACL replication Yes Yes
Max file size 250 Gb (2012R2)
64 Gb (Server 2008)
No limits
Data deduplication Yes Yes
WAN acceleration No Yes
P2P replication No Yes
Time to replicate 3.72 Gb single file 4:00 min 1:39 min
Time to replicate 100k files of 4kb each 46 min 17 min
Delta encoding Yes Yes
Compression Yes Yes
In-transit encryption No Yes
WAN optimization No Yes
NAT traversal No Yes
10M+ files support No Yes
Very large folders No Yes
Real-time response to changes Yes Yes

How much faster can you replicate with Resilio Connect? Schedule a demo to find out.