UDP (User Datagram Protocol) is a connectionless, popular transfer protocol used by some applications designed for moving large datasets over the internet protocol (IP).
For many years, TCP (Transmission Control Protocol) was the most commonly used network protocol. However, the wide availability of broadband-speed internet connections made UDP file transfer more popular — particularly for transferring large files over long-distance WANs and low-quality networks. Even Google recently announced that 50% of Chrome traffic is run by their UDP protocol, Google Quic.
Maybe you’ve heard of UDP in association with a variety of high-speed transfer products such as UDP blasters, Aspera, Signiant, and FileCatalyst, among others. Resilio also uses UDP as part of our Zero Gravity Transport (ZGT) WAN Optimization solution. We’ll cover that in more detail later in the piece.
For now, it’s worth noting that while UDP offers advantages to moving data fast and in bulk across the Internet, it is not a solution; it’s a protocol.
UDP doesn’t require an ordered delivery of packets and supports a continuous packet stream. This makes it much more efficient and better at transferring over low-quality networks compared to TCP. However, UDP also carries reliability risks as there’s no guarantee that all of your packets will arrive at their destination.
That’s why, if you want to take advantage of UDP’s speed and efficiency for transferring your large files, you need to use a UDP transfer software that includes critical features that overcome these issues.
Want to see how Resilio’s proprietary WAN Optimization solution, ZGT, uses UDPto transfer your large files over any network reliably, efficiently, and securely? Schedule a demo with our team to see Resilio ZGT in action!
In this article, we’ll discuss how UDP file transfer works, as well as its features, benefits, and downsides.
We’ll also discuss how our real-time file synchronization software system, Resilio Connect, provides fast and reliable large file transfer and sync over any network using a proprietary file UDP-based protocol called ZGT that overcomes the issues of alternative UDP solutions.
Resilio is the best UDP file transfer solution because it:
- Fully utilizes any network: Resilio’s implementation, Zero Gravity Transport™, uses a UDP-based protocol that automatically analyzes the conditions of any network (VSAT, cell, Wi-Fi, and any IP connection) and dynamically adjusts to those conditions to maximize bandwidth utilization. It uses features that overcome the limitations of traditional UDP protocols, such as interval acknowledgments, delayed retransmission, and cryptographic data integrity validation.
- Uses P2P (peer-to-peer) transfers: Competing UDP transfer solutions use slow, unreliable point-to-point transfer architectures. Resilio uses a P2P transfer architecture that allows it to sync and transfer files 3–10x faster than competing solutions, scale organically, transfer in any direction, and eliminate single points of failure.
- Offers bulletproof reliability: Resilio includes features that ensure your data always reaches its destination, such as dynamic rerouting, automatic retries, and checksum restarts.
- Easily integrates into your infrastructure and workflow: Resilio is a vendor-agnostic solution that works with just about any device, cloud storage provider, and operating system. It also easily integrates with popular solutions your team is already using, facilitating seamless integration into your current workflow.
- Provides centralized management and granular control: Through Resilio’s centralized Management Console, you can manage every aspect of replication across your entire environment — such as creating bandwidth profiles, controlling replication jobs and parameters, and programming how Resilio troubleshoots issues.
- Includes state-of-the-art security features: Resilio includes built-in security features that protect your data at rest and in transit, such as AES 256-bit encryption, mutual authentication, and proxy server capabilities.
Organizations in media, gaming, engineering, retail, tech, and more use Resilio Connect to sync and transfer large files over any network (WANs, LANs, edge, etc.). To learn how Resilio can provide fast, reliable sync for your organization, schedule a demo with our team.
UDP File Transfer Use Cases
To understand why some organizations turn to UDP file transfer, we should first review some use cases where large file transfer over WANs and unreliable networks is necessary.
Many of our clients require fast-paced work on large files across geographically distributed teams and need a solution that can quickly and reliably sync large files and file updates across multiple locations. These include organizations in:
- Media and entertainment: Many media companies consist of teams in different locations that need to collaborate on large media files fast enough to meet production deadlines. File sizes can be massive. For example, our client Blockhead uses Resilio Connect to sync high-resolution video files that can reach 20 TB in size across their two production sites.
- Gaming: Wargaming uses Resilio Connect to sync video game builds of 50+ GB across more than 20 production offices.
- Engineering: Maffeis Engineering uses Resilio Connect to sync large design files (reaching 400+ GB) across four sites for real-time collaboration among their team of engineers, architects, and designers. If files aren’t synchronized fast enough, it can cause them to miss deadlines and have catastrophic effects on their engineering projects.
- Software companies: VoiceBase is a speech-to-text software. VoiceBase uses Resilio Connect to distribute large speech models of 50+ GB across more than 400 production servers.
- Edge transfer: Some organizations require reliable data transfer over unreliable edge networks. Northern Marine Group uses Resilio Connect to sync software updates and ship data back and forth between their fleet of ships and headquarters on shore.
For these use cases, TCP-based protocols, such as the file transfer protocol (FTP) wouldn’t be a good fit because they:
- Require an ordered delivery of packets — i.e., the receiving device must acknowledge each received packet before the sending device can send another packet.
- Assumes that large amounts of packet loss (which can occur often over WANs) is a network congestion issue. In response, TCP unnecessarily reduces transfer speeds.
For these reasons, the slow transfer rate and network congestion created by TCP protocols make speedy replication of large files across WANs (and unreliable networks) impossible. This prevents geographically distributed teams from collaborating quickly and efficiently in the scenarios outlined above, as well as in many other situations.
TCP vs UDP vs Resilio ZGT: Features, Benefits, Downsides
In this section, we’ll cover the differences between TCP transfer, traditional UDP transfer, and Resilio’s proprietary UDP-based transfer protocol known as Zero Gravity Transport™ (ZGT) — which can transfer files 100x faster than traditional TCP-based protocols while being more reliable and efficient than most UDP-based protocols.
Try our transfer speed calculator to see how much time we can save you.
Acknowledgments and Reliability
An acknowledgment occurs when a receiving device signifies to a sending device that it received a data packet. The way a transfer protocol handles acknowledgements can impact the speed and reliability of transfer.
Transmission Control Protocol
As stated earlier, TCP protocols require the receiving device to acknowledge each packet before another can be sent. They also track packets to ensure that all packets were delivered to the receiver.
While this enhances the reliability of transfer (by ensuring all packets arrive at the receiving device), it also slows transfer speeds by increasing traffic over the network and making the sending device wait before transmitting more packets.
Traditional User Datagram Protocol
Traditional UDP protocols don’t require any acknowledgements for received data packets. This reduces traffic over the network and allows the sender to continue sending data packets unrestricted.
But it also makes data transfer less reliable, as there’s no way to confirm that an individual packet was received or that the entire file was delivered.
Resilio Zero Gravity Transport
Resilio’s ZGT transfer protocol uses interval acknowledgments to confirm packet receipt. Rather than requiring an acknowledgment after each received packet, ZGT sends acknowledgments in groups once per RTT. This combines the best of both worlds by minimizing traffic over the network, confirming packet receipt, and enabling a continuous data stream.
In addition, Resilio Connect uses cryptographic data integrity validation to ensure that files reach their destination intact and uncorrupted. The data hashes and metadata are sent before the files are transferred, so the receiving device can confirm the integrity of each file. If any file is incomplete or corrupted, it’s immediately discarded and scheduled for retransmission.
Lost Packet Retransmission
As stated earlier, packet loss (which happens when data packets don’t reach their destination) is a common occurrence when transferring over long-distance WANs and unreliable networks. The way a transfer protocol responds to packet loss can impact transfer speed and reliability.
Transmission Control Protocol
TCP protocols suffer from two issues with regard to packet loss:
First, TCP protocols require that lost packets be retransmitted after each acknowledgement. This increases traffic over the network and slows the rate of data transfer.
The second issue is that TCP protocols treat large amounts of packet loss as a network congestion issue and reduce transfer speed in response. Since packet loss over WANs isn’t indicative of network congestion, this response unnecessarily delays data transfer and impedes workflows.
Traditional User Datagram Protocol
UDP places no requirements on lost packet retransmission. While this can reduce traffic over the network and allow for faster transfer speeds, it also makes it unreliable (as there’s no way to ensure all UDP packets are delivered to their destination).
Resilio Zero Gravity Transport
ZGT uses a process known as delayed retransmission. Instead of sending lost packets after each acknowledgment, it retransmits lost packets in groups once per RTT.
Again, this provides the best of both worlds. It reduces unnecessary retransmissions and minimizes traffic over the network. And data is delivered reliably while ensuring bandwidth is utilized efficiently.
Data Flow and Congestion
The rate at which data can be transmitted from a sender to a receiver is controlled by:
- How much data the receiver can handle at a given time.
- How much data the network can handle at a given time.
Data flow control refers to controlling the amount of data sent to the receiver. Essentially, the receiver tells the sender the maximum speed at which to send data in order to avoid packet loss.
Congestion control is a mechanism for controlling the amount of data sent over a network. Rather than being dictated by the receiver, data transmission is controlled by how much traffic the network can effectively handle.
Transmission Control Protocol
TCP includes options for flow control, such as sliding windows, congestion avoidance algorithms, and more. While these mechanisms can help reduce packet loss, they prevent TCP from fully utilizing the bandwidth of a given network. This makes TCP unsuitable for:
- Transfers across high-speed WANs, as the protocols won’t fully-utilize the bandwidth of the network.
- Transfers over intermittent, unreliable networks, as TCP can’t adjust to the changes in network capacity over time.
Traditional User Datagram Protocol
Traditional UDP doesn’t place any limits on flow control, allowing for faster transmission of data. But unrestricted data transfer can lead to network congestion and packet loss.
Resilio Zero Gravity Transport
ZGT uses a congestion control algorithm to maximize bandwidth utilization of any network at all times.
The algorithm constantly probes the Round Trip Time of the network and analyzes the underlying network conditions (such as packet loss, latency, and throughput over time). It dynamically adjusts to these conditions in order to fully utilize the network, increase transfer speed, and maintain a uniform packet distribution over time.
This not only makes ZGT ideal for transfer over WANs, it also allows Resilio to fully-utilize any type of network connection — such as VSATs, cell (3G, 4G, and 5G), Wi-Fi, and any IP connection. So Resilio can be used for transfers over intermittent, unreliable edge networks, such as maritime file transfer.
Replication Architecture: Point-to-Point vs P2P
Whether they utilize TCP or UDP, most file transfer solutions use point-to-point replication architectures that inherently hinder their ability to transfer large files quickly and reliably.
Point-to-point replication occurs in one of two topologies:
- Hub and spoke: This file sharing topology consists of a hub server and several remote servers. The remote servers can’t share files directly with each other. Instead, they must first send the files to the hub server. Then the hub server shares the files with each remote server one by one.
- Follow-the-sun: In this topology, files are synchronized with each server sequentially. So if there are five servers: Server 1 syncs with Server 2; then Server 2 syncs with Server 3; then Server 3 syncs with Server 4; and so forth.
Both of these replication topologies are slow and unreliable, regardless of the transfer protocol being used.
The fact that replication only occurs between two servers at a time (i.e., one server to another) causes full synchronization of your entire environment to take a long time. This is especially true when syncing large amounts of data (i.e., large files or large numbers of files) or syncing across many endpoints.
Plus, synchronization is limited by your slowest endpoint. If any server in your environment is on a slow network, then it will delay synchronization for every other server behind it.
And both replication topologies introduce single points of failure. If any server in your environment goes down, it can delay sync for every other server. And if your hub server goes down in a hub and spoke topology, replication fails entirely.
Resilio Connect is one of the only file synchronization solutions that uses a peer-to-peer (P2P) replication architecture.
P2P replication turns your environment into a mesh network where every endpoint can work together to sync files. It accomplishes this because:
- Every server in a P2P environment can share files directly with every other server.
- Every server in a P2P environment can share files simultaneously.
- Resilio uses a process known as file chunking to split files up into multiple chunks that can transfer independently of each other.
Resilio’s P2P replication architecture enables it to overcome the limitations of traditional point-to-point solutions and provide file transfer/synchronization that is:
Blazing Fast
The combination of file chunking and P2P transfer yields blazing-fast sync speeds, as every endpoint can sync files across your environment concurrently.
Imagine you wanted to sync a 100 GB file across five servers. Resilio can split that large file into five chunks (20 GB each). Server 1 can share the first chunk with Server 2. As soon as it receives the first chunk, Server 2 can share it with Server 3, even while it waits to receive the remaining file chunks. The same is true for every other server: they can immediately share file chunks while they wait to receive the rest of the file.
Resilio’s P2P replication methodology enables it to sync files 3–10x faster than most point-to-point solutions.
Organically Scalable
As alluded to earlier, point-to-point replication solutions don’t scale well. Every new endpoint you add to your environment causes full synchronization to take longer. If you need to sync 1 TB of data across 5 endpoints, that large data must sync with each endpoint one by one.
But since every endpoint in a P2P environment can work together to share files, adding more endpoints only increases the available resources (CPU, bandwidth, etc.) and replication speed — i.e., more demand inherently creates more supply.
This makes Resilio organically scalable, allowing it to quickly and reliably synchronize:
- Files of any size: Our engineers successfully transferred a 1 TB dataset across Azure regions in 90 seconds.
- Files of any number: Our engineers successfully synchronized 450+ million files in a single job.
- Any number of endpoints: Resilio can sync 200 endpoints in roughly the same time it takes most point-to-point solutions to sync just two.
And with Resilio, you can take transfer speed across large environments to another level using a process known as horizontal scale-out replication.
With scale-out replication, you can cluster endpoints together in order to pool their resources and linearly increase transfer speeds.
For example, since each Resilio agent can reach speeds of 10 Gbps, our engineers were able to cluster 10 agents together and achieve speeds of 100+ Gbps. You can cluster even more agents together to achieve even greater speeds, as there’s no limit to how fast this can go.
Multidirectional Sync and Real-Time Replication
While not directly related to Resilio’s P2P architecture, Resilio has the ability to sync files in real-time. It uses notification events from the host OS and optimized checksum calculations (i.e., identification markers assigned to each file that change whenever a change is made to a file) to detect file changes as soon as they occur and immediately sync the changed portions of the file across your environment.
Resilio’s P2P architecture also enables it to sync files in any direction, such as:
- One-way: Sync in one direction to backup files to a backup site or migrate to the cloud.
- Two-way: Keep two servers in sync for multi-site collaboration.
- One-to-many: Distribute software updates across any number of servers. Or send data out to a fleet of vehicles in the field.
- Many-to-one: Consolidate data from many servers into one. Or collect data from a fleet of vehicles in the field.
- N-way sync: Keep any number of servers synchronized concurrently.
N-way sync is a particularly powerful feature that’s key to Resilio’s ability to provide fast, reliable sync speeds for:
1. Multi-site collaboration
The combination of real-time sync and multidirectional sync is great for remote collaboration. If any employee at any location makes a change to a file, that file change is immediately detected and synchronized across every other server in your environment.
2. Disaster recovery
Because it replicates data in real-time, Resilio can provide sub-five second RPOs (Recovery Point Objectives). And Resilio’s P2P architecture with N-way sync effectively turns every endpoint in your environment into a backup server. This enables you to achieve Active-Active High Availability. And in the event of a disaster, every endpoint can work together to bring your system back online for RTOs (Recovery Time Objective) within minutes of an outage.
3. Synchronization at the far edge
When operating at the far edge of networks where internet connections are unreliable (and sometimes unavailable), you can use Resilio to transfer from agent to agent. For example, the shipping agency Ross Maritime has vessels that are often out of range of high-bandwidth connections, but are occasionally in range of other vessels. But Ross Maritime uses Resilio’s P2P architecture to reliably deliver data to those ships in a “store and forward” fashion — i.e., sending data from ship to ship until it reaches its destination.
No Single Point of Failure
Since every endpoint in a P2P environment can share files directly with each other, there’s no single point of failure.
If any device or network goes down, the required files or services can be retrieved from any other device in your environment.
4 More Ways Resilio Provides Superior File Transfer and Sync
Resilio’s ZGT transfer protocol overcomes the limitations of UDP better than any other UDP-based protocol. But it also provides other capabilities that make it a superior file transfer and sync solution, such as:
- Bulletproof reliability features
- Vendor and workflow agnosticism
- Centralized management and granular control
- Built-in security features
1. Bulletproof Reliability Features
Reliable synchronization of mission-critical files is paramount for fast-paced, data-intensive workflows. As we’ve discussed, other file sync and transfer solutions are unreliable because:
- TCP protocols transfer slowly, particularly over WANs and unreliable networks. And they respond poorly to packet loss.
- Traditional UDP protocols don’t contain features that prevent network congestion and ensure packet delivery.
- Most file sync and transfer solutions use unreliable point-to-point transfer protocols.
In addition to the reliability features of Resilio listed above, Resilio includes other features that enhance reliability, such as:
- Automatic Retries: If a file transfer fails, Resilio automatically retries the transfer until it’s complete.
- Checkpoint Restarts: If a file transfer is interrupted, Resilio performs a checkpoint restart to resume the transfer at the point of interruption rather than retransferring redundant data.
- Dynamic Rerouting: If any device or network in your environment goes down, Resilio can dynamically reroute around the outage and find the optimal path for delivering your data to its destination.
2. Vendor and Workflow-Agnostic Deployment
Resilio Connect is built on open standards and universal protocols. This means that:
- Your files are never stored in any proprietary formats, and you always retain full control over your data.
- There’s no need to invest in any new or proprietary hardware or software.
Resilio is a highly-flexible, vendor-agnostic, agent-based solution that works by installing agents on each endpoint you want to replicate to and from. Resilio supports just about any:
1. Device: You can install Resilio agents on desktops, laptops, file servers, NAS/DAS/SAN devices, mobile devices (Resilio offers iOS and Android apps), IoT devices, and virtual machines (such as VMware, Citrix, and Microsoft Hyper-V).
2. Cloud storage platform: Resilio supports all major cloud object storage platforms, such as AWS, Google Cloud, and Azure Blobs. It also supports just about any other S3-compatible cloud object storage, such as Backblaze, Weka IO, MinIO, Wasabi, and more.
3. Operating system: Resilio supports Windows, MacOs, Linux, Unix, Ubuntu, FreeBSD, OpenBSD, and more.
4. Automation requirements: Using Resilio’s powerful REST API, you can script any type of function and automation your job requires. Resilio scripts offer three types of triggers: before a job starts, after a job completes, and after all jobs complete. For example, in order to deliver security updates to their vessels at sea over intermittent satellite connections, a marine construction company used Resilio to deliver the updates to each ship’s main server, and used scripts to have those servers immediately distribute the updates to each workstation on the ship.
5. Workflow integration: You can use Resilio’s REST API to integrate it with the tools your team is already using, such as development tools (e.g., TeamCity, Jenkins), and management tools (e.g., Microsoft SCOM, Splunk). For example, the documentary production company Delirio Films integrated Resilio with their media tools and creative software (Adobe Premiere, DaVinci Resolve, and Avid Media Composer), various storage systems (Facilis Terrablock and OWC Thunderbay), and the LucidLink file streaming solution.
This level of flexibility means that you can deploy Resilio on your existing IT infrastructure — at minimal cost and with minimal operational interruption — and begin replicating in as little as two hours.
3. Centralized Management and Granular Control
Unlike many traditional and open-source UDP-based file transfer solutions, Resilio Connect is easy to manage and gives you granular control over how replication occurs in your environment.
While you can program Resilio via the command-line interface, Resilio’s Management Console is a more user-friendly graphic interface that can be accessed from any web browser. Through the Management Console, you can control every aspect of replication across your environment, such as:
- Creating, managing, and monitoring replication jobs.
- Reviewing a history of all executed jobs.
- Adjusting bandwidth at each endpoint and creating profiles that control how much bandwidth is allotted to each endpoint at certain times of the day and on certain days of the week (Skywalker Sound used Resilio’s bandwidth controls in their remote work deployment to account for the differences in available bandwidth/upload speeds at each employee’s home network in order to avoid files falling out of sync).
- Adjusting replication parameters, such as data hashing, file priorities, and disk I/O.
- Creating job policies, such as choosing to sync metadata or purging files from a cache after a certain period of time.
- Programming how Resilio automatically troubleshoots replication issues, such as file conflict resolution.
- Obtaining real-time reports and data on active jobs.
4. Built-In, State-of-the-Art Security
Data security is critical in all file transfer scenarios, but especially when transferring data over WANs and vulnerable edge networks. A major downside of most file transfer tools is that they don’t provide native security features, forcing you to invest in 3rd-party security tools or VPNs.
Resilio protects your data end-to-end using built-in security features that were reviewed by 3rd-party security experts (TPN Blue certified and SOC2 certified), such as:
- AES 256-bit encryption: Resilio protects data at rest and in transit using AES 256-bit encryption.
- Mutual authentication: Every Resilio agent must provide a security key before it can receive files. This ensures your data is only delivered to approved endpoints and protects you from malicious infiltration in vulnerable edge deployment scenarios, such as IoT data ingestion.
- Cryptographic integrity validation: As stated earlier, Resilio uses data integrity validation to protect your files from corruption.
- Permission controls: You can control who’s allowed to access specific files and folders.
- Forward secrecy: Resilio protects sessions with one-time session encryption keys.
- Proxy server: With Resilio’s Proxy Server capabilities, you can block other devices from accessing or running scripts on your devices (or just accessing certain parts of a device) and support advanced firewall configurations. This is particularly useful when working with work-from-home employees, 3rd-party contractors, and other companies.
Use Resilio Connect for Fast, Reliable UDP Transfer
While UDP protocols are superior for transferring large files over WANs and unreliable networks, most UDP-based file transfer solutions/protocols can’t provide the speed and reliability that Resilio Connect can. Resilio’s superior file transfer is due to its:
- Proprietary UDP protocol: Resilio’s UDP-based protocol, ZGT, dynamically adjusts to any network in order to maximize bandwidth utilization. And it includes features that overcome the weaknesses of traditional UDP protocols, such as interval acknowledgments, delayed retransmission, and cryptographic data integrity validation.
- P2P transfer architecture: Most file transfer solutions that use UDP protocols (such as Signiant and Aspera) are built on slow, unreliable point-to-point replication architectures. Resilio’s peer-to-peer transfer architecture provides blazing-fast sync speeds, scales organically, transfers in any direction, and eliminates single points of failure.
- Bulletproof reliability: Resilio provides features that enhance reliability, such as dynamic rerouting, automatic retries, and checksum restarts.
- Vendor and workflow-agnosticism: Resilio works with just about any device, cloud storage provider, and operating system. It also seamlessly integrates with your workflow and popular solutions your team is already using.
- Centralized management: Resilio’s Management Console enables you to manage every aspect of replication across your entire environment — such as creating bandwidth profiles, automating troubleshooting protocols, and adjusting replication parameters.
- Includes state-of-the-art security features: Resilio provides built-in security features that protect your data at rest and in transit.
Organizations in media, gaming, engineering, retail, tech, and more use Resilio Connect to sync and transfer large files over any network (WANs, LANs, edge, etc.). To learn how Resilio can provide fast, reliable sync for your organization, schedule a demo with our team.
