Whether you’re replicating files for a website or an application, you need a web server file replication solution that can:
- Replicate files quickly, so users across your entire environment always have access to the most up-to-date files and don’t experience any errors.
- Replicate files in the direction(s) you need — i.e., one-to-one, one-to-many, or many-to-many, etc.
- Scale to handle your replication needs, whether that be working with larger files, more files (up to many millions of files), or replicating to more endpoints.
- Sync quickly and reliably across WANs, in the event that some of your servers are in different locations and/or high-latency networks.
- Include features that make it fault tolerant, in order to ensure availability of replicated data (without forcing you to spend time and money creating complex disaster recovery systems).
Most conventional web server file replication solutions (such as rsync, RepliWeb, and other content replication tools) fall short on one or more of the requirements above. And as your replication needs increase, these tools experience frequent breakdowns and replication issues.
But Resilio Connect is a highly-scalable, real-time file replication and synchronization solution that uses a P2P transfer architecture and WAN optimization to provide some of the fastest (10+ Gbps), most reliable transfer speeds available.
It’s powerful and flexible enough to handle any replication use case, and is particularly well-suited for large deployments that require fast, scalable, and real-time file distribution and synchronization.
If you want to learn more about how Resilio Connect can provide fast, scalable, and reliable web server file replication for your application or website, schedule a demo.
P2P Architecture for Speed & Organic Scalability
The primary difference between Resilio and other web server replication solutions lay in how they perform file replication. While traditional tools use point-to-point replication topologies, Resilio delivers faster, more scalable replication and synchronization using a P2P (peer-to-peer) transfer architecture.
Challenges with Traditional Replication and Sync Approaches
Most conventional web server file replication solutions replicate file systems using one of two topologies:
- Hub-and-spoke replication: In this topology, one server is designated as a hub-server and the other servers are client servers. All replication must first go through the hub-server. This hub-server must replicate files to all other servers one at a time. So if you’re replicating to 5 servers, the hub-server will need to send the same set of files 5 times, creating a bottleneck in your workflow.
- Follow-the-sun replication: In this topology, replication occurs from one server to another — i.e, Server 1 replicates to Server 2, then Server 2 replicates to Server 3, and so forth.
In both models, replication occurs sequentially.
In the “follow-the-sun” model, replication must complete between the first two servers before it can begin on the next server (and so on).
In the “hub-and-spoke” model, performance will be limited by:
- The bottleneck of this extra hop (which is the total round-trip time it takes to send/sync a file).
- The slowest link between the source and the target endpoint.
- Any latency incurred across the end-to-end transmission.
- Limits on file sizes, etc.
When replicating large volumes of data across many endpoints, it can result in situations and requirements such as:
- Sync delay and lack of data availability: Since synchronization occurs sequentially, large syncs with many files will take some time. This can result in delays, at best. At worst, the data will not be available to app users/website visitors across all servers (i.e., broken links and missing images on web pages where the server hasn’t received the updates).
- Scalability issues: Without fast replication, you may have situations where you need to keep users on a single server for long periods of time (aka session affinity), which impacts scalability and fault tolerance. For example, imagine a user uploading a profile image to their social media profile. If they switched to a different server mid-session before their update synced to all servers, that person wouldn’t be able to see the image they just uploaded. With session affinity, the user would stay on the same server they used to upload the image until the replication job is complete.
P2P Replication and Sync
Resilio, on the other hand, uses a distributed peer-to-peer replication architecture. In this model, every server can replicate files to every other server omnidirectionally without the need for cloud-hopping to a hub-server.
This essentially means that every server in your replication environment is a hub-server that can contribute to any replication job. And as you add more replication endpoints, speed increases.
As more endpoints are added to the replication or synchronization job, speed increases. P2P replication is key to Resilio’s organic scalability and ability to sync up to 10x faster than conventional tools (delivering 10+Gbps per endpoint).
Speed
When replicating files, Resilio also uses a process known as file-chunking — i.e., the process of splitting files into several blocks that can transfer independently to multiple destinations. File chunking enables Resilio to perform parallel, concurrent synchronization.
For example, if Server 1 needs to replicate files across your replication environment, it can split the file into six blocks and share those blocks with Server 2. Once Server 2 receives the first file block, it will begin sharing it with Server 3 even before it receives the full file. Once Server 3 receives a file block, it will begin sharing it with Server 4, and so forth.
This ability to sync concurrently means that Resilio’s replication speed increases as you add more replication endpoints to your environment and makes it a perfect solution for applications where end users must have the right data promptly.
You can also adjust replication speed to meet your needs using Resilio’s management console. For example, you can control bandwidth and bandwidth allocation policies to enhance performance for your most active servers.
Case Study: VoiceBase, a speech-to-text software for audio and video transcription, uses Resilio Connect to distribute speech models (with file sizes of 50+ GB) across 400+ production servers. “Resilio Connect enables us to reliably distribute our code, specifically new language models in a fraction of time. These copy jobs now take an hour, down from eight. Best of all, once Resilio Connect was installed, it just works: We never need to manually intervene in any way.”
Learn more about how Resilio Connect helped reduce VoiceBase’s replication time by 88%.
Organic Scalability
With conventional replication solutions, the more your replication environment grows (i.e., replicating more files, larger files, and/or to more endpoints), the longer it takes to synchronize files across servers. It also becomes more complex and costly to manage replication — and potentially impacts data availability across those systems.
In a hub-and-spoke environment, your entire system is dependent upon the hub-server. This server must be online and available at all times in order for your application to work. This forces you to invest in NFS backup servers and create a system that’s prepared to failover in the event that your hub-server goes down (which can occur due to maintenance, hardware failures, network failures, and more).
Resilio is an organically scalable file replication solution — i.e., the more servers you add to your replication environment, the better Resilio will perform.
With more servers taking part in file replication, Resilio’s replication time is reduced, because:
- There are more servers to distribute files across your environment.
- Resilio engages in load-balancing and distributes the load across multiple servers.
Resilio scales from a few servers to many thousands. This makes Resilio a particularly effective solution for large deployments with hundreds or thousands of replication endpoints.
Resilio supports files of any size, and is able to replicate 250+ million files per job, over any type of network.
In web server replication, there are a number of scenarios Resilio addresses. These include:
- One-to-one: Basic one and two-way file sync between two servers.
- Two-way (bidirectional): Files are reliably and rapidly synchronized between two or more endpoints or sites.
- One-to-many: Replicate files quickly, efficiently, and reliably from one source (any system running a Resilio Connect agent) to thousands of destinations. With Resilio, one-to-many file distribution takes virtually the same time as two-way replication between two servers.
- Many-to-one: Resilio makes it easy to consolidate files received from multiple endpoints. For example, you can automatically sync updates from multiple sources onto a single site, and control how this occurs in Resilio’s management console.
- Many-to-many: Unlike competing solutions, Resilio is designed to perform full mesh N-way sync across any number of servers. This enables you to make multiple updates to multiple servers, and sync those changes to every other server quickly and efficiently.
Low-Latency Sync with WAN Optimization
In instances where servers are located in different locations across networks, Resilio Connect uses WAN acceleration technology to overcome the latency and provide fast, predictable replication of files across WANs.
The file transfer protocols used by conventional file replication solutions aren’t optimized for WAN transfer. But Resilio uses a proprietary WAN transfer protocol known as Zero Gravity Transport (ZGT™). ZGT reduces latency and packet loss using:
- A congestion control algorithm that calculates the ideal send rate to avoid overloading the network.
- Interval acknowledgements for a group of packets that include additional info about packet loss.
- Delayed retransmission of lost packets to decrease unnecessary retransmissions and increase transfer speed.
Try our transfer speed calculator to see how much time we can save for you.
Active-Active High Availability & Resilience
Resilio’s WAN acceleration and P2P transfer architecture make Resilio one of the most fault-tolerant replication solutions available.
In a P2P environment, there is no single point of failure. If a server (or the network it’s using) goes down, Resilio can dynamically route around the outage and access other servers in order to complete a file sync.
With omnidirectional sync and the ability to fully utilize any network connectivity in your environment, Resilio can meet sub-five-second RPOs (Recovery Point Objectives) and RTOs (Recovery Time Objectives) within minutes of an outage — making it an ideal solution for quick disaster recovery.
Easy to Use & Manage
Resilio is one of the easiest server replication solutions to use because of its…
Flexible Deployment and Friendly User-Experience
Resilio is integrated with the file management tools used in popular operating systems (e.g., Windows Explorer, Mac Finder, etc.). There are no proprietary management features, so managing your replication jobs is easy and familiar. With its BYO deployment model, you can use Resilio in the cloud, on-prem, or a hybrid model. And Resilio can be configured cross-platform (on Linux, OS X, Android, iOS, and Microsoft Windows servers).
Automation and Centralized Management
Through the Resilio Connect Management Console, you can monitor and manage all replication jobs and control Resilio’s functionality. All job types can be scripted in Resilio’s REST API and integrated into your workflow.
It provides real-time notifications and detailed logs that give you insight into and control over the replication and how it occurs. And you can set up Webhooks, create rules and notifications, manage policies for jobs and users, control user permissions, and adjust replication parameters (such as bandwidth use, buffer size, disk io threads, and more). It’s easy to govern and audit which files are to be transferred and which are excluded.
Case Study: MixHits Radio, a music streaming platform, uses Resilio to update their music metadata across all of their servers. “We have gone from spending 15 hours on average per week troubleshooting conflicts in the prior solution to spending no time at all with Resilio. We configure jobs once in the Resilio Connect Management Console and never have to look at it again.”
Learn more about how Resilio helped MixHits save 60+ hours of work per month.
Quick Deployment
As an agent-based software solution (where Agents are installed on all endpoints participating in replication), installation is seamless, straightforward, and well-documented. Resilio can be installed on your existing infrastructure and begin replicating in as little as 2 hours.
Resilio supports a variety of servers, virtual cloud servers, desktops, mobile devices, and cloud storage services.
Secure, Reliable Replication
Resilio protects your data and ensures reliable replication using:
- In-transit encryption: All files are encrypted in transit with AES 256.
- Forward secrecy: One-time session encryption keys are used to protect sensitive data.
- Mutual authentication: Resilio only delivers data to designated endpoints.
- Cryptographic data integrity validation: Resilio uses cryptographic integrity validation to ensure your data arrives at its destination intact and uncorrupted.
Try Resilio Connect
For most replication deployments (and especially large deployments), Resilio Connect is the best web server file replication solution available because it provides:
- Fast replication: Resilio’s P2P transfer architecture and WAN optimization enable it to sync up to 10x faster than traditional replication solutions, ensuring your files are always where you need them, when you need them.
- Organic scalability: Because every device works together collectively, Resilio’s performance improves as your replication environment grows. It can replicate files to thousands of devices in the same time that most point-to-point solutions replicate between two, making it the ideal solution for large deployments with many replication endpoints.
- Fast transfer over any network: With Resilio’s proprietary WAN acceleration transfer protocol (ZGT™), it can fully utilize all bandwidth and optimize file transfer over any network.
- Reliable replication: P2P transfer means that there are no single points of failure. Resilio can route around network outages (and slow networks) and ensure that files always reach their destination.
- Easy management: Resilio is easy-to-use, can be deployed on your existing infrastructure in as little as two hours, and enables you to control every aspect of how replication occurs in your environment.
If you want to learn more about how Resilio Connect can provide fast, scalable, and reliable web server file replication for your application or website, schedule a demo.
