AWS DataSync vs Storage Gateway vs Resilio: What Each One Can Do

While they seem similar, AWS (Amazon Web Services) DataSync and AWS Storage Gateway serve different use cases.

AWS DataSync is an online data transfer service that moves data over the internet or AWS Direct Connect from on-premise servers to the AWS cloud.

AWS Storage Gateway is a hybrid cloud storage service that enables on premises access to virtually unlimited cloud storage by linking it to Amazon S3.

Resilio Connect does this and more: It’s a software-only agent-based solution that runs on any device and reliably transfers data across any type of network, irrespective of bandwidth and latency. Resilio transfers and synchronizes files in real-time or on-demand and in any direction. You can transfer one- and two-way, many-to-one, one-to-many, or many-to-many.

As a storage gateway, Resilio provides a file interface to any type of object storage, including S3. You can selectively sync files on-demand with cloud storage in AWS or any cloud provider — as well as on-premise storage. This keeps data egress costs low through minimized and efficient transfer. 

Resilio agents are installed directly on existing systems rather than provisioning VMs. The solution supports any OS and virtually any cloud storage provider, including AWS, Azure, GCP, MinIO, Wasabi, and more. Resilio’s speed, flexibility, and efficiency make it a superior solution for hybrid cloud, multi-cloud, remote work, data migration, and disaster recovery use cases.

In this article, we’ll discuss how AWS DataSync and AWS Storage Gateway work, when to use one tool over the other, and when to use both. We’ll also explore how Resilio Connect provides a faster, more reliable, and more efficient solution for syncing and accessing data (any type of file, block, and object) stored on premises and in any S3-compatible cloud object storage. 

Businesses such as 2K Games, Match.com, Delirio Films, Turner Sports, and more use Resilio Connect for hybrid cloud synchronization. To see Resilio in action, schedule a demo with our team.

AWS DataSync is useful for one-time data migrations from on-prem to the cloud, or for synchronizing data between two endpoints. 

However, its point-to-point architecture introduces single points of failure and limits its performance to the slowest endpoint. It also can’t sync data in real-time or sync in multiple directions. These shortcomings make DataSync unsuitable for scenarios that require rapid sync to multiple endpoints — such as distributed workforces, file distribution, or transfers over high-latency, low-quality networks (such as edge ingest and sync).

AWS Storage Gateway is useful for accessing data stored as files, tapes, and storage volumes in AWS. 

But it doesn’t work with other cloud storage providers, forcing you to use another storage gateway to access files in multi-cloud scenarios. It’s also not designed for efficient file access (i.e., minimizing data transfer costs), which can lead to unexpected charges on your monthly AWS bill.

Unlike AWS DataSync, Resilio Connect uses a P2P (peer-to-peer) architecture that provides blazing-fast, reliable, multi-directional sync.

Organizations like Sunrise Productions use Resilio to enable geographically distributed employees to collaborate on projects in real-time across AWS regions. 

Resilio eliminates cross-region latency in AWS, so changes made by users in one location are instantly reflected across all other endpoints in 4–5 seconds — roughly 10x faster than traditional point-to-point solutions like DataSync. Resilio can also perform horizontal scale-out replication and reach speeds of 100+ Gbps per cluster.

Unlike AWS Storage Gateway, Resilio provides access to files stored as objects in just about any S3-compatible cloud object storage (not just AWS). It increases efficiency by enabling you to choose which files to cache locally so you can free up space on on-premise servers while giving employees faster access to the files they need.

Overall, Resilio Connect provides file synchronization and access that is:

• Fast: Our engineers successfully replicated a 1 TB dataset across Azure regions in 90 seconds.

• Organically scalable: Resilio can sync hundreds of endpoints in roughly the same time it takes most point-to-point solutions to sync two.

• Reliable: Resilio’s P2P architecture eliminates single points of failure.

• Multidirectional: Resilio can sync one-way, two-way, one-to-many, many-to-one, and N-way.

• WAN-optimized: Resilio’s proprietary WAN acceleration protocol optimizes traffic in order to sync quickly and reliably over any network — including high-latency WANs and low-quality edge networks.

• Efficient: Resilio’s file gateway includes features such as partial downloads, selective synchronization, selective caching, and more, enabling you to minimize data egress charges.

• Centralized: You can manage, monitor, and control file sync and access over your entire on-prem and multi-cloud environment from one location.

• Secure: Resilio includes built-in security features that protect your data at rest and in transit.

Organizations use Resilio Connect to distribute and synchronize data for media workflows (CBS, Turner Sports), gaming (Blizzard, Wargaming.net), remote operations (McDonald’s, Mercedes-Benz), and more. To see for yourself how Resilio Connect can provide blazing-fast synchronization and efficient access to files stored in AWS, schedule a demo.

AWS DataSync vs. Storage Gateway

Below, we describe how AWS DataSync and Storage Gateway work, the scenarios in which they should be used, and when you can leverage both solutions.

AWS DataSync

AWS DataSync works via Virtual Machine agents that are used to read and write data from storage systems. The agents can sync data from storage machines to Amazon S3, Amazon EFS, or Amazon FSx for Windows File Server (and across storage regions). You can monitor and automate the AWS DataSync migration service through AWS CloudWatch and AWS Lambda.

DataSync also includes a tool known as DataSync Discovery, which provides visibility into on-premise storage performance and utilization to help you simplify and accelerate data migration to the AWS cloud. Through automated data collection and analysis, it provides user-generated recommendations about which AWS storage services suit your needs.

AWS DataSync Use Cases

You should use AWS DataSync when transferring data into Amazon S3 and you need:

• Online data transfer: DataSync is used to continuously ingest and transfer files from remote locations into AWS. 

• Built-in data integrity checks: Unlike Storage Gateway, AWS DataSync includes built-in data integrity checks. For Storage Gateway, you’d need to code your own data validation logic.

• File filtering: AWS DataSync enables users to create filters for files and folders that they want to migrate to S3. Storage Gateway does not.

• Multi-user access: DataSync enables multiple users to access the same S3 bucket.

DataSync is also an appropriate solution when transferring from on-premise devices or Amazon S3 to AWS Elastic File System, as well as from on-premise blocks or tapes to AWS (as Storage Gateway can’t perform these transfers).

Storage Gateway

AWS Storage Gateway operates via the Storage Gateway Appliance from Amazon that is hosted on a data center though you can also purchase and use a hardware appliance. Data can be stored in three types of interfaces:

• File Gateway: Object storage for data lakes, backups, and machine learning workflows.

• Volume Gateway: Cloud-backed iSCSI block storage volumes.

• Tape Gateway: Virtual tapes in AWS.

AWS Storage Gateway Use Cases

You should use AWS Storage Gateway if you want to extend your on-premise storage into the AWS cloud and provide on-prem users with low-latency file access.

Storage Gateway is also the better solution in certain data migration scenarios. For example, if you want to migrate files into Amazon S3, you don’t need to filter files and folders, and/or your team can code your own data validation logic (or you don’t need data validation at all), then Storage Gateway can perform the migration and is 25% less expensive than AWS DataSync.

When to Use Both AWS DataSync and Storage Gateway

You can use AWS DataSync to migrate existing data to Amazon S3, and then use the AWS Storage Gateway to access the data sync with your on-premise devices.

How Resilio Provides Superior File Sync and Access

Resilio Connect is a file synchronization software system that can also be used as an object storage gateway for files stored in any S3-compatible cloud storage.

If you’re storing data in AWS services other than S3 (such as Amazon EFS, FSx, etc.), you’ll still need to use DataSync and/or Storage Gateway. But Resilio can provide fast, reliable, scalable data replication across Amazon S3 — as well as efficient, low-latency access for your on-premise file systems.

Note: For one-time and continuous data migrations into AWS from the edge, you should also consider using AWS Snowball (which includes resources for cloud computing) and AWS Snowcone. 

P2P Architecture: Fast, Reliable, Scalable Replication

While you can use AWS DataSync to transfer and sync data in the AWS cloud, it’s a point-to-point solution that suffers from many shortcomings, such as:

• Slow replication: Replication can only occur between two endpoints at a time.

• Poor scalability: Since replication can only occur between two endpoints at a time, synchronization can take a long time, particularly for large environments (i.e., large files, large numbers of files, large numbers of endpoints).

• Unreliability: Point-to-point solutions create single points of failure. If replication slows or fails between any two endpoints, it can delay sync for your entire environment.

Unlike AWS DataSync’s point-to-point architecture, Resilio Connect syncs data in real-time using a proprietary peer-to-peer architecture that makes it fast, organically scalable, highly reliable, and multi-directional.

Blazing Fast Replication and Synchronization

Every endpoint in a P2P environment can replicate data simultaneously, enabling Resilio Connect to provide some of the fastest replication speeds available.

While DataSync can only perform scheduled syncs, Resilio Connect can sync on a schedule, in real-time, and manually. Using optimized checksum calculations (identification markers are assigned to each file that changes when a change is made to the file) and notification events from the host operating system, Resilio can immediately detect and replicate changes made to a file in real-time.

Resilio also uses a process known as file chunking to break files down into multiple chunks that can transfer independently from each other. If, for example, you wanted to sync a 10 GB file across your environment, Resilio could split that file into five chunks of 2 GB each. 

Endpoint 1 could share the first chunk with Endpoint 2. Endpoint 2 can then share that chunk with Endpoint 3, even before it receives the remaining four chunks. Soon, every endpoint will be working together to sync your system, resulting in sync that is 3–10x faster than traditional point-to-point solutions.

Organic Scalability

Because DataSync can only sync point-to-point, it scales poorly. When replicating large amounts of data across many endpoints, a full synchronization of your system will take a long time and performance will be limited by the slowest endpoint.

Resilio’s P2P replication architecture eliminates this issue and makes your environment organically scalable. Since every endpoint can take part in replication simultaneously, adding more endpoints only increases replication speed and available resources.

Resilio can perform horizontal scale-out replication, enabling it to:

• Sync hundreds of endpoints in roughly the same time it takes point-to-point solutions to sync two.

• Reach replication speeds of 100+ Gbps per cluster (our engineers successfully replicated a 1 TB data set between Azure regions in 90 seconds).

• Sync files of any size, type, or number. We’ve successfully synced 450+ million files in a single job.

Bulletproof Reliability

Since replication can only occur point-to-point with DataSync, if replication between two endpoints fails, then it can delay synchronization for your entire environment.

But Resilio’s P2P environment eliminates these single points of failure, since every endpoint can sync with any other endpoint at any time. If an endpoint or network goes down, the necessary files or services can be retrieved from any other endpoint in your system. 

This enables Resilio Connect to provide Active-Active HA for disaster recovery — it minimizes data loss via backups and active/passive or active/active failover strategies. When recovering from a disaster (such as server failure), Resilio can utilize every endpoint in your environment and achieve sub-five-second RPOs and RTOs within minutes of an outage.

Resilio can also dynamically reroute around outages or downed networks in order to find the optimal path and ensure your data always reaches its destination.

Multi-Directional Replication

DataSync can only sync files one-way and two-way. But Resilio Connect can sync files in any direction, such as:

• One-way: You can sync data to a backup site at a data center or in the cloud.

• Two-way: Businesses can keep two endpoints synchronized.

• One-to-many: DevOps teams can distribute files and software updates across many servers or to a fleet of vehicles.

• Many-to-one: Organizations can consolidate data from many endpoints to a backup or central site.

• N-way: Remote and hybrid workforces can keep hundreds of servers synchronized simultaneously, enabling geographically distributed teams to collaborate on files and sync any changes across your entire network.

WAN Acceleration: Fully Utilize Any Network

One area where Resilio really outshines DataSync is with WAN transfer. WAN networks suffer from high latency and varying degrees of packet loss. When syncing data over WAN networks (such as geographically distributed cloud regions, on-prem devices, or to/from remote locations), you need a solution that can optimize network utilization.

DataSync optimizes WAN transfers with an AWS-designed WAN transfer protocol that uses incremental transfers, sparse file detection, in-line compression, and multi-threaded connections between the local DataSync agent and the in-cloud components. While these methods reduce the size of data being transferred, they don’t efficiently manage/optimize traffic over the network — which is important for maximizing transfer speeds over high-latency and low-quality networks.

Resilio Connect inherently performs incremental transfers (i.e., only replicating changed portions of files) and enables multithreaded transfers. But Resilio also utilizes a proprietary UDP-based WAN acceleration protocol known as Zero Gravity Transport™ (ZGT).

ZGT reduces network traffic through advanced deduplication and transport efficiency to overcome latency and loss. It fully utilizes any network connection to optimize transfers via:

• A congestion control algorithm: ZGT uses a congestion control algorithm that constantly probes the Round Trip Time to identify and maintain the ideal data packet send rate.

• Interval acknowledgments: Most protocols send an acknowledgment after each packet receipt. But ZGT sends acknowledgements in groups to reduce network congestion.

• Delayed retransmission: Most protocols retransmit lost packets immediately. ZGT retransmits lost packets in groups once per RTT to reduce unnecessary retransmissions.

But Resilio is especially useful for syncing data from edge devices. ZGT enables Resilio to fully utilize any type of network connection, such as VSAT, cell (3G, 4G, 5G), Wi-Fi, or any IP connection. It can reliably ingest data from the edge into the cloud, and immediately sync it across all cloud and on-prem endpoints. Clients such as Northern Marine Group use Resilio to sync data across their fleet of geographically distributed sea vessels.

Case Study: Shifo

Shifo provides over 3,000 remote communities with health data solutions. They use Resilio Connect to sync their servers across remote communities with underdeveloped network connections (such as Uganda). 

“You can’t imagine how difficult it was for us [prior to Resilio]: Delays, nothing being synchronized; now I don’t hear any of those problems. Now we can focus on solving healthcare problems [instead of moving files].” 

Learn more about how Shifo uses Resilio Connect to reliably sync data to remote, poorly-connected areas.

Efficient, Low-Latency File Access

Resilio Connect also works as a storage gateway, allowing you to ingest, sync, and access files — through SMB (Server Message Block) and NFS (Network File Sharing) protocols — stored in any S3-compatible cloud object storage (such as AWS).

Many organizations that use AWS data storage don’t properly calculate storage fees and are surprised by large bills due to egress charges. AWS users are charged egress fees within and across AWS cloud regions, as well as out of AWS.  

In contrast, Resilio Connect’s object storage gateway is designed to enhance efficiency and optimize AWS egress costs through features such as:

• Selective sync: You can control which files/folders sync to which endpoints. This enables you to reduce costs by ensuring files only sync to necessary endpoints.

• Selective caching: Many storage gateways (such as AWS Storage Gateway) cache recently accessed files locally. Resilio Connect enables you to cache whichever files you want on local devices. You can store frequently accessed files locally and store infrequently accessed files in long-term cloud storage to free up space on on-prem servers and give employees quicker access to the files they need.

• Create policies: You can create policies that control how files are synchronized, downloaded, cached, and purged. By automating this process, you free up employees to work on their tasks.

• Partial downloads: Employees can perform partial downloads to get quicker access to just the portions of files that they need and minimize egress charges.

• Unified interface: All employees are presented with a unified interface for browsing and accessing files that operates much like Microsoft OneDrive.

If you’re storing files as tapes or storage volumes, you’ll still need to use AWS Storage Gateway to access these files. But you can use Resilio in conjunction with Storage Gateway to quickly and efficiently sync and access files stored in Amazon S3.

Granular, Centralized Control over Your Environment

Resilio is an agent-based software that installs on your existing infrastructure on virtually any device, such as desktops, servers, laptops, NAS/DAS/SAN devices, mobile devices (Resilio offers Android and iOS apps), and VMs (such as VMware, Citrix, and hypervisors). It works with any operating system, such as Windows, Linux, MacOS, Unix, Ubuntu, FreeBSD, OpenBSD, and more. 

Resilio Connect also supports any S3-compatible cloud object storage provider, such as AWS, Google Cloud Platform, Azure, Wasabi, MinIO, and more. You can set up and begin replicating in as little as two hours.

Because of Resilio’s flexibility and versatility, you can use it to sync, manage, and access files stored across all of your on-premise and cloud endpoints from one centralized location.

Resilio’s Management Console also provides granular control over how replication occurs. You can increase efficiency and program your jobs to occur exactly as you need by:

• Optimizing bandwidth utilization: You can configure profiles that govern how much bandwidth is allocated to each endpoint at certain times of the day and on certain days of the week.

• Sync automation: You can automate how syncs occur at each endpoint, including which files sync to which endpoints. You can also perform real-time, scheduled, or manual syncs.

• Performance optimization: You can optimize resource utilization by adjusting replication parameters, such as data hashing, file priorities, buffer size, packet size, disk I/O threads, and more.

• Scripting and functionality: Using Resilio’s powerful REST API, you can script any type of functionality your replication jobs require.

• Automate troubleshooting: You can create troubleshooting policies, such as file conflict resolution, to minimize human intervention.

Case Study: Deutsche Aircraft

Deutsche Aircraft is a Resilio client that manufactures commercial aircraft. After switching from DFSR, they began using Resilio Connect to synchronize their Microsoft DFS namespace, increase replication efficiency, save time, and protect their business-critical data. 

“We have a 10Gbps network but prefer to use under 1Gbps for data transfer and replication. With Resilio, we’re able to keep that down to 250Mbps during the day and at night move back up to 1GBps… Resilio Connect is much easier to manage than DFSR. Using the Resilio Connect management console, you can see everything you need to know. Everything is visible.”

Learn how Deutsche Aircraft uses Resilio Connect to significantly reduce the time it takes to manage data synchronization.

Native Security Features

Many replication and sync solutions require you to invest in 3rd party security tools and VPNs. But Resilio Connect includes built-in security features that protect your data end-to-end in all scenarios:

• AES encryption: Resilio encrypts data at rest and in transit using AES 256-bit encryption.

• Mutual authentication: Before receiving any data, every endpoint must provide an authentication key. This ensures that data is only delivered to approved, verified endpoints.

• Permission controls: Using the Resilio Management Console, you can control who can access specific files and folders.

• Immutable copies: Resilio protects files from ransomware by storing immutable copies in the cloud.

• Forward secrecy: Resilio protects each session with one-time session encryption keys.

Use Resilio Connect for Fast, Efficient File Sync and Access in AWS

Resilio Connect provides a fast, flexible, efficient, and centralized solution for syncing and accessing files stored in any S3-compatible cloud object storage, such as AWS. Resilio provides:

• Blazing fast sync speeds: Through its P2P architecture, Resilio can reach speeds of 100+ Gbps per cluster and sync data 3–10x faster than point-to-point solutions (such as DataSync).

• Organic scalability: Resilio scales organically. Every endpoint you add to your environment increases sync speed and resources, enabling Resilio to sync large files and large numbers of files across hundreds of endpoints in roughly the same time it takes most solutions to sync two.

• WAN optimization: Resilio’s proprietary WAN acceleration protocol enables you to fully utilize any network (including VSATs, cell, Wi-Fi, and any IP connection) and optimize transfer speeds across cloud regions (we replicated a 1 TB dataset across Azure regions in 90 seconds).

• Bulletproof reliability: Resilio’s P2P architecture eliminates single points of failure. It can dynamically route around outages and retry transfers until they’re complete — ensuring your files always reach their destination.

• Multi-directional replication: Resilio can sync files in any direction, such as one-way, two-way, one-to-many, many-to-one, and N-way.

• Efficient file access: Resilio provides low-latency, efficient access to files stored in any S3-compatible cloud object storage. You can reduce costs and increase productivity through selective sync, selective caching, partial downloads, and more.

• Centralized control: Resilio enables you to manage all of your cloud and on-premise endpoints from one centralized location. And it gives you granular control over replication so you can increase efficiency and optimize costs.

• Built-in security: Resilio’s built-in encryption and security features protect your data end-to-end.

Organizations use Resilio Connect to distribute and synchronize data for media workflows (CBS, Turner Sports), gaming (Blizzard, Wargaming.net), remote operations (McDonald’s, Mercedes-Benz), and more. To see for yourself how Resilio Connect can provide blazing-fast synchronization and efficient access to files stored in AWS, schedule a demo.