Reliable and Secure Edge-to-Cloud Solution: Resilio Connect

Data is generated and stored everywhere — at the edge, in the core, and in the cloud.  But the reality is that more and more data is being generated and stored at the edge. Conventional approaches to edge data collection and transfer simply can’t keep up. 

There are numerous approaches to acquiring, collecting, and transferring data from the edge to the core. Yet, in places where there’s poor or unreliable connectivity, it’s a must-have to overcome latency and unreliable networks with intermittent connectivity to reliably transfer data on time, all the time.  

Some popular use cases for edge-to-cloud data transfer include:

• Multi-site data collection from the edge: Ingesting data from multiple edge devices and transferring files to a central location. Optionally, some companies may also need to synchronize files at the edge with the core or cloud.

• Continuous real-time vs on-demand data flow: Ingesting data from the edge to the cloud (or other locations) quickly and regularly, either on-demand, on a schedule, or in real-time.

• Real-time automated data transfer and sync: Immediately and efficiently transferring new or updated data collected from the edge is a must-have for some applications. When new files or updates are collected, they should be automatically transferred from the edge to the cloud and other endpoints.

Simplicity is also a big factor here. When storing data in multiple clouds and facilities, managing that data from a single place is critical to avoid having to manage multiple access points.

At the same time, edge-to-cloud workflows are often hampered by challenges like unreliable and spotty network connectivity (i.e., weak connections that drop in and out), operating in areas with underdeveloped network infrastructure, bandwidth constraints, high latency, and more. 

Want to try our fast, reliable, and efficient software-only approach for reliably syncing data from edge to cloud? Schedule a demo with our team.

Our real-time edge-to-core data transfer solution overcomes extreme latency and loss — for reliably collecting and transferring data across VSAT, cell, radio, Wi-Fi, and intermittent connections — to get your data where it needs to be in fixed and predictable time frames.

Resilio enables operators to securely acquire and transfer payloads of any size and type from the hardest-to-reach places to any other destination — using any network and device. The solution overcomes extreme latency and loss — for reliable use across VSAT, cell, radio, Wi-Fi, and intermittent connections — to transfer data in fixed and predictable time frames.

Resilio Connect enables businesses to:

• Predictably move data over any network: Resilio utilizes a proprietary WAN acceleration protocol to minimize network traffic to preserve bandwidth and optimize transfers over any network — including low-quality, unreliable networks at the edge. You can use Resilio with any type of network connectivity, such as VSATs, Wi-Fi, cell (3G, 4G, 5G), and any type of IP connection. You can also sync files and deploy updates efficiently and in predictable time frames.

• Sync reliably in any direction for continuous operations: Resilio ensures your data always reaches its destination. Its P2P (peer-to-peer) architecture eliminates single points of failure. It can dynamically route around outages and perform checksum restarts to resume interrupted transfers from the point of failure. And it preserves data across all endpoints with cryptographic integrity validation.

• Centrally control all global data movement: Through Resilio’s management console, you can adjust key parameters, control bandwidth, visualize and monitor transfers, receive notifications on job progress, and view performance metrics in real-time for all of your endpoints from one location.

• Automate for easy management: Resilio enables you to automate transfers through an intuitive central management console, custom scripting, and APIs. This makes Resilio a “set-it-and-forget-it” solution from a management perspective, as it eliminates the need to manage operations once they’ve begun.

• Use with your existing infrastructure: Resilio is a flexible solution that works with just about any device, cloud service provider, and operating system. You can deploy Resilio on your existing IT infrastructure (including edge devices, mobile devices servers, desktops, and more) and integrate with 3rd-party management tools (like Ansible, Microsoft SCCM, Splunk, LogRhythm, Loggly, Grafana, and more).

• Secure data end to end: Resilio protects all data at rest and in transit. Encrypt all payloads without reliance on 3rd-party security services or VPNs. 

Clients such as Northern Marine Group, a marine shipping company, use Resilio to sync data across their fleet of geographically distributed sea vessels. They used Resilio’s scripting engine to program their system to distribute/install updates and report back on status automatically, giving them more speed and insight into the remote update process.

In this article, we’ll describe how you can use Resilio Connect to quickly and reliably collect, sync, and access data from edge devices and networks to any cloud.

Resilio customers in media (Turner Sports, Skywalker Sound), remote operations (Northern Marine Group, Lindblad Expeditions), gaming (2K Games, Wargaming), and more use Resilio Connect to ingest and sync data over any network for multi and hybrid cloud scenarios. To learn more about how Resilio Connect can help you sync edge to cloud, schedule a demo with our team.

Predictably Ingest, Sync, and Transfer Over Any Network

There are two common types of edge synchronization scenarios:

1. Syncing data from in-field devices — for example, on-site servers for remote teams or IoT (Internet of Things) devices such as smart devices and sensors on in-field equipment — as well as edge computing, and cloud computing.
   
   
2. Syncing data from the extreme edge in areas with little network connectivity, such as at sea or in countries with poor network coverage.

In both cases, you’re relying on edge networks. Edge networks are plagued by spotty network connections that drop in and out, poor infrastructures, high latency, and varying degrees of packet loss.

These connectivity issues can make it infeasible for many organizations to transfer data over edge networks, which is why they often turn to physical data transfer solutions like AWS Snowball and Seagate Lyve. These come with their own downsides, such as the hassle of handling physical devices, the risks involved in shipping them across extreme edge environments, and more.

But Resilio Connect is optimized to overcome extreme latency and packet loss in order to transfer large amounts of data over any type of network, no matter how unreliable or low-quality.

Our solution uses a proprietary WAN acceleration protocol — known as Zero Gravity Transport™ (ZGT) — that enables it to maximize transfers and bandwidth utilization over any network.

ZGT optimizes network traffic and bandwidth utilization via:

• A congestion control algorithm: ZGT uses a congestion control algorithm that constantly probes the RTT (Round Trip Time) in order to calculate and maintain the ideal data packet send rate.

• Interval acknowledgements: Rather than sending an acknowledgement for each packet receipt, ZGT sends acknowledgements for groups of packets to decrease traffic over the network.

• Delayed retransmission: Rather than retransmitting lost packets with each acknowledgement, ZGT retransmits lost packets in groups once per RTT to reduce unnecessary retransmissions.

ZGT allows Resilio to work over any type of network connection, such as VSAT, Wi-Fi, cell (3G, 4G, and 5G), and any IP connection. This makes Resilio a great solution for continuously ingesting data from the edge, regularly updating a subset of data at the edge that needs to remain at the edge, and deploying updates efficiently to many remote systems in fixed time frames.

Even when connectivity is not available, Resilio enables offline access for data sharing. Plus, our solution can be configured to throttle bandwidth on slow uplinks and scale performance across high-speed networks.

Case Study: Ross Maritime

Ross Maritime is a shipping agency that specializes in providing logistical solutions that streamline the passage of passenger and commercial vessels through ports. To do so, they must coordinate with government agencies (such as US Customs and Port Authorities), which requires them to track, secure, and update documents like customs forms, cargo manifests, and passenger logs.

“With Resilio, all data, no matter how large, syncs reliably even when connectivity is poor.  Furthermore, Resilio is efficient, only moving the blocks of data that have changed to minimize usage on expensive metered connectivity.  It is the perfect platform to deploy modern digital applications at sea.”

Learn more about how Resilio Connect helps Ross Maritime increase productivity through secure and reliable edge sync.

Reliable P2P Edge Ingest, Sync, and Replication

Moving data from the edge of the network is already hindered by high latency, long distances, and poor connectivity issues. For timely data processing and decision-making, data collected at the edge must be quickly ingested and synchronized across multiple endpoints.

However, traditional transfer and sync solutions use inferior point-to-point replication architectures, which can be configured as:

• Hub and spoke replication: This consists of a hub server and multiple remote servers. The remote servers can’t share files directly with each other. Instead, all data must first be sent to the hub server, which then syncs the data with the remote servers one by one.

• Follow-the-sun replication: Replication occurs from one server to another sequentially.

Both methods aren’t reliable as they suffer from single points of failure. If replication is delayed or interrupted on any single server, it can delay synchronization for every other server in your environment. And if the hub server goes down in the hub and spoke model, replication fails entirely.

They also suffer from slow transfer and sync speeds. Since files can only be replicated between two servers at a time, syncing an entire environment can take a long time, especially for large environments with hundreds of endpoints, and/or large files and large numbers of files.

But Resilio Connect is designed to avoid these issues and provide reliable data ingestion, transfer, and sync in any scenario.

Multidirectional, Scalable P2P Replication

Resilio uses a P2P replication architecture. In a P2P environment, every server can share files directly with every other server. And all servers can take part in the synchronization process simultaneously.

P2P synchronization enables Resilio Connect to:

Ingest, Transfer, and Sync Data Reliably and Predictably

Since every server can share files with every other server, P2P replication eliminates single points of failure. If any server goes down, the necessary files or services can be provided by any other server in your environment. And Resilio can dynamically route around outages (such as downed servers or networks) and find the optimal path to deliver your data to its destination.

Resilio is also incredibly fault-tolerant. If a transfer is interrupted, Resilio will perform a checksum restart to resume the transfer where it left off (rather than starting over). It will retry failed transfers until they’re complete. And it uses cryptographic integrity validation to ensure your files remain uncorrupted when they reach their destination. 

You can use Resilio’s selective caching capabilities to cache any files you choose on local devices, so they’re always quickly available. If you’re on an unreliable network that drops in and out, you’ll still be able to access files and sync them over the LAN. When the connection resumes, Resilio will automatically sync your files across your entire environment.

Sync in Any Direction

Resilio Connect can sync files in any direction, such as one-way, two-way, one-to-many, many-to-one, and N-way. Multidirectional sync capabilities are especially useful in edge sync scenarios. For example:

• Distribution: Logistics organizations can use one-to-many sync to distribute updates and compliance files out to a fleet of geographically distributed vehicles or vessels.

• Consolidation: Logistics organizations can use many-to-one sync to collect in-field data from their fleet of geographically distributed vehicles or vessels.

• Edge to cloud: Data collected at the edge can be ingested to a cloud endpoint, then immediately distributed to multiple other cloud and on-premises endpoints via N-way sync. Resilio can sync data across multiple cloud platforms (Resilio is cloud vendor-agnostic) and edge locations concurrently and reliably.

• Disaster recovery: P2P sync effectively turns every server in your environment into a backup site, enabling you to create Active-Active High Availability. In the event of a disaster, every server in your environment can work together (N-way sync) to bring your system back online. Resilio can achieve sub-five-second RPOs (Recovery Point Objectives) and RTOs (Recovery Time Objectives) within minutes of an outage.

Resilio can also sync files in real-time. It uses optimized checksum calculations — identification markers assigned to each file that change when a change is made to the file — and notification events from the host OS to immediately detect and replicate just the file deltas (i.e., the changed portions of files). You can also perform manual synchronization and sync on a set schedule.

Move Data Faster

Resilio uses a process known as file chunking to break files down into multiple chunks that transfer independently of each other. Every server in your environment can share file chunks at the same time and enable low-latency synchronization.

Imagine you want to sync a 10GB file across five servers. Resilio can split that file into five chunks of 2GB each. Server 1 can share the first chunk with Server 2. As soon as Server 2 receives that first chunk, it can begin sharing it with Server 3 even before it receives the remaining file chunks.

Soon, every server will be sharing file chunks concurrently, allowing you to sync your environment 3–10x faster than point-to-point solutions.

Case Study: Maffeis Engineering

Maffeis Engineering designs and engineers large-scale structures, such as soccer stadiums and the world’s longest swing railway bridge spanning the Suez Canal. They use Resilio Connect to protect and sync their mission-critical project assets in real-time for remote, geographically distributed teams.

“When a change is made to one of our 3D design plans, it’s critical that everyone working on the project immediately sees the change and is working from the most current version of the file. With Resilio Connect, our engineers always have the most recent designs, wherever they happen to be located.”

Learn more about how Resilio Connect helps them overcome latency and sync data 5–10x faster than their previous solution.

Centralized Control and Automation

Managing data stored in disparate locations is one of the biggest challenges for organizations that operate or collect data at the edge. Mission-critical workloads can be stored on multiple devices, in multiple locations, and across multiple private clouds and public clouds.

Resilio Connect removes this complexity by providing a single interface from which you can manage and monitor all data in your hybrid or multi-cloud environment. And it gives you granular control over how data is replicated across your environment.

You can use Resilio’s Management Console to:

• Automate any type of functionality your job requires through set-it-and-forget-it transfers, Resilio’s REST API, and custom scripting.

• Manage each individual endpoint and job function across your environment from one location.
• Visualize and monitor transfers, configure notifications on job progress to be delivered to email or Webhooks, and obtain performance metrics in real-time.

• Adjust replication parameters (such as buffer size, disk io, and more) in order to optimize resource utilization and sync cost-effectively.

• Create and control replication jobs.

• Adjust bandwidth allocation, and create profiles that govern how much bandwidth each endpoint is allotted at certain times of the day and on certain days of the week.

• Manage files stored on any device or any cloud storage platform.

End-users can also browse files from an interface that operates much like Microsoft File Explorer — providing unified and familiar user experience.

Case Study: Delirio Films

Delirio Films produces documentary films and series, such as Reggie and They Call Me Magic. They use Resilio Connect to keep their media assets current and synchronized for everyone on their geographically distributed team.

“Resilio also makes it easy to scale operations and expand our user base. The ability to set up different jobs and handle all the user folder paths remotely was a game changer. We even had an editor who worked from Prague in the Czech Republic. Despite the time difference, we could manage and monitor new media traveling to him effortlessly.” 

Learn more about how Resilio Connect helps them easily and reliably sync data across production sites and enhance productivity.

Protect Data with Built-In Security

As stated earlier, many sync and file gateway solutions don’t include security features and force you to buy 3rd party security tools or VPNs.

But Resilio Connect includes native, state-of-the-art security features that were reviewed by 3rd party security experts, such as:

• AES 256-bit encryption: Resilio encrypts data in flight and at rest using AES 256-bit encryption.

• Data immutability: Resilio stores immutable copies of files in the cloud to protect against ransomware.

• Mutual authentication: Before initiating a transfer with any endpoint in your environment, Resilio requires the endpoint to provide an authentication key — ensuring your data is only delivered to approved locations.

• Cryptographic data integrity validation: To ensure your data arrives at its destination uncorrupted, Resilio uses cryptographic data integrity validation.

• User permission control: Through Resilio Management Console, you can set permissions that govern who is allowed to access specific files and folders.

• Forward secrecy: Resilio protects each session with a one-time session encryption key.

• Central monitoring: Monitor system-wide event logging for security audits from a centralized location.

Install Resilio on Your Existing Infrastructure

Resilio’s flexibility is one of the key features that make it a superior edge-to-cloud sync solution. As a vendor-agnostic, agent-based software, Resilio can be installed on your existing IT infrastructure and begin replicating in as little as two hours.

You can install Resilio agents on just about any:

• Device: Resilio supports file servers, desktops, laptops, NAS systems, DAS systems, SAN systems, mobile devices (Resilio offers iOS and Android apps), and virtual machines (such as VMware, Citrix, and hypervisors). You can also install Resilio agents on IoT devices with supporting operating systems.

• Cloud provider: Resilio supports most cloud services, such as AWS, Google Cloud, Azure, Wasabi, MinIO, Backblaze, Oracle, and more. This means you’re not limited by vendor lock-in and can build or tear down projects in any cloud at a moment’s notice.

• Operating system: Resilio works with just about any operating system, such as Windows, MacOS, Linux, Unix, Ubuntu, FreeBSD, OpenBSD, and more.

And in areas with spotty network connections that drop in and out, you can use Resilio while offline. When a connection is reestablished, Resilio will automatically sync your files across your environment.

Use Resilio Connect for Edge to Cloud Sync and Access

To overcome the challenges of syncing data from the edge to the cloud, Resilio Connect provides a solution that:

• Can move and sync data across any cloud: Resilio can transfer data from on-premises into just about any cloud, from cloud to cloud, and across cloud regions.

• Maximizes any network connection: Resilio’s proprietary WAN acceleration protocol, ZGT, optimizes transfers over any network and enables you to utilize any type of connection (such as VSAT, Wi-Fi, cell, and any IP connection). It also allows you to ingest data from remote locations with little network coverage and sync across multiple clouds and on-premises sites.

• Reliably syncs data: Resilio’s P2P architecture eliminates single points of failure. Resilio can dynamically route around outages, perform checksum restarts to resume interrupted transfers, and retries all transfers until they’re complete.

• Syncs data in any direction: Resilio can sync data one-way, two-way, one-to-many, many-to-one, and N-way. Multidirectional sync capabilities are especially useful for distributing, consolidating, and syncing data from the edge.

• Gives you granular control over your environment: Resilio’s Management Console provides a centralized location from which you can manage replication across your entire environment.

• Deploys flexibly: Resilio is hardware and cloud vendor agnostic. You can deploy it on your existing IT infrastructure and begin replicating in as little as two hours.

• Enables centralized data access: Resilio provides a file gateway that allows you to access data stored in any cloud or on-premises endpoint from one location. The gateway is optimized with features that enable you to maximize productivity and minimize cloud storage costs.

• Keeps data secure: Resilio includes built-in security features that protect your data at rest and in transit.

Resilio customers in media (Turner Sports, Skywalker Sound), remote operations (Northern Marine Group, Lindblad Expeditions), gaming (2K Games, Wargaming),  healthcare (Shifo), and more use Resilio Connect to ingest and sync data in real-time and on-demand. To learn more about how Resilio Connect can help you sync edge to cloud, schedule a demo with our team.

Frequently Asked Questions About Edge to Cloud

What is edge-to-cloud technology?

Edge-to-cloud technology extends the computing power of cloud infrastructure and cloud resources to all of an organization’s apps and data. Essentially, it refers to solutions that enable businesses to transfer and store data that is collected at the edge (i.e., devices that are close to the source of data, whether they be IoT devices or data collected at remote locations with little connectivity) to a centralized data center or to multiple sites where the data will be processed and utilized.

The primary benefits of edge-to-cloud technology are that it helps organizations quickly ingest and distribute data in order to make rapid, data-driven decisions, improve productivity, enhance the customer experience, and quickly provision new services. Edge-to-cloud tech can come in the form of data storage services, data replication, and synchronization solutions (like Resilio Connect), and managed services (such as edge cloud computing resources).

What is edge-to-core-to-cloud?

Edge-to-core-to-cloud refers to the process of transferring data collected at the edge to a centralized data center (whether at a physical site or in the cloud) and then syncing that data to a cloud storage ecosystem (i.e., one or multiple cloud storage sites). 

This can be accomplished through physical data transfer devices (such as AWS Snowball and Snowcone or Azure Data Box) or SaaS data sync solutions (like Resilio Connect).