<\/p>\n\n\n\n
- \n
- Reduced costs:<\/strong> Caching data locally helps you avoid costly AWS egress fees<\/a>.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Updated files:<\/strong> Distributed teams that collaborate on files can ensure that all locally stored files are up to date and avoid redundancies.<\/li>\n<\/ul>\n\n\n\n<\/div>\n\n\n\n
The complexity and scale of data storage are growing faster than IT budgets and resources, especially for enterprise teams supporting multiple offices and hybrid workforces. <\/p>\n\n\n\n
Using the RefreshCache API<\/a> can be a cumbersome process that involves managing cron jobs, scripts, AWS credentials, or AWS Lambda functions. Plus, determining which directories need to be refreshed is difficult. Luckily, AWS file gateway recently released a new cache refresh feature that automates the process and makes cache management much easier.<\/p>\n\n\n\n
In this article, we\u2019ll describe how the new file gateway cache refresh feature works and how to set up an automated cache refresh process in the file gateway.<\/p>\n\n\n\n
We\u2019ll also discuss some of the challenges that persist with AWS File Gateway cache management (and File Gateways as a whole) \u2014 such as the fact that it only allows you to cache your most recently accessed files and can only be used to access files stored in the AWS cloud.<\/strong><\/p>\n\n\n\n
In the second part of this article, we\u2019ll go over our own file synchronization and object storage gateway solution \u2014 Resilio Platform \u2014 and how it makes caching, synchronizing, and accessing files easier and more efficient.<\/p>\n\n\n\n
Want to learn more about how Resilio provides fast, efficient access to all of your on-premises and cloud files? <\/em>Schedule a demo with our team.<\/em><\/a><\/p>\n\n\n\n
Resilio Platform is a high-performance file replication and synchronization system. One popular use case for Resilio is a file gateway<\/a> for object storage. Resilio Platform can mount any type of storage \u2014 file, object, or block storage device. As a file gateway for object storage, Resilio presents objects as files \u2014 and enables rapid replication and efficient caching across multiple storage buckets. <\/p>\n\n\n\n
In scenarios where you\u2019re replicating object storage across regions or multiple sites, Resilio enables high-performance low latency real-time and on-demand replication. Resilio enables you to keep hundreds of millions of objects and files current across sites on-demand and in real-time, across any distance. Its policy-driven approach allows granular control over cache management and data movement. <\/strong><\/p>\n\n\n\n
Unlike AWS File Gateway, Resilio is designed to be:<\/p>\n\n\n\n
- \n
- Flexible: <\/strong>Resilio works by mounting agents directly on storage hardware or virtual machines. You can install it directly on your existing IT infrastructure and use it to access all files stored on-prem or in any cloud from one location.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Efficient:<\/strong> Resilio has features that enable you to optimize cloud costs, such as selective sync and caching (in addition to caching recently accessed files, you can cache any files you want). And since you can use Resilio to access all files from one solution, you don\u2019t need to invest in multiple storage gateways.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Reliability:<\/strong> Resilio uses a peer-to-peer replication architecture, which eliminates single points of failure and ensures files are always synchronized as quickly as possible. And its proprietary WAN acceleration technology enables it to fully utilize any network (including VSATs, cell, Wi-Fi, and any IP connection) to optimize data transfer speeds.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Automated: <\/strong>In Resilio, any UI action \u2014 including caching, hydration, and synchronization \u2014 can be automated using the API.\u00a0 You can also integrate with 3rd-party systems and trigger hydration or 3rd-party actions based on the cache\u2019s state.<\/li>\n<\/ul>\n\n\n\n<\/div>\n\n\n\n
Organizations use Resilio Platform to ingest, sync, and access data for media workflows (Turner Sports, Innovative), gaming (Wargaming, Larian Studios), remote operations (Mercedes-Benz, Buckeye Power Sales), and more. If you want to learn how Resilio Platform can help efficiently cache, sync, and access data on-prem and in the cloud, <\/em>schedule a demo with our team<\/em><\/a>.<\/em><\/p>\n\n\n\n
How to Manage AWS File Gateway Cache<\/h2>\n\n\n\n
AWS File Gateway (one of the components of AWS Storage Gateway) caches your most recently accessed files on local devices. Before the new cache refresh feature, you\u2019d have to manage this process manually utilizing an API to configure cron-jobs, scripts, AWS credentials, or AWS Lambda functions.<\/p>\n\n\n\n
Still, determining which directories needed to be refreshed was difficult. Many users would simply refresh the entire file share, creating unnecessary work for the Amazon S3 (Amazon Simple Storage Service) API and negatively impacting the performance of the file gateway.<\/p>\n\n\n\n
File Gateway\u2019s new cache refresh feature<\/a> allows you to automatically refresh the metadata cache to stay current with changes in your S3 buckets.<\/p>\n\n\n\n
How File Gateway\u2019s Cache Refresh Feature Works<\/h3>\n\n\n\n
The new cache refresh feature makes managing your cache easier and provides you with more flexibility. <\/p>\n\n\n\n
With this new feature, you can:<\/p>\n\n\n\n
- \n
- Create multiple file shares for a single Amazon S3 bucket<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Sync the local cache with an S3 bucket based on access frequency<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Limit the number of S3 buckets necessary to manage the file shares on your File Gateway<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Define multiple prefixes for a single S3 bucket<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Map a single prefix to a single gateway file share<\/li>\n<\/ul>\n\n\n\n<\/div>\n\n\n\n
The File Gateway cache refresh feature works based on the \u201cduration since last access.\u201d You set a time limit (anywhere from 5 minutes to 30 days) for when a cache is to be refreshed.<\/p>\n\n\n\n
For example, let\u2019s assume you set the time limit to 2 hours. If a user accesses a directory after that 2-hour time limit has expired, the gateway will automatically refresh the cache. But if a user accesses the directory before the 2-hour time limit completes, the gateway will treat the cache as current and not perform a refresh.<\/p>\n\n\n\n
This process ensures that your frequently accessed directories are automatically refreshed and synced with S3 buckets, while infrequently accessed directories are refreshed only when needed. This frees you up from cumbersome cache management processes and reduces unnecessary work on the gateway.<\/p>\n\n\n\n
How to Set Up an Automated Refresh for File Gateway Cache<\/h3>\n\n\n\n
To set up your automated cache refresh process:<\/p>\n\n\n\n
1. Visit your AWS Storage Console. Go to the \u201cFile Shares\u201d<\/em> tab.<\/p>\n\n\n\n
2. Choose a file share. Then click the \u201cActions\u201d<\/em> drop-down menu, and click \u201cEdit share settings\u201d<\/em>.<\/p>\n\n\n
\n![\"AWS](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/12\/aws-storage-gateway-file-share-settings-1024x588.png\")
<\/figure>\n<\/div>\n\n\n3. A window will pop up that allows you to add values for your automated cache refresh.<\/p>\n\n\n
\n![\"Edit](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/12\/aws-automated-cache-refresh-from-s3-after-timeframe-1024x706.png\")
<\/figure>\n<\/div>\n\n\n<\/div>\n\n\n\nResilio Connect\u2019s Fast, Efficient File Gateway Alternative<\/h2>\n\n\n\n
While the new S3 File Gateway cache feature provides more flexibility and automation for managing your AWS file cache, it still suffers from some issues that make it suboptimal for certain use cases.<\/p>\n\n\n\n
For example, it only allows you to cache recently accessed files. And it doesn\u2019t provide any additional features to increase efficiency and reduce data egress costs.<\/p>\n\n\n\n
In addition, the AWS File Gateway can only be used to access files stored in AWS. Organizations that store data in multi and hybrid cloud scenarios will need to buy and use multiple gateways to access their data.<\/p>\n\n\n\n
Our file synchronization and gateway software system, Resilio Connect, provides a superior alternative to AWS File Gateway for organizations that need to extend their on-premises applications to the cloud. <\/strong><\/p>\n\n\n\n
It offers efficient file caching and access, enabling you to transform your on-premises servers and NAS devices into flexible storage gateways. Plus, it’s focused on very large file systems and data movement at the best possible speeds with all transfers being UDP-based and WAN-optimized.<\/p>\n\n\n\n
Resilio Platform includes features that make it a superior solution for caching and syncing large datasets between any S3-compatible cloud object storage and your on-premises devices or data centers, such as:<\/p>\n\n\n\n
- \n
- Efficiency and cost-reduction<\/li>\n\n\n\n
- Versatility and granular control<\/li>\n\n\n\n
- Reliable sync and access<\/li>\n\n\n\n
- Bulletproof security<\/li>\n<\/ul>\n\n\n\n<\/div>\n\n\n\n
Efficient, Cost-Effective Caching and Access<\/h3>\n\n\n\n
Resilio Platform is designed to enhance the cost-effectiveness of storing data in the cloud. Unlike other gateways, our solution is built to reduce costs associated with data egress fees and multi-cloud storage.<\/strong><\/p>\n\n\n\n
AWS File Gateway only provides access to data stored in the AWS cloud. However, many organizations choose to store data in multiple clouds and on-premises storage solutions in order to increase redundancy and maintain high availability. To access their data, the businesses would need to use multiple different cloud gateways for each cloud platform.<\/p>\n\n\n\n
But Resilio Platform is cloud vendor agnostic. You can use it with just about any cloud storage platform, such as AWS, GCP, Azure, Wasabi, MinIO, and more. You can deploy Resilio on your existing infrastructure at low cost, and manage and access all of your cloud and on-prem data from one location.<\/p>\n\n\n\n
Resilio\u2019s built-in security features mean that you won\u2019t have to invest in 3rd-party security solutions or VPNs, as is the case with other gateway solutions.<\/p>\n\n\n\n
And Resilio includes features that help you enhance productivity and reduce costs associated with data egress, such as:<\/p>\n\n\n\n
- \n
- Selective caching:<\/strong> You can choose which files are stored on local devices, providing employees with faster access to the data they need and reducing data egress fees associated with downloading files.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Selective sync:<\/strong> You can choose which specific files and folders sync to which endpoints. This ensures that files only sync to the destinations where they\u2019re needed so you can reduce data transfer fees.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Partial downloads:<\/strong> You can fully or partially download files and folders, so you can get quicker access to the files you need and reduce data egress costs.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Automation:<\/strong> You can automatically sync, cache, download, and purge any file based on the policies you set.<\/li>\n<\/ul>\n\n\n\n<\/div>\n\n\n\n
![\"Operations](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/06\/operations-team-resilio-connect-management-console.png\")
<\/figure>\n<\/div>\n\n\n<\/div>\n\n\n\nVersatility and Granular Control Over Your Environment<\/h3>\n\n\n\n
File Gateway works by installing AWS agents on virtual machines on your hardware. And it only provides access to files stored in the AWS cloud. <\/p>\n\n\n\n
But Resilio Platform is a much more versatile solution. It provides low-latency access to files via SMB (Server Message Block) and NFS (Network File System) protocols.<\/p>\n\n\n\n
You can deploy Resilio agents directly onto your file servers, NAS\/DAS\/SAN systems, desktops, laptops, mobile devices (Resilio offers iOS and Android apps), and virtual machines (such as VMware, Citrix, hypervisors, and more).<\/p>\n\n\n\n
Resilio supports just about any:<\/p>\n\n\n\n
- \n
- Operating system:<\/strong> Microsoft Windows, macOS, Linux, Unix, FreeBSD, OpenBSD, Ubuntu, and more.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Cloud storage platform: <\/strong>AWS, Google Cloud Platform, Azure, Wasabi, MinIO, Backblaze, and more.<\/li>\n<\/ul>\n\n\n\n<\/div>\n\n\n\n
![\"Resilio](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/08\/image3-1024x776.png\")
<\/figure>\n<\/div>\n\n\n<\/div>\n\n\n\nResilio is easy to deploy and enables you to sync and access files across your entire on-prem and cloud environment from a single location, saving you from having to invest in and manage multiple gateways.<\/p>\n\n\n\n
Plus, Resilio\u2019s Management Console provides you with granular control over replication in your environment. <\/strong>You can:<\/p>\n\n\n\n
- \n
- Create bandwidth profiles for each endpoint that govern how much bandwidth is allocated to each endpoint at certain times of the day and on certain days of the week.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Manage every server in your environment, and manage up to 50,000 agents per console.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Get insight into the status of individual replication jobs, with real-time performance metrics that can get sent to email or Webhooks<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Review a history of all executed jobs<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Establish job priorities<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Automate replication jobs and script any type of functionality using Resilio\u2019s REST API.<\/li>\n<\/ul>\n\n\n\n<\/div>\n\n\n\n
All end-users are provided with the same view of files from a unified interface that operates much like Microsoft OneDrive.<\/p>\n\n\n
\n![\"How](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/04\/always-keep-on-this-device-option-761x1024.png\")
<\/figure>\n<\/div>\n\n\n<\/div>\n\n\n\nCase Study: Deutsche Aircraft<\/h4>\n\n\n\n
Commercial aircraft manufacturer Deutsche Aircraft switched from DFSR to Resilio Platform to sync their Microsoft DFS namespace, manage data access, secure mission-critical data, and increase efficiency.<\/p>\n\n\n
\n![\"Deutsche](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/09\/Deutsche-aircraft-1.jpg\")
<\/figure>\n<\/div>\n\n\n<\/div>\n\n\n\n\u201cWe have a 10Gbps network but prefer to use under 1Gbps for data transfer and replication. With Resilio, we\u2019re able to keep that down to 250Mbps during the day and at night move back up to 1GBps\u2026 Resilio Platform is much easier to manage than DFSR. Using the Resilio Platform management console, you can see everything you need to know. Everything is visible.\u201d<\/em><\/strong><\/p>\n\n\n\n
Reliable File Synchronization and Access<\/h3>\n\n\n\n
Keeping data synchronized and up to date is one of the top concerns for businesses and IT teams.<\/p>\n\n\n\n
AWS File Gateway synchronizes data using an unreliable point-to-point replication architecture that can be configured in one of two models:<\/p>\n\n\n\n
- \n
- Hub and spoke:<\/strong> This model consists of a hub server and several remote servers. The remote servers can\u2019t share files directly with each other. Instead, they must first transfer files to the hub server, which then syncs those files with each remote server one by one.<\/li>\n<\/ul>\n\n\n\n
<\/p>\n\n\n\n
- \n
- Follow the sun:<\/strong> In this model, replication occurs from one server to another sequentially.<\/li>\n<\/ul>\n\n\n\n<\/div>\n\n\n\n
Both of these replication architectures are as slow as their weakest endpoint \u2014 i.e., if replication fails or is delayed on one endpoint, it can delay synchronization for every other endpoint in your environment.<\/p>\n\n\n\n
They also introduce single points of failure. If any endpoint goes down, it can delay sync for your entire environment. And if the hub server goes down in a hub and spoke architecture, then replication fails entirely.<\/p>\n\n\n\n
Resilio Platform uses a blazing fast and highly reliable P2P (peer-to-peer) replication architecture that eliminates the challenges experienced by point-to-point systems.<\/p>\n\n\n\n
In a P2P architecture, every server can communicate with every other server and take part in replication simultaneously, resulting in:<\/p>\n\n\n\n
Faster Sync Speeds<\/h4>\n\n\n\n
Resilio uses a process known as file chunking<\/strong> to break files down into multiple chunks that can transfer independently of each other. Every server can share file chunks simultaneously.<\/p>\n\n\n\n
For example, imagine you want to sync a file across five servers. Resilio can use file chunking to break that file down into five chunks. Server 1 can share the first file chunk with Server 2. While it\u2019s waiting to receive the other four chunks, Server 2 can immediately share the first chunk with one of the other servers. Soon every server will be sharing chunks concurrently, allowing you to sync your system 3\u201310x faster than with AWS file gateway or other point-to-point solutions.<\/strong><\/p>\n\n\n
\n![\"P2P](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2022\/07\/p2p-vs-client-server.gif\")
<\/figure>\n<\/div>\n\n\n<\/div>\n\n\n\nAs your environment grows, point-to-point solutions take longer to sync, as replication occurs from just one server to another.\u00a0<\/p>\n\n\n\n
But Resilio\u2019s P2P architecture allows it to scale organically. More endpoints create more supply (i.e., bandwidth, CPU, etc.) and sync speeds increase as your environment grows. Resilio performs horizontal scale-out replication, allowing it to reach speeds of 100+ Gbps per server. <\/strong><\/p>\n\n\n\n
In fact, our engineers <\/em>successfully transferred a 1 TB dataset across Azure regions in 90 seconds<\/em><\/a>.<\/em><\/p>\n\n\n\n
Case Study: VoiceBase<\/h4>\n\n\n\n
VoiceBase \u2014 a software that provides speech-to-text transcriptions for video and audio \u2014 uses Resilio Platform to distribute speech model updates (50+ GB files) across 400+ production servers each month.<\/p>\n\n\n\n
\u201c<\/strong>Resilio Platform enables us to reliably distribute our code, specifically new language models in a fraction of the time. These copy jobs now take an hour, down from eight. Best of all, once Resilio Platform was installed, it just works: We never need to manually intervene in any way.<\/em><\/strong>\u201d<\/strong><\/p>\n\n\n\n
- Follow the sun:<\/strong> In this model, replication occurs from one server to another sequentially.<\/li>\n<\/ul>\n\n\n\n
- Hub and spoke:<\/strong> This model consists of a hub server and several remote servers. The remote servers can\u2019t share files directly with each other. Instead, they must first transfer files to the hub server, which then syncs those files with each remote server one by one.<\/li>\n<\/ul>\n\n\n\n
- Automate replication jobs and script any type of functionality using Resilio\u2019s REST API.<\/li>\n<\/ul>\n\n\n\n
- Establish job priorities<\/li>\n<\/ul>\n\n\n\n
- Review a history of all executed jobs<\/li>\n<\/ul>\n\n\n\n
- Get insight into the status of individual replication jobs, with real-time performance metrics that can get sent to email or Webhooks<\/li>\n<\/ul>\n\n\n\n
- Manage every server in your environment, and manage up to 50,000 agents per console.<\/li>\n<\/ul>\n\n\n\n
- Create bandwidth profiles for each endpoint that govern how much bandwidth is allocated to each endpoint at certain times of the day and on certain days of the week.<\/li>\n<\/ul>\n\n\n\n
- Cloud storage platform: <\/strong>AWS, Google Cloud Platform, Azure, Wasabi, MinIO, Backblaze, and more.<\/li>\n<\/ul>\n\n\n\n
- Operating system:<\/strong> Microsoft Windows, macOS, Linux, Unix, FreeBSD, OpenBSD, Ubuntu, and more.<\/li>\n<\/ul>\n\n\n\n
- Automation:<\/strong> You can automatically sync, cache, download, and purge any file based on the policies you set.<\/li>\n<\/ul>\n\n\n\n
- Partial downloads:<\/strong> You can fully or partially download files and folders, so you can get quicker access to the files you need and reduce data egress costs.<\/li>\n<\/ul>\n\n\n\n
- Selective sync:<\/strong> You can choose which specific files and folders sync to which endpoints. This ensures that files only sync to the destinations where they\u2019re needed so you can reduce data transfer fees.<\/li>\n<\/ul>\n\n\n\n
- Selective caching:<\/strong> You can choose which files are stored on local devices, providing employees with faster access to the data they need and reducing data egress fees associated with downloading files.<\/li>\n<\/ul>\n\n\n\n
- Map a single prefix to a single gateway file share<\/li>\n<\/ul>\n\n\n\n
- Define multiple prefixes for a single S3 bucket<\/li>\n<\/ul>\n\n\n\n
- Limit the number of S3 buckets necessary to manage the file shares on your File Gateway<\/li>\n<\/ul>\n\n\n\n
- Sync the local cache with an S3 bucket based on access frequency<\/li>\n<\/ul>\n\n\n\n
- Create multiple file shares for a single Amazon S3 bucket<\/li>\n<\/ul>\n\n\n\n
- Automated: <\/strong>In Resilio, any UI action \u2014 including caching, hydration, and synchronization \u2014 can be automated using the API.\u00a0 You can also integrate with 3rd-party systems and trigger hydration or 3rd-party actions based on the cache\u2019s state.<\/li>\n<\/ul>\n\n\n\n
- Reliability:<\/strong> Resilio uses a peer-to-peer replication architecture, which eliminates single points of failure and ensures files are always synchronized as quickly as possible. And its proprietary WAN acceleration technology enables it to fully utilize any network (including VSATs, cell, Wi-Fi, and any IP connection) to optimize data transfer speeds.<\/li>\n<\/ul>\n\n\n\n
- Efficient:<\/strong> Resilio has features that enable you to optimize cloud costs, such as selective sync and caching (in addition to caching recently accessed files, you can cache any files you want). And since you can use Resilio to access all files from one solution, you don\u2019t need to invest in multiple storage gateways.<\/li>\n<\/ul>\n\n\n\n
- Flexible: <\/strong>Resilio works by mounting agents directly on storage hardware or virtual machines. You can install it directly on your existing IT infrastructure and use it to access all files stored on-prem or in any cloud from one location.<\/li>\n<\/ul>\n\n\n\n
- Updated files:<\/strong> Distributed teams that collaborate on files can ensure that all locally stored files are up to date and avoid redundancies.<\/li>\n<\/ul>\n\n\n\n
![\"Hot\/Live](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/04\/hot-dr-warm-dr-cold-dr-and-offsite-copy-699x1024.png\")
<\/figure>\n<\/div>\n\n\n![\"Cross](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/08\/cross-region-replication-Azure-3-1024x668.png\")
<\/figure>\n<\/div>\n\n\n![\"Mutual](\"https:\/\/blog.resilio.com\/wp-content\/uploads\/2023\/04\/mutual-authentication-in-transit-encryption-integrity-validation-process.png\")
<\/figure>\n<\/div>\n\n\n