Google Cloud Storage FUSE (GCSFuse): Use Cases, Limitations & Alternative

GCSFuse — also known as Google Cloud Storage FUSE — is an open-source Fuse adapter for mounting and accessing cloud storage buckets (using Google Cloud Storage) as local file systems. It lets you read and write objects in your bucket using standard file system semantics.

While users find it useful in some scenarios, GCSFuse is only suitable for limited use cases. 

It gives users an easy way to store files as objects in Google cloud storage but it’s not reliable, performs poorly, and is not POSIX compliant. It’s insufficient for advanced, high-performance file sharing or cloud storage gateway scenarios. It’s also limited to Google Cloud, so you can’t rely on it if your infrastructure spans across multiple cloud providers and on-prem storage. 

In this article, we’ll describe what GCSFuse is, what its applicable use cases are, and how to set it up.

We’ll also discuss how our file synchronization software system and object storage gateway, Resilio Connect, gives you efficient, low-latency access to data stored in any cloud, on-premise, or hybrid cloud infrastructure from any location. Resilio overcomes the limitations of GCSFuse and many other object storage gateways because it:

• Provides access to files in on-premise storage (direct-attached, SAN, or NAS) and any S-3 compatible cloud object storage — such as GCP, AWS S3, Azure Blobs, Backblaze, and more — allowing you to turn on-prem devices into flexible storage gateways.

• Can be managed with a graphic user interface as well as command-line code. End-users can also browse and access files from anywhere as easily as they would in the office through a unified interface that operates much like Microsoft OneDrive.

• Scales organically to support environments of any size and caches files locally at each endpoint for faster access.

• Delivers efficient caching and other features that enable you to enhance productivity while minimizing cloud egress costs.

• Uses a P2P (peer-to-peer) replication architecture that allows it to quickly sync files across your entire environment, providing low-latency access to files for applications and remote workers.

• Employs a proprietary WAN acceleration protocol that optimizes transfers over any network connection, enabling it to quickly sync files over long-distance WANs, low-grade consumer networks, and edge networks.

• Sync files in real-time in any direction (which makes file locking unnecessary).

• Comes with features that provide bulletproof reliability during data transfers and syncs, such as checkpoint restarts and dynamic rerouting.

Organizations in gaming, media, logistics, retail, construction, and more use Resilio Connect to sync and access files for remote collaboration among distributed teams. To learn more about how Resilio can help your organization quickly, securely, and efficiently access files, schedule a demo with our team.

What Is GCSFuse and What Is It Used For?

GCSFuse uses FUSE and Cloud Storage APIs to present cloud storage buckets as local folders on your file system (by translating object storage names into a file and file directory on your system).

Put simply, it’s an open-source gateway that makes files stored in the cloud browseable and accessible on your local devices. Users and applications can interact with locally mounted buckets as they would with any local file system.

You can run GCSFuse from anywhere that’s connected to cloud storage, such as on-premise devices, Google Kubernetes Engine, or VMs.

GCSFuse Use Cases and Limits

GCSFuse provides a file system interface for objects stored in the cloud. However, it differs from NFS and CIFS file systems. If you’re thinking about using GCSFuse, you should be aware of its limitations, such as:

• It only works with the Google Cloud Platform.

• It is not POSIX compliant.

• When uploading files to the cloud, it doesn’t transfer object metadata (except for symlink and mtime targets).

• It doesn’t support file locking, so you shouldn’t store version control system repositories in GCSFuse mount points.

• The semantics for GCSFuse are different from the semantics of traditional file systems.

• It doesn’t provide concurrency control, version control, or merging. When multiple edits are made to the same file, that last edit wins, and all other edits are lost.

• It can be very slow and has a much higher latency than traditional file systems.

• It can be unreliable — i.e., it provides no caching capabilities, performs poorly, and doesn’t scale well.

The high latency of GCSFuse is particularly important, as it severely limits the suitable use cases. When reading or writing individual small files at a time, throughput may be reduced. Because of this, GCSFuse shouldn’t be used as the backend for storing a database.

However, there are some use cases where GCSFuse will suffice, such as:

• Situations where using a file system interface is easier than using an HTTP API through a cloud storage library. For example, data processing, reporting, general file administration, and batch/cron jobs.

• Applications where fetching files from cloud storage every time you need to work on them is untenable. Mounting cloud buckets to a VM with GCSFuse may be easier. For example, when periodically analyzing unstructured data.

• If you want to read from and write to cloud storage within your Kubernetes pods, you can use GCSFuse with the Google Kubernetes Engine API to consume buckets as volumes.

GCSFuse Pricing

GCSFuse is a free tool. However, you will be charged for all data transfers and operations performed by GCSFuse in your Google Cloud Storage. So be sure to calculate Google Cloud transfer charges and estimate how your use of GCSFuse will translate.

How to Set Up GCSFuse

The following tutorials will show you how to deploy GCSFuse, mount GCS buckets, and upload objects into buckets.

Deploying GCSFuse

You can install GCSFuse by taking the following steps via your local shell on Debian or Ubuntu:

1. Add the GCSFuse distribution URL as a package source (which you can find on github).

export GCSFUSE_REPO=gcsfuse-`lsb_release -c -s`

echo "deb https://packages.cloud.google.com/apt $GCSFUSE_REPO main" | sudo tee /etc/apt/sources.list.d/gcsfuse.list

2. Import the public key for the Google Cloud APT repository.

curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

3. Update the list of available packages.

sudo apt-get update

4. Install GPSFuse.

sudo apt-get install fuse gcsfuse

You can confirm the installation by using:

gcsfuse -v

If you successfully installed GCSFuse, you should see the following output:

gcsfuse version 0.41.12 (Go version go1.18.4)

Mounting a Bucket

Use the following steps to mount a Google Cloud Storage bucket to your local file system:

1. Use the following command to generate Application Default Credentials:

gcloud auth application-default login

2. Create a directory to mount the storage bucket to:

mkdir "$HOME/folder-name"

3. Use this GCSFuse command to mount your storage bucket:

Gcsfuse Bucket_Name "$HOME/folder-name"

GCSFuse will return something similar to the following output if the mount was successful:

2024/01/05 16:01:23.434521 Opening GCS connection...

2024/01/05 16:01:23.732467 Mounting file system "my-bucket"...

2024/01/05 16:01:23.235881 File system has been successfully mounted.

Uploading Objects into Buckets

1. Use the cp command to copy the file from its saved location to your mounted bucket folder.

cp kitten.png "$HOME/folder-name/file-name"

2. Use the ls command on the mounted folder to verify that the object was copied to your local file system.

ls "$HOME/folder-name"

GCSFuse will return the name of the object if the copy was successful.

3. You can also use the following command to verify that the file is in your Google Cloud bucket:

gcloud storage ls gs://Bucket_Name

It will return the name of the object if it was successfully uploaded into the bucket.

Using Resilio Connect for Fast, Efficient Access to Cloud Objects

While GCSFuse provides a free file gateway for objects stored in Google Cloud Platform that’s helpful in limited use cases, Resilio Connect provides a turnkey object storage gateway that’s far more reliable, efficient, and flexible.

Resilio Connect is a real-time file synchronization software system that also works as an efficient object storage gateway. It provides efficient, low-latency access to data stored in any file, block, or object storage (including on-premises devices and any S3-compatible cloud storage platform).

In this section, we’ll discuss the features and use cases of Resilio’s file gateway and why it’s a superior alternative to GCSFuse.

Workflow Friendly Integration and Unified Data Management

Resilio Connect is a vendor-agnostic, workflow-friendly solution that:

• Seamlessly integrates into your current IT infrastructure and file-based workflows/applications, while optimizing cloud storage costs.

• Enables you to unify data management from one vendor-agnostic platform.

Easily Install into Your Current IT Infrastructure and File-Based Workflows

Resilio Connect is an agent-based solution that’s built on universal standards and open protocols. As such, there’s no need to invest in any new hardware or software. It also doesn’t store files in any proprietary formats, so you always maintain control over your data.

You can install Resilio agents onto just about any:

• Device, including desktops, laptops, file servers, NAS/DAS/SAN devices, mobile devices (Resilio offers iOS and Android apps), IoT devices, and virtual machines (such as Citrix, VMware, and Microsoft Hyper-V).

• Operating system, including Windows, MacOS, Linux, Unix, Ubuntu, FreeBSD, OpenBSD, and more.

• Cloud storage service, including any S3-compatible cloud object storage platform, such as Google Cloud Storage, AWS S3, Azure Blobs, Backblaze, MinIO, and more.

You can also use Resilio’s powerful REST API to automate your workflows and integrate it with popular tools that your team is already using, such as:

• Media tools and creative software: Adobe Premiere, Media Composer, Avid Pro Tools, and more.

• Management tools: Splunk, LCE, Microsoft SCOM, and more.

• Development tools: Jenkins, TeamCity, and more.

Because of Resilio’s flexibility, you can quickly and easily deploy it on your current IT infrastructure with minimal operational interruption and continue using the storage and tools you’re already using without having to invest in new hardware or platforms. 

For example, Delirio Films (a documentary production company) uses Resilio with their Mac and Windows desktops, various storage systems, such as Facilis Terrablock and OWC Thunderbay, and editing software, such as DaVinci Resolve, Avid Media Composer, and Adobe Premiere.

Unify Data Management from One Vendor-Agnostic Platform

Resilio provides a centralized Management Console you can use to obtain granular control and insight into every aspect of replication and data access. 

From the Management Console, you can:

• Create and manage replication jobs.

• Manage S3 objects and buckets.

• Monitor all Resilio agents, endpoints, and job functions.

• Adjust replication parameters, such as data hashing, buffer size, disk I/O, and more.

• Review a history of all executed jobs.

• Adjust bandwidth at each endpoint and even create profiles that govern how much bandwidth is allocated to each endpoint at certain times of the day and certain days of the week.

• Collect logs and configure notifications to be delivered to your email or Webhooks.

• Use Resilio’s REST API to script any type of function and automation your workflow requires.

End-users can also browse and access files from a user-friendly interface that operates much like Microsoft OneDrive, providing everyone in your organization with a unified view of files.

Resilio’s flexibility and workflow-friendly integration reduces costs and complexity by allowing you to unify data management. No matter what storage (on-premise, cloud, hybrid cloud, or multi-cloud) or tools you use, you can manage data replication and access across your entire environment from one unified solution.

Efficient File Gateway for Remote Teams and Applications

Resilio’s file gateway is designed to operate efficiently. You can store frequently accessed files on your local devices while storing infrequently accessed files in long-term cloud storage. 

Resilio’s features and capabilities also allow you to obtain low-latency access to files in office or remotely, while helping you increase productivity and minimize costs associated with cloud storage and data egress.

Achieve Low-Latency Access and Real-Time Sync

Resilio’s P2P replication architecture (which we’ll discuss in more detail later) allows you to achieve low-latency access to files at each endpoint. 

While end-users can remotely browse and download files via SMB (Server Message Block) and NFS (Network File System) protocols, Resilio also delivers global file access directly on each local endpoint — making it the ideal solution for fast-paced, high-performance workflows involving large datasets and enterprise applications.

As stated earlier, Resilio doesn’t support file locking. However, Resilio’s real-time, multidirectional synchronization enables fast collaboration across multiple geographically distributed sites. 

Resilio uses notification events from the host OS and optimized checksum calculations (identification markers assigned to each file that change when a change is made to the file) to detect and replicate file changes in real-time. And it can sync files in any direction, such as one-way, two-way, one-to-many, many-to-one, and N-way.

The combination of real-time and N-way sync is key to Resilio’s ability to facilitate remote collaboration. If any employee at any location makes a change to a file, that change is immediately synchronized to every other office and remote endpoint — so everyone always has access to the same versions of files.

When switching to a remote work model during the COVID-19 pandemic, Skywalker Sound used Resilio’s sync capabilities to sync files across their geographically distributed team for fast, real-time collaboration. They even used Resilio’s bandwidth allocation capabilities (more on this later) to overcome the differences in network speed and quality for each remote employee — ensuring that files never fell out of sync.

Enhance Productivity and Reduce Data Transfer Costs through Efficient Sync, Caching, and Downloading

Unlike GCSFuse, Resilio can cache files on local devices. It also gives you more control over how files are cached, synchronized, and downloaded via:

• Selective caching: Most file gateways only cache frequently accessed files. But Resilio allows you to cache frequently accessed files as well as any files you choose, so you can provide end-users with faster access to the files they need while reducing data egress costs.

• Transparent Selective Sync: Files can be automatically pushed in real-time from person to person or from one person to many. And files can also be pulled from the cloud on-demand through a feature known as Transparent Selective Sync. With TSS, you have full control over how files are synchronized, including which files sync to which specific endpoints. 

• Full or partial downloads: Employees can download entire files and folders or just the portions of files that they need. This gives employees faster access to the data they need while reducing data egress costs.

• Policy-based automation: You can eliminate manual processes and free employees up to focus on their tasks by creating policies that automate how files are synced, cached, downloaded, and purged from devices.

High-Performance Synchronization over Any Network

Resilio’s P2P sync architecture and proprietary WAN acceleration protocol enable it to:

• Sync files across your environment 3-10x faster than traditional solutions.

• Scale organically to support files and environments of any size.

• Sync reliably over any network.

Quickly Sync Files across Your Environment

Traditional replication solutions use inefficient hub-and-spoke architectures. A hub-and-spoke topology consists of a hub server and multiple remote servers. The remote servers can’t share files directly with each other. Instead, files must first be sent to the hub server, which then syncs the files with each remote server one by one.

This replication methodology is slow, since file transfers can only occur between two servers at a time. And if any of your endpoints are on a slow network, it can delay synchronization for the other endpoints in your environment.

But Resilio uses a P2P replication architecture, where every endpoint can share files directly with every other endpoint. And all endpoints can take part in file synchronization simultaneously.

Resilio also uses a process known as file chunking, where files are split into multiple chunks that can transfer independently of each other. So if you wanted to sync a 10 GB file across five endpoints, Resilio can split that file into five chunks. Endpoint 1 can share the first chunk with Endpoint 2. Endpoint 2 can immediately share that chunk with Endpoint 3, even before it receives the remaining file chunks. Soon every endpoint in your environment will be sharing file chunks simultaneously, allowing you to sync your environment 3-10x faster than hub-and-spoke solutions.

Scale Organically to Support Environments and Files of Any Size

Resilio can sync files of any size, type, or number. In fact, our engineers successfully synchronized 450+ million files in a single job.

Resilio’s P2P architecture also allows you to organically scale your environment. Since every endpoint can take part in synchronization, then adding more endpoints increases the available resources (CPU, bandwidth, etc.). So sync speed increases as your environment grows. Resilio can sync 200 endpoints in roughly the same time it takes most hub-and-spoke solutions to sync just two.

You can try our transfer speed calculator to see how much time we can save you.

Resilio can also perform a process known as horizontal scale out replication. This allows you to cluster Resilio agents together in order to pool resources and linearly increase sync speeds. Each Resilio agent can reach speeds of 10 Gbps. Our engineers used scale-out replication to cluster 10 agents and reach speeds of 100 Gbps (though you can create even larger clusters and there’s no limit to how fast it can go).

Reliably Sync over Any Network

Most file sync solutions don’t contain features to enhance file transfers over WANs (wide area networks). WANs typically suffer from high latency and varying degrees of packet loss. And the transfer protocols that most sync solutions use (such as TCP/IP) can’t sync over these networks quickly and reliably.

While Resilio can use TCP/IP to sync over LANs, it also uses a proprietary UDP-based WAN acceleration protocol known as Zero Gravity Transport™ (ZGT).

ZGT intelligently analyzes the underlying conditions of a network (such as packet loss, latency, and throughput over time) and automatically adjusts to those conditions in order to maintain a consistent rate of transfer over time.

To do this, ZGT uses:

• A congestion control algorithm: ZGT uses a congestion control algorithm that constantly probes the Round Trip Time of any network to identify and maintain the ideal data packet send rate.

• Interval acknowledgments: UDP protocols aren’t required to send acknowledgments for each received packet, allowing them to transmit data packets faster. ZGT sends acknowledgments for groups of packets once per RTT to reduce network traffic.

• Delayed retransmission: ZGT retransmits lost packets once per RTT in order to decrease unnecessary retransmissions.

Because ZGT efficiently transfers over any network, Resilio can sync files over any type of network connection, such as VSAT, cell (3G, 4G, and 5G), Wi-Fi, and any IP connection. Resilio can even optimize transfers of intermittent edge networks and in remote locations with little network connectivity. For example, Northern Marine Group uses Resilio to sync data to its vessels at sea. And Shifo uses Resilio to sync healthcare data across their offices in countries with underdeveloped network infrastructure, such as Uganda.

This means that, no matter where your employees or offices are located, you can be sure that your data will sync across all endpoints quickly, reliably, and in predictable time frames.

Resilio also contains other features that enhance reliability, such as:

• Automatic retries and checksum restarts: Resilio automatically retries all failed transfers until they’re complete. If a transfer is interrupted, Resilio will perform a checksum restart to automatically resume the transfer at the point of failure.

• Dynamic rerouting: If any endpoint goes down, Resilio will dynamically reroute around the outage to ensure your files reach their destination.

• Offline access and local sync: Resilio enables users to access and work on files offline, so you can continue to work even when using networks that cut in and out. While offline, you can sync locally across LANs. When the connection resumes, Resilio will automatically sync file changes across your entire environment.

Our marine construction client uses this Resilio feature to deploy updates to their ships at sea. First, they transfer the updates from HQ to the ship’s central server. Then the server syncs the updates to each workstation on board the ship over the LAN.

Bulletproof Native Security Features

Many file gateways, such as GCSFuse, don’t include native security features that protect your data. This forces you to invest in 3rd party security solutions and VPNs.

Data security is especially important in remote work scenarios where some employees may be working over unsecured consumer-grade networks (such as LTE/3G, ADSL, or Wi-Fi). Not only are these networks vulnerable, but accessing data via a VPN over consumer-grade networks is slow.

Resilio Connect overcomes both of these issues by providing fast synchronization and access to data at each endpoint and built-in security features that protect your data end-to-end.

Resilio provides TPN Blue-certified and SOC2-certified security features, such as:

• End-to-end encryption: Data is encrypted at rest and in transit using AES 256-bit encryption.

• Mutual authentication: Before any endpoint can receive files, it must provide a security key — ensuring your data is only delivered to approved endpoints.

• Cryptographic data integrity validation: Resilio ensures data arrives at its destination intact and uncorrupted using cryptographic integrity validation. If an agent receives a file that’s incomplete or corrupted, the file is immediately discarded and scheduled for retransmission.

• Forward secrecy: Resilio uses one-time session encryption keys to protect each session.

• Permission controls: Resilio provides granular control over who can access specific files and folders.

• Proxy server: Resilio’s Proxy Server capabilities make it easier to support situations where remote employees have custom networking setups (using NAT and other firewalling technologies). You can also use it to block specific computers from accessing or running scripts on your devices/computers when working with remote employees, 3rd-party contractors, and advanced firewall configurations.

Use Resilio Connect for Secure, Efficient File Sync and Access

Resilio Connect is an ideal alternative to GCSFuse that offers fast, efficient, and reliable access to data stored anywhere — in the cloud or on premises. Our solution:

• Easily integrates with the existing infrastructure, workflow, and tools your team is already using.

• Provides low-latency access to files from any remote location.

• Contains features that allow you to enhance productivity and minimize cloud storage costs, such as selective sync and caching.

• Syncs files in real-time using a blazing-fast, multidirectional, and organically scalable P2P architecture.

• Uses a proprietary WAN acceleration protocol to quickly and reliably sync over any network.

• Contains additional features that enhance reliability, such as automatic retries, checksum restarts, offline access, and dynamic rerouting.

• Includes built-in security features that protect your data end-to-end.

Organizations in gaming, media, logistics, retail, construction, and more use Resilio Connect to sync and access files for remote collaboration among distributed teams. To learn more about how Resilio can help your organization quickly, securely, and efficiently access files, schedule a demo with our team.