Skip to content

Optimizing agent for large amount of small files delivery

By default Resilio Agent is optimized for a mixed dataset delivery: few millions of files with average size of ~5 megabytes.

This guide provides strategies to efficiently manage and optimize use cases involving millions of small files, each averaging just a few kilobytes. Here are optimization steps suggested:

  1. In the Agent Profile:

    1. Set Number of disk I/O threads to 12 (default value is 4). This will help an Agent to access more files at the same time.
    2. Add the fs_prefer_sequential_indexing custom parameter with valuefalse (default true). This allows Agent to process multiple small files in parallel without waiting previous files indexing to complete.
    3. Set the peer_max_active_downloads custom parameter to 2000 (default 500). This allows Agent to increase safety net of parallel files delivery from 500 to 2000 per connection to another Agent.
    4. Set the max_active_downloads custom parameter to 2000 (default 1000). This increases global Agent-scale limit of simultaneous files delivery
    5. (Applicable to cloud storage) Set the net.http.threads custom parameter to 4 ... 8 (default 0 which stands for reusing existing TCP threads). This allows agent to use multiple independent http threads when pushing data to the cloud.
    6. (Applicable to Windows endpoints) Set the Asynchronous disk I/O to Yes (default No).
      This allows OS to queue disk I/O requests on a very low level, resulting in faster processing and spending less I/O operations on every file.

  2. Restart the Agents that have this updated profile assigned.