Skip to content

Optimizing Agent for large amount of small files delivery

By default, the Resilio Agent is optimized for mixed datasets: a few million files with an average size of ~5 MB.

This guide provides strategies to efficiently manage and optimize use cases involving millions of small files, each averaging just a few KB.

Individual Agents or Agents in non-scaleout groups

To optimize performance of individual Agents or non-scaleout groups:

  1. In the Agent Profile:

    1. Change the following parameters:

      • Number of disk I/O threads: 12 (default: 4)
        This helps an Agent access more files concurrently.

      • Asynchronous disk I/O: Yes (default: No)
        This allows the OS to queue disk I/O requests at a low level, resulting in faster processing and fewer I/O operations per file.

        Note

        Applicable to Windows endpoints.

    2. Add the following custom parameters:

      • fs_prefer_sequential_indexing: false (default: true)
        Allows the Agent to process multiple small files in parallel without waiting for previous files' indexing to complete.
      • peer_max_active_downloads: 2000 (default: 500)
        Increases the per-peer limit of parallel file downloads from 500 to 2000.
      • max_active_downloads: 2000 (default: 1000)
        Increases the global Agent-level limit for simultaneous file downloads.

      • net.http.threads: 4 ... 8 (default: 0, which reuses existing TCP threads)
        Enables multiple independent HTTP threads when pushing data to cloud storage.

        Note

        Applicable to cloud-storage Agents.

  2. Restart the Agents that have this updated profile assigned.

Agents in scaleout cluster groups

To optimize performance of Agents running in scaleout groups:

  1. In the Agent Profile:

    1. Set the Number of disk I/O threads parameter to 8.

    2. Add the following custom parameters with these values:

      • scaleout.exclusive_dl_time_ms: 2000
      • peer_max_active_downloads: 2000
      • max_active_downloads: 2000
      • fs_prefer_sequential_indexing: false
      • peer_max_queue: 536870912
      • peer_max_seed_queue_mb: 256
      • direct_torrent_use_suffix: 0
      • max_active_uploads: 10000

  2. Restart the Agents that have this updated profile assigned.