Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Comment: Add info on performance tuning

...

Code Block
languagebash
$ rclone copy -P amazon_store:rclone-tutorial ~/test-download
Transferred:   	      192M / 192 MBytes, 100%, 77.531 MBytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            2 / 2, 100%
Elapsed time:        2.4s
$ ls ~/test-download/
test1.file  test2.file


Tuning rclone performance

By default, rclone is not optimized for our infrastructure. Increasing the maximum number of parallel transfers and the chunk size can increase transfer speed. This will however take more bandwidth and RAM, so depending on which node this is run on, the results will vary. The following flags should be used:

Code Block
languagebash
--bwlimit=0 # Do not limit bandwidth
--buffer-size=128M # Buffer for each transfer
--checkers=32 # Run 32 checksum checkers in parallel
--transfers=32 # Run 32 transfers in parallel

Please be advised that the actual performance gain depends on both the source and destination system, as well as the current usage of those systems. Also, depending on the type of data transferred (many small files, or few large files?), results will vary. Use these parameters as a starting point for your individual fine tuning.

Use these parameters as follows:

Code Block
languagebash
$ rclone --bwlimit=0 --buffer-size=128M --checkers=32 --transfers=32 copy -P ~/local/source amazon_store:bucket-name

rclone browser - Graphical user interface

...