This was was damn confusing, but the solution is absurdly obvious. I needed to relocate a large number of files from a single source into a collection of subfolders. (These subfolders were essentially worker queues, so I wanted a roughly even distribution every time new files appeared in the source folder.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # Dumps all files into the same folder | |
| folders=($(ls targets/)) | |
| find source/ -type f -print0 | \ | |
| xargs -r0 -P 10 -I{} \ | |
| mv –target-directory=targets/${folders[$(expr $RANDOM % ${#folders[@]})]}/ |
But I noticed that every time this executed, all my source files were ending up in the same queue folder (and not being evenly distributed). What gives?
Turns out, my call to $RANDOM was being executed only once at runtime, so that value was being set statically, and used for all subsequent mv commands. The Eureka moment what when I realized that as a subshell command, I need to escape my dollar-signs so that they’d be ignored by the parent shell, and translated by the child.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # As an example, how to randomly distribute files to a collection of folders | |
| find source/ -type f -print0 | \ | |
| xargs -r0 -P 10 -I{} sh -c " | |
| folders=($(ls targets/)) | |
| mv –target-directory=targets/\${folders[\$(expr \$RANDOM % \${#folders[@]})]}/ | |
| " |
I suddenly found all my files going to the correct folders. Yet another reminder to always keep scope in mind.
