IDEA: Operate data flow on file-by-file basis

[tjk584]

Idea for potential improvement

Use inotify to trigger each script on every file that is moved to a specific directory, instead of using flags to trigger subsequent scripts. The inotify daemon could then trigger a background instance of the dataflow script for each file triggered on, allowing the dataflow to work in parallel.

Files moved could be watched for specific events such as "moved_to" for rsync to trigger after rsync_to_nas or rsync_to_campus

• Note: for rsync_to_nas, inotify events aren't triggered on CD/DDS computers when a different computer moves file to the NAS. Files would have to be transferred via the CD/DDS computers instead of directly to the NAS
• If scripts are executing in parallel, the logging would have to be changed slightly so the logfiles are still readable (as parallel processes would write to the file at the same time)

[tjk584]

Could potentially make it so both CD and DDS computers perform data flow computations at the same time (each triggering on different files). Not sure how to ensure they don't operate on the same files, but its a thought

[RemingtonRohel]

@tjk584 This is implemented and can be closed, correct?

[tjk584]

Nah, not quite. When convert_and_restructure or rsync_to_campus runs, they collect all files in their respective source directories. This issue suggests running each script so they operate on a single file at a time.