Keep.sh

Cleaning up model output with keep.sh

After doing a large model run and the analysis of all the output files is complete, it is often a good idea to delete output files to free up hard drive space on the machine. However, deleting all the data files is undesirable since the model would need to be re-run to reproduce all of the results leading up to the final output time. As a compromise, you may consider keeping every Nth output file, while deleting the rest. This task can be automated, as explained below.

You can use the shell script keep.sh which should be placed in your $HOME/bin. You also need to make the script executable, use the command: chmod 755 keep.sh. The code for keep.sh follows:

#!/bin/bash
#Usage: keep <pattern> <N>
#Purpose: keep every Nth file with name matching <pattern>

pattern=$1
N=$2

FILES=$pattern

count=0

for f in $FILES
do
  if [ $count -lt $N ]
  then
    if [ $count -ne 0 ]
    then
      rm $f
    fi
  fi

  if [ $count -eq $N ]
  then
    count=0
  fi

  count=$(($count+1))

done

echo done.

Example Usage

The command keep.sh 'out*.mat' 10 would keep the files, "out010.mat, out020.mat", etc. if the original list was all files numbered from 001, 002, ... ,100.

Notes:

• We have to enclose the pattern in single-quotes (') so that bash will properly parse the wild-card string (*).
• Having zero-padded output files is recommended to avoid sequence loss.