submit can be run twice if another command comes in before the prior one returns.

[rcoreilly]

has been happening with grunti which fires off a status after submit -- that subsequent status ends up triggering another submit in a few cases..

given the glacial disk access on blanca, it is likely that > 10sec are passing for the first _sub to check the git repo updates, and so the second one gets the exact same set of updates.

thus, it would be good for _sub to create and remove a lock file so it is never run 2x. the problem here is cleaning up the lock after crashes, but presumably that is doable.

meanwhile, here's some lines to add to submit() method in grunter.py that should prevent the problem:

    if os.path.isfile('job.sbatch'):
        print("Error: job.sbatch exists -- attempt to submit job twice!")
        return