A folder can exist and still be an unfinished download, so I want an archive job to tell me what happened when it finds that folder again. My own job had a gap there. It skipped the existing folder without checking its contents, recorded the status as existing, and returned a failed batch at the end. That kept it from claiming success, but it also meant a rerun couldn't certify a completed download.
I found that by reading the script after a successful model archive, where all seven repositories were marked verified and the recorded checksum checks passed for each one. The original run hadn't shown me an interruption. I tested the restart branch separately with a fake existing directory and mocked download commands.
The test returned a failure, with one existing status and six mocked verified statuses, and because no network transfer ran in that test, the six successful stubs prove nothing about downloaded files. The existing folder was skipped without a verification command. The batch failed closed.
The reserve is checked before each download
The archive was a batch of image, audio, and video models, with each repository pinned to a specific revision. Before each download, the script read free space on the target volume and compared it with the repository's estimated size plus a 200 GB reserve. The largest estimate in the batch was 127 GB, so that entry required at least 327 GB free at its preflight check. Those are decimal gigabytes in the script.
The reserve check ran again for the next repository, giving the job a chance to stop between downloads as free space changed, and if the target wasn't mounted at startup, the script refused to write. That guard matters when a missing mount could send large files to the system disk.
There's a boundary on that promise. The script checked free space immediately before a repository, it didn't monitor free space during the transfer. The size was an estimate supplied to the script, and the check wasn't a reservation that prevented another process from using the disk. A busy volume or an underestimate could still eat into that margin after the check passed.
The job stopped starting new repositories when the preflight found too little space. It also exited with a failure if the result list didn't cover the full batch. That's a useful distinction for an unattended job, because a short list of successful downloads doesn't mean the planned batch finished.
What the status actually means
For each new destination, the script started a download at the pinned revision. If that command returned a nonzero code, it recorded download_failed and moved on to the next repository. A successful download then got a separate verification command against the same revision, with missing files treated as a failure.
Only a successful verification produced verified; a failed check produced verification_failed, and either kind of failure made the final batch exit nonzero, with the results file recording a status for each attempted repository. In the original run, all seven entries were verified. The archive occupied 315G according to the recorded disk-usage output, which describes the archived directory at the time and doesn't measure peak temporary space during the transfers.
The final archive size is known, but the peak isn't. The 200 GB buffer was a policy choice, and the completed run shows it was sufficient for that run; I can't extend that conclusion to every interruption or concurrent workload without measuring them.
The pinned revision gives the verifier a concrete target, but it doesn't tell me whether a model produces good results or fits in memory, and one of the downloaded models wasn't loaded or tested for speech in that work. A checksum can't tell me how it sounds.
A safe failure still needs a recovery path
The awkward branch is the existing destination. The script sees a directory and skips both the download and the check. It records existing, then the final status test rejects the batch because that entry isn't verified. That avoids a false success, but it treats a complete prior download and a partial download the same way until someone checks them.
I reproduced that branch against a copied script in an isolated directory, with the mount and disk checks mocked and the external commands replaced by successful stubs. I created the first destination in advance and left the other destinations absent. The run recorded the preexisting one as existing, made no verification call for it, and exited with code 1. The remaining six success statuses came from stubs, so they say nothing about actual files.
A better restart path would verify an existing directory against its pinned revision before assigning a success status. If the check failed, the job could report the incomplete state and leave recovery explicit. I'd also want the results file to distinguish a fresh download from a previously verified archive, since those are different histories even when both pass the same check.
There's another reason to keep the result file specific to the run: this script writes it at the end, so a process that dies earlier won't leave a complete final batch ledger. Per-repository checkpoints would make an interrupted run easier to investigate, while verification on restart would decide what can be trusted.
How do you handle an existing destination in your download jobs, verify it on every restart or keep a separate checkpoint for each completed repository?
