Error submitting to a slurm queue

[zpalmer618]

When submitting pbqff to a slurm queue, the following error is printed in the pbqff.log file:
thread 'main' panicked at /home/brent/.cargo/git/checkouts/psqs-bb7f245cb9c9f2af/2b4e5c7/src/queue/slurm.rs:142:17: assertion failed: fields.len() == 8

Investigating the issue in slurm.rs, reveals that it checks squeue -u $USER for 8 fields, but the present compute cluster will potentially print 9 fields due to an additional whitespace in the (REASON) portion of squeue:
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
1810906 requeue grad_aqz cpgbn PD 0:00 1 (ReqNodeNotAvail, UnavailableNodes:compute-12-13)

[ntBre]

See:

pbqff/crates/psqs/src/queue/slurm.rs

Lines 117 to 121 in 9555311

	/// run `squeue -u $USER`. form of the output is:
	///
	/// JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
	/// 30627992 compute c3oh- mdavis R 46-17:12:23 1 node2
	fn stat_cmd(&self) -> String {

and:

pbqff/crates/psqs/src/queue/slurm.rs

Lines 140 to 147 in 9555311

	if !line.contains("JOBID") {
	let fields: Vec<_> = line.split_whitespace().collect();
	assert!(fields.len() == 8);
	// exclude completing jobs to combat stuck completing bug
	if fields[4] != "CG" {
	ret.insert(fields[0].to_string());
	}
	}

We really only need the JobID and State fields, so we should probably request only those from squeue. There's also a JSON format mentioned in the docs, but it doesn't seem widely supported yet.

I suspect that you can work around it with something like this, if you can export the function in such a way that pbqff calls the redefined squeue function instead of the real one:

function squeue {
    /usr/bin/squeue "$@" | awk '{for (i = 9; i <= NF; i++) { $i = "" }; print}'

But we haven't gotten that to work yet.