SEC seems to stop handling requests while inside SecStore

[davepacheco]

While working on oxidecomputer/omicron#6215, I ran into a bug (in my code) that caused the Nexus impl of SecStore to hang inside saga_update. While that was going on, requests to Nexus to list sagas hung. That was surprising. SecStore functions can be talking to storage over the network and the SEC should not become unresponsive while that's happening.

[davepacheco]

I should clarify: what I observed was that Nexus's internal saga list API hung. It's conceivable that it was hanging somewhere else (e.g., trying to talk to the database).

[TimoFreiberg]

I might have found the cause of this bug:
saga_update awaits the return oneshot channel in sec_send here while the Mutex around live_state is held (all the call sites of saga_update).
cmd_saga_list acquires that same Mutex here
(through SagaView::from_saga here, SagaStateView::from_run_state here, and <SagaExecutor as SagaExecManager>::status() here)

This might be fixable by moving the ack_rx.await farther out so no locks are held during this await, if the consequences are worth it.
The ordering of operations should stay preserved (while other saga actions could now run immediately after a SecExecMsg was sent instead of only after the response was received, holding the Mutex until exec_tx.send succeeds should ensure SecExecMsgs are sent and processed in the same order as previously)
But now other parallel saga nodes could start sending their messages before the previous message is confirmed, so i think in some error cases you could end up with more unconfirmed inflight messages than before?
I'll leave it up to you to judge that tradeoff!

This TODO about liveness brought me here :)