Attention MIL implementation

[fjsaezm]

Hello! Thank you for providing a simple implementation of so many models.
I have a question regarding the Attention Based MIL.

In the original implementation, the attention scores are computed as
$a_i = \frac{e^{f_i}}{\sum_{j} e^{f_j}}$,
that is, using a (masked) sotfmax on the obtained latent function $f$.

However, in your implementation, you are using a sigmoid:

weights = torch.sigmoid(scores)

which causes the attention values to lose the property of adding up to one. Wouldn't a softmax be more appropiate?

Also, the code of the functions used in this model are not provided:

self.score = MonoAdditiveAttentionScore(D, D)
self.pool = CountMILPool(D)

Could you also give the implementation of this functions?

Thank you!

[EdwardRaff]

Wouldn't a softmax be more appropiate?

Not for binary problems, which was all we looked at. MIL is not easily defined for multi-class problems. For binary problems, sigmoid w/ BCE and softmax are equivalent.

Could you also give the implementation of this functions?

Ugh, not sure what happened. I can look later, but unfortunately, I'm swamped. I can't promise any timeline on finding the code.

[fjsaezm]

Thank you for the quick response!
However, I was not referring to the classification part (which is done in the pooling).
I was speaking about the attention weights computation. Actually, I have observed that the softmax function is commented in the code:

#weights = F.softmax(scores, dim=1) #(B, T, 1) still, but sum(T) = 1
weights = torch.sigmoid(scores)

Shouldnt the softmax be used there?

[EdwardRaff]

Ah, yea that looks like a small error - it looks like others had it correct. It looks like this was an accidental grab of some other research I was doing in an alternative MIL model that wasn't intended to be in the code.