The docs for this PR livehere. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Validation for cvt-13 lines up with paper (81.678 top-1 for me), acts are off from reference impl by minute amounts (MSE of logits for 1 sample off on the order of 1e-10). Initial problems had to do with norm before attn and attn residual when there is a cls_token. I'll finish this off later today most likely.

@fffffgggg54 is this still in progress ?

@fffffgggg54 is this still in progress ?

Yes. I am a bit busy and traveling right now, but I am still working on getting the validation to line up with the paper. It has been a pain to try and find which part of the model deviates from the reference impl.

Some updates@rwightman@Smartappli: I have added thereference implementation into a branch on my fork and compared validation performance. There are some slight numerical deviations and the top-1 is off by an insignificant amount. After digging in the reference repo's validation setup, I changed the validation configurations so that the crop settings match what the authors used. The throughput of my implementation sees a ~10% increase over the reference impl on win10/pt2.2.0/fp32. Fused attn was not available. There is a bit of cleanup still (head, torchscript?, stem), but this is the last technical hurdle I was hung up on.