When would you support USE_FLASH_ATTENTION compile?

[xbcReal]

🚀 The feature, motivation and pitch

Hi, I wanna a faster transformer implemention in pytorch, and I found one in pytorch code ,whose path is pytorch/aten/src/Aten/native/transformers/cuda/, and it needs support USE_FLASH_ATTENTION compile. Furtherly I found some asm ptx code in utils.h, and amd-pytorch doesn't support it until now. So do you have any plan to support this feature?

[darren-amd]

Hi @xbcReal,

This is supported now; support was added through the upstream issue: pytorch#112997. Thanks for reporting it!