Purpose of attention stats

[taha-yassine]

I'm currently studying the code to get a better sense of how the method works.

In

entropix/entropix/main.py

Line 86 in e55e9a3

logits, kvcache, scores, stats = xfmr(xfmr_weights, model_params, next_token, cur_pos, freqs_cis[cur_pos:cur_pos+1], kvcache)

some attention stats are computed but never used, only the attention scores from the last transformer block (?) are provided to the sampler. Is it just an oversight or am I missing something?

Also, is there a Discord (or similar) to discuss stuff like this? Seems better suited that GH issues.

Thanks!