Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

程序会卡在test Syncing处 #105

Open
chaibosong opened this issue Oct 28, 2024 · 8 comments
Open

程序会卡在test Syncing处 #105

chaibosong opened this issue Oct 28, 2024 · 8 comments

Comments

@chaibosong
Copy link

我在s3dis数据集上,使用两张h100显卡训练

sh scripts/train.sh -g 2 -d s3dis -c semseg-pt-v3m1-0-base -n semseg-pt-v3m1-0-base

总是在在Syncing ...处卡住并最后time out,resume也会卡在这里
微信图片_20241028203534

@Gofinge
Copy link
Member

Gofinge commented Oct 29, 2024

正常来说慢慢等就好了,会在timeout前搞定,你看看单卡走测试流程呢?

@chaibosong
Copy link
Author

chaibosong commented Oct 30, 2024

我使用了4张h100,使用给出的config训练S3DIS数据集,只能跑出69.75%的精度,无法达到文中提到的73%精度,数据增强和超参数都与给出的config保持一致。请问可能是什么问题呀?运行这个config文件:
https://github.com/Pointcept/Pointcept/blob/main/configs/s3dis/semseg-pt-v3m1-1-rpe.py

semseg-pt-v3m1-1-rpe.py和semseg-pt-v3m1-0-rpe.py之间是什么区别呀

@Gofinge
Copy link
Member

Gofinge commented Oct 30, 2024

我最近做无监督预训练也有跑过,应该是能比较容易达到的。RPE是用了relative positional encoding,关闭flash attention和conditional positional encoding。当时的结论是s3dis上rpe好一些,不过后面随着预训练发展以及scaling up,搭配预训练下使用标准的ptv3 config也能达到最高的结结果,这些等今年晚点发布吧。

@chaibosong
Copy link
Author

所以您汇报的73.6结果使用预训练了吗还是?
semseg-pt-v3m1-1-rpe.py和semseg-pt-v3m1-0-rpe.py
-1-rpe 和 -0-rpe,这里1和0分别代表什么呀

@Gofinge
Copy link
Member

Gofinge commented Oct 30, 2024

Scratch结果应该是这样的,PPT(多数据集联合训练,最高)是用了标准的参数。现在其实不太推荐scratch了,小数据规模下带来的结论是不准确的。0,1 表示config编号

@chaibosong
Copy link
Author

chaibosong commented Nov 2, 2024

请问可以提供与Pointcept框架分离,第二种执行方式的ptv3的scratch训练脚本吗,如果您方便的时间可以提供我将不胜感激!

@Gofinge
Copy link
Member

Gofinge commented Dec 2, 2024

Inference demo和Pointcept分离是有可能的,Training process和Pointcept分离是不可能的,因为Pointcept本身就是提供达到Sota的训练工具

@chaibosong
Copy link
Author

感谢您的耐心回复!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants