Key already registered with the same priority: Python

[liangYe10086]

when i run command line " python -m wdb your_file.py " in a docker ,the program exit with the following infomation .
Key already registered with the same priority: Python
dose anyone see the similar problem ?

[vasica38]

@liangYe10086 did you manage to solve the issue somehow?

[mapix]

pip list | grep torch |  awk '{print $1}' | xargs pip uninstall -y

wdb not compatible with pytorch, uninstall all pytorch related packages works for me

[kkleidal]

Bump on this issue. I've been seeing the same thing. I did some more debugging and I think I know the cause:

I made some progress on debugging this, by getting down into GDB and looking at how the DLLs are being loaded.

Using this script:

import torch
import wdb
wdb.set_trace()
wdb.start_trace()
print("Hello world")

I was able to reproduce Key already registered with the same priority: Python

Basically the following DLLs were being loaded consecutively:

/usr/local/lib/python3.6/dist-packages/caffe2/python/caffe2_pybind11_state.cpython-36m-x86_64-linux-gnu.so
/usr/local/lib/python3.6/dist-packages/caffe2/python/caffe2_pybind11_state_gpu.cpython-36m-x86_64-linux-gnu.so

the first one succeeds, then the second one fails when registering a class to a static (global) registry, since there is already a class with the same name “Python” bound in the registry. This leads me to think that PyTorch expects only one of those two DLLs to be loaded, depending on whether or not I'm using a GPU.

Moreover, using this script:

import torch
print("Hello world")
print(torch.randn(4, 5) @ torch.randn(5, 4))

neither of the two DLLs are ever loaded anyway, which is fishy. It means Pytorch probably isn’t the one loading them.

I looked into the source for wdb and found it was using a library called importmagic. Sure enough, if I add:

import torch
import sys
import importmagic
index = importmagic.SymbolIndex()
index.build_index(sys.path)
print("Hello world")
print(torch.randn(4, 5) @ torch.randn(5, 4))

which is how WDB uses importmagic behind the scenes, the program tries to load both caffe DLLs and results in the “Key already registered with the same priority: Python”, cutting out WDB as a middleman.

Therefore, WDB is likely trying to use importmagic to recursively walk and import all Python packages in the path, including cpython plugins and their DLLs. I’m thinking it does this to build a symbol table of all possible functions to break on (though I’m not sure; haven’t explored too much). But doing so imports two packages torch never expects to be imported at the same time, resulting in the key collision in the global registry.

I don't have a fix for this, but hopefully it gives the maintainers more context.

Pytorch 1.6.0 CUDA 10.1

wdb==3.3.0
wdb.server==3.3.0

[kkleidal]

A workaround is to:

pip3 install wdb
pip3 uninstall -y importmagic3

WDB disables the feature of autoimporting variables that aren't imported in the python console if importmagic3 isn't installed, but that's the only feature that's disabled.