Missing (non private) elements in dictionary

[medihack]

There seem to be some elements missing in the dictionary of pydicom (v2.4.3) that are in the registry of the DICOM PS3.6 2023d - Data Dictionary

For example:

pydicom.datadict.dictionary_has_tag("Content​Date") # -> False
pydicom.datadict.dictionary_has_tag("Instance​Creation​Date") # -> False
pydicom.datadict.dictionary_has_tag("Content​Time") # -> False
pydicom.datadict.dictionary_has_tag("Instance​Creation​Time") # -> False

Does pydicom only include an excerpt of elements from the standard? I wonder even more as those DICOM keywords are even found in the pydicom codebase (e.g. in fileset.py). Do I have to do something to enable those?

[scaramallion]

We haven't updated to 2023d yet, I had a PR that did so, but it did some other stuff that didn't end up working as planned so I closed it.

I'll probably submit a separate PR with just the dict updates tomorrow. If you're curious you need to run the various scripts here (they need a little bit of tweaking to fix an issue introduced by switching to https addresses, though).

[scaramallion]

Although I'm pretty sure those tags should be present, let me take a look

[scaramallion]

>>> datadict.dictionary_has_tag("ContentDate")
True
>>> pydicom.__version__
'2.4.2'  # version is wrong, is actually 2.4.3

???

Also:

pydicom/pydicom/_dicom_dict.py

Line 111 in f86e0df

0x00080023: ('DA', '1', "Content Date", '', 'ContentDate'), # noqa

[medihack]

Ok, and now something extremely strange:

In [18]: key = "ContentDate"

In [19]: pydicom.datadict.dictionary_has_tag(key)
Out[19]: True

In [20]: pydicom.datadict.dictionary_has_tag("Content​Date")
Out[20]: False

In [21]: pydicom.datadict.dictionary_has_tag(key)
Out[21]: True

In [22]: pydicom.datadict.dictionary_has_tag("Content​Date")
Out[22]: False

In [23]: pydicom.datadict.dictionary_has_tag(key)
Out[23]: True

In [24]: pydicom.datadict.dictionary_has_tag("Content​Date")
Out[24]: False

In [25]: pydicom.datadict.dictionary_has_tag(key)
Out[25]: True

I really don't understand what is happening here. It doesn't make any sense :-(

[medihack]

I have to apologize. It seems there is some kind of invisible character inside the "ContentDate" string. I never saw that before. It is NOT a problem of pydicom!

In [32]: "ContentDate" == "Content​Date"
Out[32]: False

In [33]: "ContentDate" == "ContentDate"
Out[33]: True

The first one is the keyword directly copied from the official DICOM registry page. It contains some kind of invisible character between "Content" and "Date". Such a nasty thing. And I am so sorry to bother you with such stuff :-(

[mrbean-bremen]

Good to hear that it is not a problem with pydicom - it sounded really mysterious! Invisible characters sounds familiar - ran into something like this a long time ago...

[mrbean-bremen]

Ah, and I had to scrub this kind of characters in the DICOM validator code too - at least in some older versions of the standard they appeared in some of the DICOM tags in the docbook version of the standard.

EDIT: Just in case somebody does not read all the comments below: here is the explanation by @dclunie where these characters come from, and that they are perfectly legal (excerpt from the release notes of each DICOM standard):

"Zero width spaces (U+200B) are used in some places to allow long words (such as PS3.6 keywords and UIDs) to break within table columns and avoid tables becoming too wide to fit on a page. These need to be filtered out before using these words literally."

[scaramallion]

I've had this thing bite me before, too. If your editor supports showing invisible characters then it can be handy to turn it on. Another thing to be careful of with copy + paste is that the encoding may not be ASCII. There was a reported issue in pynetdicom where most UIDs being sent to pynetdicom were being encoded as ASCII but one was UTF-8, probably because the value had been copy + pasted into their equivalent of a data dictionary.

[medihack]

Unfortunately, in VS Code "Unicode Highlight: Invisible Characters" is turned on by default (and also enabled in my VS Code), but this one isn't highlighted (not even when rendering whitespace is enabled). Vim, for example, does show it as "Content<200b>Date". I also wonder why they put something like this on the official NEMA site. And yes, good to know that this is not a pydicom problem :-) And I also learned something new ... the hard way :-/

EDIT: It is highlighted in a local VS Code install but not in a Cloud IDE like Gitpod inside the browser or in any REPL (python / ipython).

[edmcdonagh]

Is this an issue in the current docbook version of the standard, or just the HTML?

[medihack]

I copied those keywords from the HTML site here: https://dicom.nema.org/medical/dicom/current/output/chtml/part06/chapter_6.html