GDCM-based decompression failure

[naterichman]

I just wanted to post this in case anyone else runs into it, I don't think its an issue with pydicom, but in the python bindings for GDCM.

I've run into this a number of times, seems as though some US Dicoms (particularly from the Philips EPIQ CVx) are produced with jpeg compression, but are invalid. In this case I get an error trying to decompress with nearly every library I've used (dcmtk, pylibjpeg, gdcm, etc.)

The problem is just doing dcm = dcmread('<file>') and dcm.decompress() wth GDCM installed silently fails. GDCM seems to print to stderr:

Not a JPEG file: starts with 0xc4 0x8f
...

without returning anything in a meaningful way that the python code can detect and raise an error from this.

The resultant pixel_array is simply all 0's. The best workaround I've found is to just raise an error myself in this case:

dcm.decompress()

if np.max(dcm.pixel_array) == 0:
    raise RuntimeError("GDCM decompression failed")

but I'm curious if anyone has a better way to tell when that fails. Also apologies I don't have any deidentified data I can share for this. I will try to de-identify some.

[scaramallion]

Is it too much trouble for you to attach an example of a faulty JPEG? I'd like to have a closer look

[naterichman]

I will do my best, I need to get permission from the image creator to share it publicly. Either way I'll need to remove most of the metadata except that relevant for viewing

[scaramallion]

No problem, even just the raw jpg stream is fine.

[naterichman]

Unfortunately I am not going to be able to provide the source data. I was able to extract a few frames from it using dcmj2pnm and there is identifying information burned into the pixel data. I can provide some more information from my forensics, but I'm not sure how much help that will be.

The NumberOfFrames indicates 28 frames, and so does the BOT. However looking at the actual fragments, and filtering jpeg SOI and EOI image markers, there are actually only 24 frames. Most of the frames have 7 fragments per frame, but one frame has 14 fragments, and one frame has 28 fragments, (22*7 + 14 + 28 = 196). The first 4 entries of the BOT are 0, after that the actual offsets start.

From what I can tell the error that I'm seeing when decoding is coming from GDCM trying to parse the non-standard frames (i.e. the 14 and 28 fragment frames) as if they were the normal number of fragments. For example on the frame that has 14 fragments, after it has read the first 7 fragments, GDCM prints out Not a JPEG file: starts with 0xc4 0x8f for each of the next 7 lines, where the starts with a b corresponds with the first two bytes of the fragment.

I did notice that the decode_data_sequence function used in the pydicom.pixel_data_handlers.gdcm_handler.create_data_element function does not seem to do anything with the BOT, and I'm not familiar enough with GDCM internals to know if the BOT is used in the GDCM decode internals.

Again sorry for not being able to share the jpg stream

[naterichman]

Still working on getting permission to share the pixel data. But in the meantime, I have found some more interesting things.

If I try to decompress the whole image with either gdcmconv -w or dcmdjpeg I get the error. But if I use dcmj2pnm and extract frame by frame, i.e.:

for frame in {1..<NumberOfFrames>}; do
    dcmj2pnm --write-jpeg --frame $frame $IMAGE > "${frame}.jpeg"
done

I get 28 images I can open in a normal image viewer.

Additionally I've spent some time stepping through dcm.decompress() in the gdcm_handler and I find that the 28 frame image is comprised of 196 fragments (7 fragments per frame). I tried manually concatenating chunks of 7 fragments and dumping that into files and I can view those, and they seem to be valid jpegs:

from pydicom import dcmread
from pydicom.encaps import decode_data_sequence

dcm = dcmread(PATH)

data_seq = decode_data_sequence(dcm.PixelData)

for i in range(0, len(data_seq), 7):
    raw = b''.join(data_seq[i:i+7])
    with open(f'/tmp/dicom/{i}.jpeg', 'wb') as fp:
        fp.write(raw)

$ /tmp/dicom/file 0.jpeg
0.jpeg: JPEG image data, JFIF standard 1.01, aspect ratio, density 1x1, segment length 16, comment: "Intel(R) JPEG Library, version [2.0.16.48]", baseline, precision 8, 800x600, components 3

Like I said, I'll continue trying to post the actual data, but maybe some of this will ring a bell