Add read_raw_xdf

[MaxMascini]

Describe the new feature or enhancement

Port the xdf module (mnelab.io.xdf) from mnelab to mne-python. Currently, mnelab causes some dependency issues with numpy so we've had to copy and import the module separately.
The mnelab module already has an integrated mne class and read_raw_xdf:
Resolving streams, converting to mne RawArray, resampling streams, and functionality of converting an automatically-detected marker stream to raw annotations, among others.

Describe your proposed implementation

After basic testing on my end, the entire module appears already completely compatible with mne-python. Just integrating it into mne-python base package would be useful, and hopefully very easy.

Describe possible alternatives

Create a new module that handles converting xdf to mne Raw array. Resolve streams, resample data, and functionality of converting a marker stream to raw annotations would be incredibly useful (and are already implement in mnelab's xdf module)

Additional context

I would love to help, but I would not consider myself a weathered python developer. If someone can point me in the right direction with testing and where to plop the module, I would be than happy to make a pull request :)

[welcome]

Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴

[MaxMascini]

I noticed that there is an open PR for this from 2022 #10420 - What was/is the status on this?

[mscheltienne]

That might also be useful for mne-lsl at some point. There is already some parsing code of the metadata structure into MNE compatible fields: https://github.com/mne-tools/mne-lsl/blob/main/src/mne_lsl/utils/meas_info.py

[MaxMascini]

Pinging @cbrnr - Do you know what happened to the old PR to implement this?

[cbrnr]

Hi @MaxMascini! We never really decided what to do with it, and whether or not it is suitable for integration into MNE-Python.

Currently, there is a bug with our resampling implementation (cbrnr/mnelab#385), which produces incorrect timings if the recording has gaps (this is hopefully a rare situation, but of course it can happen). So until this is fixed, it cannot be moved to MNE-Python.

May I ask what the dependency issues of MNELAB with NumPy are?

Incidentally, I've recently considered moving the XDF reader out of MNELAB into a lightweight package. This would make it easier for users to access the reader without the PySide6 dependencies. Migrating it to MNE-Python might be premature since various things (including the interface) might still be changing, and MNE-Python puts a strong emphasis on stability and backward compatibility, which I cannot (and don't want to) commit to yet.

[MaxMascini]

Thanks for getting back to me @cbrnr! I think it's an excellent middle-ground to move the XDF utilities to a lightweight module for now.

Regarding the dependency issues we've encountered, we believe that the dependency issues are related to numba/numpy (which may just be a conda/mamba dependency solver issue).

When creating an environment from scratch
with mamba install python==3.12.7 mne mne-bids mnelab jupyterlab

We get this when trying to import read_raw_xdf:

---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Cell In[1], line 17
     15 from mne_bids.stats import count_events
     16 from pyxdf import resolve_streams
---> 17 from mnelab.io.xdf import read_raw_xdf
     18 from xml.etree.ElementTree import ParseError

File c:\ProgramData\miniforge3\envs\mnelab-test\Lib\site-packages\mnelab\__init__.py:14
     11 from PySide6.QtGui import QIcon
     12 from PySide6.QtWidgets import QApplication
---> 14 from mnelab.mainwindow import MainWindow
     15 from mnelab.model import Model
     17 __version__ = "0.9.2"

File c:\ProgramData\miniforge3\envs\mnelab-test\Lib\site-packages\mnelab\mainwindow.py:33
     30 from pyxdf import resolve_streams
     32 from mnelab.dialogs import *  # noqa: F403
---> 33 from mnelab.io import writers
     34 from mnelab.io.mat import parse_mat
     35 from mnelab.io.npy import parse_npy

File c:\ProgramData\miniforge3\envs\mnelab-test\Lib\site-packages\mnelab\io\__init__.py:6
      1 # © MNELAB developers
      2 #
      3 # License: BSD (3-clause)
      5 from mnelab.io.readers import read_raw
----> 6 from mnelab.io.writers import write_raw, writers

File c:\ProgramData\miniforge3\envs\mnelab-test\Lib\site-packages\mnelab\io\writers.py:9
      7 import mne
      8 import numpy as np
----> 9 from numpy.rec import fromarrays
     10 from scipy.io import savemat
     13 def write_fif(fname, raw):

ModuleNotFoundError: No module named 'numpy.rec'

When trying to create an environment with numpy>=2.1, we get this environment solving error and tree:

LibMambaUnsatisfiableError: Encountered problems while solving:
  - nothing provides vc 9.* needed by numba-0.46.0-py27h0e003f4_1

Could not solve for environment specs
The following packages are incompatible
├─ mne >=1.8.0  is installable with the potential options
│  ├─ mne 1.8.0 would require
│  │  └─ numba with the potential options
│  │     ├─ numba 0.46.0 would require
│  │     │  └─ vc 9.* , which does not exist (perhaps a missing channel);
│  │     ├─ numba [0.46.0|0.47.0|0.48.0|0.50.1] would require
│  │     │  └─ numpy >=1.14.6,<1.21.0a0  with the potential options
│  │     │     ├─ numpy [1.14.6|1.15.4|1.16.2|1.16.3|1.16.5] would require
│  │     │     │  └─ vc 9.* , which does not exist (perhaps a missing channel);
│  │     │     ├─ numpy [1.14.6|1.15.0|1.15.1|1.15.2] would require
│  │     │     │  └─ openblas >=0.2.20,<0.2.21.0a0 , which requires
│  │     │     │     └─ vc 14 , which does not exist (perhaps a missing channel);
│  │     │     ├─ numpy [1.14.6|1.15.2|...|1.16.4], which can be installed;
│  │     │     ├─ numpy 1.16.4 would require
│  │     │     │  └─ python >=2.7,<2.8.0a0  but there are no viable options
│  │     │     │     ├─ python [2.7.12|2.7.13|2.7.14|2.7.15] would require
│  │     │     │     │  └─ vc 9.* , which does not exist (perhaps a missing channel);
│  │     │     │     └─ python 2.7.15 would require
│  │     │     │        └─ vc >=9,<10.0a0 , which does not exist (perhaps a missing channel);
│  │     │     ├─ numpy 1.16.5, which can be installed;
│  │     │     ├─ numpy [1.16.6|1.17.0|1.17.1|1.17.2|1.17.3], which can be installed;
│  │     │     ├─ numpy 1.17.5, which can be installed;
│  │     │     ├─ numpy [1.18.1|1.18.4|...|1.19.4], which can be installed;
│  │     │     └─ numpy [1.19.5|1.20.0|1.20.1|1.20.2|1.20.3], which can be installed;
│  │     ├─ numba [0.49.0|0.49.1] would require
│  │     │  └─ numpy >=1.14.6,<1.18.0  with the potential options
│  │     │     ├─ numpy [1.14.6|1.15.4|1.16.2|1.16.3|1.16.5], which cannot be installed (as previously explained);
│  │     │     ├─ numpy [1.14.6|1.15.0|1.15.1|1.15.2], which cannot be installed (as previously explained);
│  │     │     ├─ numpy [1.14.6|1.15.2|...|1.16.4], which can be installed;
│  │     │     ├─ numpy 1.16.4, which cannot be installed (as previously explained);
│  │     │     ├─ numpy 1.16.5, which can be installed;
│  │     │     ├─ numpy [1.16.6|1.17.0|1.17.1|1.17.2|1.17.3], which can be installed;
│  │     │     └─ numpy 1.17.5, which can be installed;
│  │     ├─ numba [0.51.0|0.51.1|0.51.2|0.52.0] would require
│  │     │  └─ numpy >=1.16.5,<1.21.0a0  with the potential options
│  │     │     ├─ numpy [1.14.6|1.15.4|1.16.2|1.16.3|1.16.5], which cannot be installed (as previously explained);
│  │     │     ├─ numpy 1.16.5, which can be installed;
│  │     │     ├─ numpy [1.16.6|1.17.0|1.17.1|1.17.2|1.17.3], which can be installed;
│  │     │     ├─ numpy 1.17.5, which can be installed;
│  │     │     ├─ numpy [1.18.1|1.18.4|...|1.19.4], which can be installed;
│  │     │     └─ numpy [1.19.5|1.20.0|1.20.1|1.20.2|1.20.3], which can be installed;
│  │     ├─ numba 0.53.0 would require
│  │     │  └─ numpy >=1.16.6,<1.21.0a0 , which can be installed;
│  │     ├─ numba [0.53.0|0.53.1] would require
│  │     │  └─ numpy >=1.19.5,<1.21.0a0 , which can be installed;
│  │     ├─ numba 0.53.1 would require
│  │     │  └─ numpy >=1.17.5,<1.21.0a0 , which can be installed;
│  │     ├─ numba [0.54.0|0.54.1] would require
│  │     │  └─ numpy >=1.17,<1.21 , which can be installed;
│  │     ├─ numba [0.55.0|0.55.1] would require
│  │     │  └─ numpy >=1.18,<1.22 , which can be installed;
│  │     ├─ numba 0.55.2 would require
│  │     │  └─ numpy >=1.18,<1.23 , which can be installed;
│  │     ├─ numba [0.56.2|0.56.3|0.56.4] would require
│  │     │  └─ numpy >=1.18,!=1.22.0,!=1.22.1,!=1.22.2,<1.24 , which can be installed;
│  │     ├─ numba [0.57.0|0.57.1] would require
│  │     │  └─ numpy >=1.21,!=1.22.0,!=1.22.1,!=1.22.2,<1.25 , which can be installed;
│  │     ├─ numba [0.58.1|0.59.0|0.59.1] would require
│  │     │  └─ numpy >=1.22.3,<1.27 , which can be installed;
│  │     └─ numba 0.60.0 would require
│  │        └─ numpy >=1.22.3,<2.1  with the potential options
│  │           ├─ numpy [1.22.3|1.22.4|...|2.0.2], which can be installed;
│  │           └─ numpy [2.0.0rc1|2.0.0rc2|2.1.0rc1|2.2.0rc1] would require
│  │              └─ _numpy_rc, which does not exist (perhaps a missing channel);
│  └─ mne 1.8.0 would require
│     └─ __osx, which is missing on the system;
└─ numpy >=2.1.*  is not installable because there are no viable options
   ├─ numpy [2.0.0rc1|2.0.0rc2|2.1.0rc1|2.2.0rc1], which cannot be installed (as previously explained);
   └─ numpy [2.1.0|2.1.1|2.1.2|2.1.3] conflicts with any installable versions previously reported.

Let me know if you have any ideas - or if you see something we've missed!

[cbrnr]

Hmmm, I don't use conda/mamba, but MNELAB only requires numpy >= 2.0.0, which should be installable according to the output. Did you specifically request numpy >= 2.1.0?

[cbrnr]

Also, maybe np.rec.fromarrays is not even necessary here. I think this is the only place where NumPy >= 2 is required (the import for versions < 2 would be from numpy.core.records import fromarrays). So it could be either a simple try/except, or even better, maybe record arrays are not even needed here. Would you have time to investigate and if you find a solution make a PR?