Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ValueError: The header keyword 'SIM SRC0 amplitudes' with its value is too long #70

Open
carmeloarci opened this issue Dec 20, 2023 · 3 comments
Open

ValueError: The header keyword 'SIM SRC0 amplitudes' with its value is too long #70

carmeloarci opened this issue Dec 20, 2023 · 3 comments

Comments

@carmeloarci
Copy link

carmeloarci commented Dec 20, 2023 •
edited
Loading

It is not possible to complete a simulation of a star cluser using SCOPESIM since python wants to save a string that is too long to be handled in a fits header keyword. It is not possible to save the fits image file.

In the example, we used 5 randomly placed stars using the scopesim_templates.stars tool. Specifically MICADO and MORFEO irdb were considered.

Attached the python script to reproduce the error
MORFEO_debug2.py.txt

Traceback (most recent call last):
File "/python/simcado/MORFEO_debug2.py", line 100, in
hdus_J_small_test_3DM = micado.readout(filename= "scopesim_debug.fits")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/utils.py", line 1053, in wrapper
output = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/optics/optical_train.py", line 328, in readout
hdul.writeto(fname, overwrite=True)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1006, in writeto
self.verify(option=output_verify)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/verify.py", line 82, in verify
errs = self._verify(opt)
^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1401, in _verify
result = hdu._verify(option)
^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 1162, in _verify
errs = super()._verify(option=option)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 599, in _verify
return super()._verify(option)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 1099, in _verify
errs.append(card._verify(option))
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1166, in _verify
keyword, valuecomment = self._split()
^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 838, in _split
image = self.image
^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 530, in image
self._image = self._format_image()
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1025, in _format_image
raise ValueError(
ValueError: The header keyword 'SIM SRC0 amplitudes' with its value is too long
@hugobuddel
Copy link
Collaborator

Thanks. You can workaround this problem by disabling the source fits keywords:

micado["source_fits_keywords"].include = False

Nevertheless we should fix this. Unfortunately, I cannot reproduce your exact error. The file produced by your script for small n does not contain the SIM SRC0 amplitudes keyword if I run it.

Could you please share the output of scopesim.bug_report() with us; it should be right above the Traceback?

However, increasing n from 5 to 6 will lead to a related error:

ValueError: FITS header values must contain standard printable ASCII characters; "scopesim_templates.stellar.stars.stars(filter_name='J', amplitudes=<Quantity [18.46960607, 21.22000502, 16.16989011, 19.55818477, 19.90220094,\n           15.61757236] mag>, spec_types=['A0V', 'A0V', 'A0V', 'A0V', 'A0V', 'A0V'], x=array([0.43299294, 0.81188355, 0.86067838, 0.12246919, 0.35946662,\n       0.45763185]), y=array([0.89122441, 0.36107683, 0.12356588, 0.70669564, 0.16268664,\n       0.86055453]), library='pyckles', ra=0, dec=0)" contains characters not representable in ASCII or non-printable characters.

The error above is caused by the key SIM SRC0 function_call. We explicitly 'prevent' the error I got by this line in fits_headers.py:

hdu.header[f"FN{key.split()[1]}"] = hdu.header.pop(key)

for that key. That line will convert the card from a HIERARCH card to a normal one, which supports the CONTINUE keyword. And indeed for n=5 this key indeed does not appear in the headers. But this code is not reached for n=6 because the error occurs earlier.

We would need to fix that 'hack', but once fixed, we could perhaps employ something similar for the other keywords that can reach an arbitrary length like SIM SRC0 amplitudes.

For completeness, this is the output of scopesim.bug_report() for me:

Python:
3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:34:09) [GCC 12.3.0]

Installed Python packages:
           anisocado: 0.3.0
             astropy: 5.3.3
      beautifulsoup4: 4.12.2
            docutils: 0.20.1
                lxml: 4.9.3
          matplotlib: 3.7.2
      more-itertools: 10.1.0
               numpy: 1.25.2
               pooch: 1.7.0
              pyyaml: 6.0.1
            requests: 2.31.0
      requests-cache: 1.1.0
               scipy: 1.11.2
       scopesim_data: could not be loaded.
  scopesim_templates: 0.4.5a0
         skycalc-ipy: 0.3.0a0
             synphot: 1.2.1
                tqdm: 4.66.1

Installed IRDB packages:
                .git: version number not available.
             .github: version number not available.
           Armazones: 2023-07-11
                 ELT: 2023-07-11
                ERIS: version number not available.
                FORS: version number not available.
                 GTC: version number not available.
             HARMONI: version number not available.
               HAWKI: 2023-06-14
                 HST: version number not available.
                KMOS: version number not available.
                LFOA: 2022-04-12
             LaPalma: version number not available.
               MCIFU: version number not available.
               METIS: 2023-07-11
              MICADO: 2023-07-14.dev
          MICADO_Sci: version number not available.
                MUSE: version number not available.
                NACO: version number not available.
              OSIRIS: version number not available.
             Paranal: 2022-04-09
                UVES: version number not available.
               VISIR: version number not available.
                 VLT: version number not available.
            ViennaLT: version number not available.
                WFC3: version number not available.
            _REPORTS: version number not available.
                docs: version number not available.
                irdb: version number not available.
        test_package: 2023-07-20.dev
               .idea: version number not available.
       .pytest_cache: version number not available.
    _ZIPPED_PACKAGES: version number not available.
              MORFEO: 2023-07-11
  .ipynb_checkpoints: version number not available.

Operating System info:
   System: Linux
  Release: 5.4.0-167-generic
  Version: #184-Ubuntu SMP Tue Oct 31 09:21:49 UTC 2023
  Machine: x86_64

@hugobuddel hugobuddel mentioned this issue Dec 20, 2023
Open
@carmeloarci
Copy link
Author

MORFEO_debugSRC0.py.txt
In this version of the file you may reproduce the error, avoiding randimization.

(scope_test) [carmelo@dyno simcado]$ python MORFEO_debug.py
['A0V']
Observe
Save
ERROR - Unhandled exception occured, see log file for details.
Traceback (most recent call last):
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/utils.py", line 1053, in wrapper
output = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/optics/optical_train.py", line 328, in readout
hdul.writeto(fname, overwrite=True)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1006, in writeto
self.verify(option=output_verify)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/verify.py", line 82, in verify
errs = self._verify(opt)
^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1401, in _verify
result = hdu._verify(option)
^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 1162, in _verify
errs = super()._verify(option=option)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 599, in _verify
return super()._verify(option)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 1099, in _verify
errs.append(card._verify(option))
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1166, in _verify
keyword, valuecomment = self._split()
^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 838, in _split
image = self.image
^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 530, in image
self._image = self._format_image()
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1025, in _format_image
raise ValueError(
ValueError: The header keyword 'SIM SRC0 amplitudes' with its value is too long
ERROR - Python:
3.11.5 | packaged by conda-forge | (main, Aug 27 2023, 03:34:09) [GCC 12.3.0]

Installed Python packages:
anisocado: 0.3.0
astropy: 5.3.3
beautifulsoup4: 4.12.2
docutils: 0.20.1
lxml: 4.9.3
matplotlib: 3.8.0
more-itertools: 10.1.0
numpy: 1.26.0
pooch: 1.8.0
pyyaml: 6.0.1
requests: 2.31.0
requests-cache: 1.1.0
scipy: 1.11.3
scopesim_data: could not be loaded.
scopesim_templates: 0.4.4
skycalc-ipy: 0.2.0
synphot: 1.2.1
tqdm: 4.66.1

Installed IRDB packages:
ELT: 2023-07-11
Armazones: 2023-07-11
MICADO: 2023-07-14.dev
MORFEO: 2023-07-11

Operating System info:
System: Linux
Release: 5.14.18-100.fc33.x86_64
Version: #1 SMP Fri Nov 12 17:38:44 UTC 2021
Machine: x86_64

Traceback (most recent call last):
File "/python/simcado/MORFEO_debug.py", line 110, in
hdus_J_small_test_3DM = micado.readout(filename= "scopesim_debug.fits")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/utils.py", line 1053, in wrapper
output = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/scopesim/optics/optical_train.py", line 328, in readout
hdul.writeto(fname, overwrite=True)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1006, in writeto
self.verify(option=output_verify)
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/verify.py", line 82, in verify
errs = self._verify(opt)
^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/hdulist.py", line 1401, in _verify
result = hdu._verify(option)
^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 1162, in _verify
errs = super()._verify(option=option)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/image.py", line 599, in _verify
return super()._verify(option)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/hdu/base.py", line 1099, in _verify
errs.append(card._verify(option))
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1166, in _verify
keyword, valuecomment = self._split()
^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 838, in _split
image = self.image
^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 530, in image
self._image = self._format_image()
^^^^^^^^^^^^^^^^^^^^
File "/home/carmelo/.conda/envs/scope_test/lib/python3.11/site-packages/astropy/io/fits/card.py", line 1025, in _format_image
raise ValueError(
ValueError: The header keyword 'SIM SRC0 amplitudes' with its value is too long

hugobuddel added a commit to AstarVienna/irdb that referenced this issue Dec 22, 2023
@hugobuddel
Disble SourceDescriptionFitsKeywords for MICADO
880a6ae 
The SourceDescriptionFitsKeywords still cause problems, see
AstarVienna/ScopeSim_Templates#70
AstarVienna/ScopeSim_Templates#69
@hugobuddel hugobuddel mentioned this issue Dec 22, 2023
Merged
@hugobuddel
Copy link
Collaborator

Thanks for sharing the output of scopesim.bug_report(). You are using ScopeSim_Templates 0.4.4, with which I can indeed reproduce the error.

The latest ScopeSim_Templates is 0.5.0, in which this particular problem does not occur. However, that is a coincidence, because 0.5.0 does not store the default source parameters. It would probably still fail with ScopeSim_Templates 0.5.0 if you would have explicitly listed the desired amplitudes in the script.

The short term fix would be to default to not storing these source description fits keywords, since they don't work properly, and they are probably not actively used, so they cause more problems then they solve, see AstarVienna/irdb#150 .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@hugobuddel @carmeloarci