Rebound Params issue

[brryan]

Running disk_nbody_cyl.in on 8 ranks on an interactive skylake-gold session gives me this error:

terminate called after throwing an instance of 'std::runtime_error'
  what():  ### PARTHENON ERROR
  Condition:   pin_hash == pin_hash_root
  Message:     Parameter input object must be the same on every rank, otherwise I/O may be
                unable to write it safely. If you reached this error message, look to make sure
                that your calls to functions that look like pin->GetOrAdd are all called
                exactly the same way on every MPI rank.
  File:        /vast/home/brryan/github/artemis/external/parthenon/src/outputs/output_utils.cpp
  Line number: 340

Weirdly, this error goes away if I run with only 4 ranks. If I fill out the reb_simulation struct on all ranks rather than just the first (line ~153 of nbody.cpp) I can run for at least a few cycles. We may just have to keep this struct synced across ranks. Not sure why this hasn't come up before.

[adamdempsey90]

what is this actually testing? That the parameter file is the same on all ranks?

[adamdempsey90]

It's the restart string of bytes we stuff into the params (in InitializeFromRestart)?

[adamdempsey90]

If it's complaining about the reb_sim we add at line 198, then that's never used when outputting/reading, so the fatal is kind of annoying. I don't think there's anything wrong with what we're doing, i.e., we read/write the rebound sim correctly.

[adamdempsey90]

Can we just add some flag to parameters that says don't write/read these from the outputs?

[brryan]

Yeah I think it's just reb_sim, not the string we create from that to store for restarts which should be common to all procs. I think you're right that Parthenon is too aggressive in testing for consistency across parameters, since in general we don't save custom types so they don't matter. I'll take a look to see if this would be straightforward to fix.