Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Revert "Converted to the new STK simple_fields workflow" #1234

Merged
merged 1 commit into from
Jan 5, 2024

Conversation

jrood-nrel
Copy link
Contributor

@jrood-nrel jrood-nrel commented Jan 5, 2024

Reverts #1233

This PR was causing Nalu-Wind to segfault when using the Intel compiler.

@jrood-nrel jrood-nrel requested a review from psakievich January 5, 2024 02:27
@jrood-nrel jrood-nrel enabled auto-merge (squash) January 5, 2024 02:27
@jrood-nrel jrood-nrel merged commit 69e080b into master Jan 5, 2024
3 checks passed
@jrood-nrel jrood-nrel deleted the revert-1233-master branch January 5, 2024 02:52
@djglaze
Copy link
Contributor

djglaze commented Jan 10, 2024

Hi @jrood-nrel,

Could you share some more details about the seg-faults you were seeing? I'm unable to trigger any misbehavior with the intel-2021.1.2 compiler here at Sandia, with either the unit or regression tests. What compiler did you use, what machine was it on, and which test(s) were seg-faulting? Thanks!

@jrood-nrel
Copy link
Contributor Author

This was on Eagle. It's a case Ganesh is running.

MPT: #1  0x00002b53f0858c96 in mpi_sgi_system (
MPT: #2  MPI_SGI_stacktraceback (
MPT:     header=header@entry=0x7ffefbf08990 "MPT ERROR: Rank 242(g:242) received signal SIGSEGV(11).\n\tProcess ID: 72481, Host: r3i2n11, Program: /lustre/filesystem/scratch/user/spack-manager/spack/opt/spack/linux-rhel7-skylake_avx512/intel-20.0.2/"...) at sig.c:340
MPT: #3  0x00002b53f0858e8f in first_arriver_handler (signo=signo@entry=11, 
MPT:     stack_trace_sem=stack_trace_sem@entry=0x2b53fedc0080) at sig.c:489
MPT: #4  0x00002b53f0859123 in slave_sig_handler (signo=11, 
MPT:     siginfo=<optimized out>, extra=<optimized out>) at sig.c:565
MPT: #5  <signal handler called>
MPT: #6  0x00002b53e38c2314 in sierra::nalu::max_extent(stk::mesh::FieldBase const&, unsigned int) ()
MPT:    from /lustre/filesystem/scratch/user/spack-manager/spack/opt/spack/linux-rhel7-skylake_avx512/intel-20.0.2/nalu-wind-master-33lj2rbocawi6fvq2lqcqugtdhpzwbrf/lib/libnalu.so
MPT: #7  0x00002b53e37d439e in sierra::nalu::NodalGradAlgDriver<stk::mesh::Field<double, void, void, void, void, void, void, void> >::post_work() ()
MPT:    from /lustre/filesystem/scratch/user/spack-manager/spack/opt/spack/linux-rhel7-skylake_avx512/intel-20.0.2/nalu-wind-master-33lj2rbocawi6fvq2lqcqugtdhpzwbrf/lib/libnalu.so
MPT: #8  0x00002b53e2670ba7 in sierra::nalu::WallDistEquationSystem::solve_and_update() ()
MPT:    from /lustre/filesystem/scratch/user/spack-manager/spack/opt/spack/linux-rhel7-skylake_avx512/intel-20.0.2/nalu-wind-master-33lj2rbocawi6fvq2lqcqugtdhpzwbrf/lib/libnalu.so
MPT: #9  0x00002b53e223c980 in sierra::nalu::EquationSystems::initial_work() ()
MPT:    from /lustre/filesystem/scratch/user/spack-manager/spack/opt/spack/linux-rhel7-skylake_avx512/intel-20.0.2/nalu-wind-master-33lj2rbocawi6fvq2lqcqugtdhpzwbrf/lib/libnalu.so
MPT: #10 0x00002b53e25ef833 in sierra::nalu::TimeIntegrator::prepare_for_time_integration() ()
MPT:    from /lustre/filesystem/scratch/user/spack-manager/spack/opt/spack/linux-rhel7-skylake_avx512/intel-20.0.2/nalu-wind-master-33lj2rbocawi6fvq2lqcqugtdhpzwbrf/lib/libnalu.so
MPT: #11 0x00002b53e25efce5 in sierra::nalu::TimeIntegrator::integrate_realm() ()
MPT:    from /lustre/filesystem/scratch/user/spack-manager/spack/opt/spack/linux-rhel7-skylake_avx512/intel-20.0.2/nalu-wind-master-33lj2rbocawi6fvq2lqcqugtdhpzwbrf/lib/libnalu.so
MPT: #12 0x0000000000415121 in main ()
Simulations:
- name: sim1
  optimizer: opt1
  time_integrator: ti_1
Time_Integrators:
- StandardTimeIntegrator:
    name: ti_1
    realms:
    - realm_1
    second_order_accuracy: true
    start_time: 0
    termination_step_count: 100
    time_step: 0.0002667
    time_step_count: 0
    time_stepping_type: fixed
realms:
- boundary_conditions:
  - target_name: wing
    wall_boundary_condition: bc_wing
    wall_user_data:
      turbulent_ke: 0.0
      use_wall_function: false
      velocity:
      - 0
      - 0
      - 0
  - target_name: wing-pp
    wall_boundary_condition: bc_wing_pp
    wall_user_data:
      turbulent_ke: 0.0
      use_wall_function: false
      velocity:
      - 0
      - 0
      - 0
  - inflow_boundary_condition: bc_inflow
    inflow_user_data:
      specific_dissipation_rate: 919.3455
      turbulent_ke: 0.0010422
      velocity:
      - 75.0
      - 0.0
      - 0.0
    target_name: inlet
  - open_boundary_condition: bc_open
    open_user_data:
      pressure: 0.0
      specific_dissipation_rate: 919.3455
      turbulent_ke: 0.0010422
      velocity:
      - 0.0
      - 0.0
      - 0.0
    target_name: outlet
  - periodic_boundary_condition: bc_front_back
    periodic_user_data:
      search_tolerance: 0.0001
    target_name:
    - front
    - back
  check_for_missing_bcs: true
  equation_systems:
    max_iterations: 4
    name: theEqSys
    solver_system_specification:
      velocity: solve_mom
      turbulent_ke: solve_scalar
      specific_dissipation_rate: solve_scalar
      pressure: solve_elliptic
      ndtw: solve_elliptic
    systems:
    - WallDistance:
        convergence_tolerance: 1.0e-08
        max_iterations: 1
        name: myNDTW
    - LowMachEOM:
        convergence_tolerance: 1.0e-08
        max_iterations: 1
        name: myLowMach
    - ShearStressTransport:
        convergence_tolerance: 1.0e-08
        max_iterations: 1
        name: mySST
  initial_conditions:
  - constant: ic_1
    target_name: fluid-hex
    value:
      pressure: 0
      specific_dissipation_rate: 919.3455
      turbulent_ke: 0.0010422
      velocity:
      - 75.0
      - 0.0
      - 0.0
  material_properties:
    specifications:
    - name: density
      type: constant
      value: 1.2
    - name: viscosity
      type: constant
      value: 9.0e-06
    target_name: fluid-hex
  mesh: mesh/ffa_w3_500_525_288_121_32.exo
  #automatic_decomposition_type: rcb
  #rebalance_mesh: yes
  #stk_rebalance_method: parmetis
  #use_edges: yes
  #check_jacobians: true
  name: realm_1
  output:
    output_data_base_name: results/ffa_w3_500_32_sst.e
    output_frequency: 100
    output_node_set: false
    output_variables:
    - velocity
    - density
    - pressure
    - pressure_force
    - viscous_force
    - tau_wall_vector
    - tau_wall
    - turbulent_ke
    - specific_dissipation_rate
    - minimum_distance_to_wall
    - sst_f_one_blending
    - turbulent_viscosity
    - element_courant
    - q_criterion
    - vorticity
    - assembled_area_force_moment
  restart:
    restart_data_base_name: restart/sst_ffa_w3_500_32.rst
    restart_frequency: 500
  solution_options:
    name: myOptions
    options:
    - hybrid_factor:
        specific_dissipation_rate: 1.0
        turbulent_ke: 1.0
        velocity: 1.0
    - alpha_upw:
        specific_dissipation_rate: 1.0
        turbulent_ke: 1.0
        velocity: 1.0
    - upw_factor:
        specific_dissipation_rate: 0.0
        turbulent_ke: 0.0
        velocity: 1.0
    - limiter:
        pressure: true
        velocity: true
    - noc_correction:
        pressure: true
    - projected_nodal_gradient:
        ndtw: element
        pressure: element
        specific_dissipation_rate: element
        turbulent_ke: element
        velocity: element
    - relaxation_factor:
        pressure: 0.3
        specific_dissipation_rate: 0.7
        turbulent_ke: 0.7
        velocity: 0.7
    - turbulence_model_constants:
        SDRWallFactor: 0.625
    projected_timescale_type: momentum_diag_inv
    turbulence_model: sst
  use_edges: true
linear_solvers:
- dump_hypre_matrix_stats: false
  hypre_cfg_file: hypre_file.yaml
  hypre_cfg_node: hypre_simple_precon
  kspace: 100
  max_iterations: 100
  method: hypre_gmres
  name: solve_mom
  output_level: 0
  preconditioner: boomerAMG
  recompute_preconditioner_frequency: 1
  reuse_linear_system: true
  segregated_solver: true
  simple_hypre_matrix_assemble: true
  tolerance: 1e-5
  type: hypre
  write_matrix_files: false
- dump_hypre_matrix_stats: false
  hypre_cfg_file: hypre_file.yaml
  hypre_cfg_node: hypre_simple_precon
  kspace: 100
  max_iterations: 100
  method: hypre_gmres
  name: solve_scalar
  preconditioner: boomerAMG
  recompute_preconditioner_frequency: 1
  reuse_linear_system: true
  simple_hypre_matrix_assemble: true
  tolerance: 1e-5
  type: hypre
  write_matrix_files: false
- dump_hypre_matrix_stats: false
  hypre_cfg_file: hypre_file.yaml
  hypre_cfg_node: hypre_elliptic
  kspace: 40
  max_iterations: 100
  method: hypre_gmres
  name: solve_elliptic
  preconditioner: boomerAMG
  recompute_preconditioner_frequency: 1
  reuse_linear_system: true
  simple_hypre_matrix_assemble: true
  tolerance: 1e-5
  type: hypre
  write_matrix_files: false

@djglaze
Copy link
Contributor

djglaze commented Jan 10, 2024

@jrood-nrel Thanks for the information! The stack trace was juuuust enough to figure it out. I messed up a Field name when configuring a ScalarNodalGradAlgDriver instance inside the WallDistEquationSystem, so it ended up referencing a null Field when retrieving some sizing information for updating either a periodic BC field or an overset mesh Field after the nodal grad calculation was complete. This mistake should be independent of compiler and we have 17 regression tests that exercise this equation system. I find it a bit troubling that my local GCC build ran just fine with precisely zero diffs.

Either way, I'll wait to reintroduce this simple_fields update until after @psakievich is finished with updating nalu-wind with his smart fields and field manager changes. This patch should be significantly smaller once that work is done.

psakievich added a commit that referenced this pull request Apr 30, 2024
* Add Timers for FSI (#1221)

* Add Timers for FSI

* Style

* Add FSI section to Input File docs. (#1220)

* Add FSI section to Input File docs.

* Updating the FSI docs to incorporate PR comments.

* Relax projection for FSI mapping to account for curvature (#1223)

* Relax projection for FSI mapping to account for curvature

* Style

* Format again

* Add option to dump mesh on failed jacobian check (#1226)

* Add option to dump mesh on failed jacobian check

* Style

* Cleaner, parallel consistent impl

* Correctly set current_coordinates for restart (#1227)

* Actually set current_coordinates for restart

* Remove redundant call updating displacements

* Revert "Remove redundant call updating displacements"

This reverts commit dca015c.

* Fix restart issue for FSI simulations (#1228)

---------

Co-authored-by: Ganesh Vijayakumar <[email protected]>

* Update FieldRegistry.h (#1229)

missing std::.

* FSI: Set ramping defaults to false (#1231)

* FSI: Set ramping defaults to false

Since we are moving to split meshes as the current strategy setting
ramping defaults to `False`.

I have been investigating why the temporal ramping is causing
simulations to fail since the bug fixes regarding hub motion, and I
can't find any issues with it at the moment. So it seems best to keep
that turned off as well.

* Update real defaults

* add a string-function temperature IC/dirichlet bc (#1198)

* add a string-function temperature IC/dirichlet bc

* undo stk deprecation fixes

---------

Co-authored-by: psakievich <[email protected]>

* Converted to the new STK simple_fields workflow (#1233)

STK is migrating to a new strategy for registering and managing
Fields, where sizing information is purely specified at run-time
instead of the previous technique of specifying it in a confusing
blend of both compile-time and run-time information.  The compile-time
specification was just a suggestion, as it could be overridden
(possibly inconsistently) at run-time to support variable-length
Fields.  This made it unclear what the true size of a Field was
and where it should be specified.

As an example, registering a vector field on the entire mesh
previously looked like this:

  using VectorField = stk::mesh::Field<double, stk::mesh::Cartesian3d>;
  VectorField & field = meta.declare_field<VectorField>(stk::topology::NODE_RANK, "velocity");
  stk::mesh::put_field_on_mesh(field, meta.universal_part(), 3, nullptr);

and now, it looks like this:

  using VectorField = stk::mesh::Field<double>;
  VectorField & field = meta.declare_field<double>(stk::topology::NODE_RANK, "velocity");
  stk::mesh::put_field_on_mesh(field, meta.universal_part(), 3, nullptr);

  stk::io::set_field_output_type(field, stk::io::FieldOutputType::VECTOR_3D); // Optional

The only template parameter for a Field is now the datatype parameter.
Sizing information now exclusively comes from put_field_on_mesh() calls.
The optional set_field_output_type() function call registers with the
IO sub-system how a multi-component Field should be subscripted in
Exodus files.  If this call is left off, you will get the default
[_1, _2, _3] subscripting.  With the above call, you will instead get
[_x, _y, _z] subscripting.

The MetaData::use_simple_fields() flag is set everywhere possible in
the code to prevent accidental regressions before the old behavior
is formally deprecated and removed.  This will yield a run-time error
if the old-style extra template parameters are used anywhere.  These
calls to use_simple_fields() can be removed in the future once the
STK Mesh back-end has removed support for the old behavior.

This wasn't a completely straightforward conversion due to nalu-wind
making heavy use of various algorithm selections based on the
templated Field type.  The ScalarFieldType, VectorFieldType,
TensorFieldType, and GenericFieldType types are now all identical,
so different techniques had to be used to switch behaviors.

* Fixes errors in ghosting update for some overset mesh problems. (#1214)

Instead of trying to modify the ghosting for overset problems with
relative motion, we now just rebuild the ghosting from scratch.
This appears to fix errors such as those reported in #936

Co-authored-by: dcdemen <[email protected]>
Co-authored-by: psakievich <[email protected]>

* Revert "Converted to the new STK simple_fields workflow (#1233)" (#1234)

This reverts commit 8e8f4d5.

* Fixed the iblank inconsistency at shared nodes. The iblank field is returned to Tioga after the stk::mesh::copy_owned_to_shared operation, prior to field interpolation (#1238)

* Multiphase milestone (#1222)

* Divide through by density to get velocity form

* Fix VOF velocity flux to finalize

* Fix missing pressure gradient density norm and sharpen interfaces more

* Add diffusion term to VOF

* center droplet and provide velocity

* first version of sloshing tank

* sloshing tank case, can change parameters

* Mass-momentum consistency with initial vel scale and diffusion values

* Remove unused forced mass flux from pressure equation

* clean-up and safe settings for VOF advection

* Density face definition for minimized buoyancy noise

* rho_ref == initial density

* Change initial density to a specified function and introduce wall bcs for VOF

* sloshing tank pressure profile

* user function for generic flat water level in z

* Cleaning

* Balanced Buoyancy Forcing

* Complete milestone implementation

* Full implementation with fixed testing

* Fix allocation of fields

* Formatting

* Documentation of VOF advection scheme

---------

Co-authored-by: whorne <[email protected]>
Co-authored-by: Michael Kuhn <[email protected]>

* Rebase and pressure gradient improvements for VOF

* Fixed interface width work

* Allow slip of VOF at wall bcs

---------

Co-authored-by: psakievich <[email protected]>
Co-authored-by: neilmatula <[email protected]>
Co-authored-by: Ganesh Vijayakumar <[email protected]>
Co-authored-by: 四月是你的谎言 <[email protected]>
Co-authored-by: rcknaus <[email protected]>
Co-authored-by: djglaze <[email protected]>
Co-authored-by: ddement <[email protected]>
Co-authored-by: dcdemen <[email protected]>
Co-authored-by: Jon Rood <[email protected]>
Co-authored-by: itopcuoglu <[email protected]>
Co-authored-by: whorne <[email protected]>
Co-authored-by: Michael Kuhn <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants