Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fcoll/vulcan: add two_phase read_all #12894

Merged
merged 3 commits into from
Oct 30, 2024
Merged

fcoll/vulcan: add two_phase read_all #12894

merged 3 commits into from
Oct 30, 2024

Commits on Oct 30, 2024

  1. fcoll/vulcan: cleanup write_all operations 

    This is a cleanup of the write_all operation, in preparation for adding read_all implementation to the vulcan component. Specifically:

- remove the multiple group option: this was envisioned for the vulcan component, but never fully implemented. Therefore, having a few stubs at some locations (and an mca parameter) doesn't make sense, and are being removed to simplify the code.

- remote the write_chunksize option: this was an artifact of the code based having evolved from dynamic_gen2 (where the option makes sense). However, in vulcan it doesn't really make sense and was actually not correctly implemented eitherway, so if somebody would have used that option, it would prbably have failed.

The changed have been validated with the ompio testsuite as well as the hdf5 testphdf5, t_shapesame, and t_filters_parallel tests.

Signed-off-by: Edgar Gabriel <[email protected]>
    @edgargabriel
    edgargabriel committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    6043ac5 View commit details
    Browse the repository at this point in the history

  2. fcoll/vulcan: minor code reorg 

    extract some code into stand alone routines in preparation for adding a vulcan read_all implementation.
Specifically, this pr adds new routines for :
 - mca_fcoll_vulcan_calc_blocklen_disps
 - mca_fcoll_vulcan_calc_file_offsets
 - mca_fcoll_vulcan_calc_io_array

which will be reused in the read_all as well.
Also, some white-space cleanup of the code.

Signed-off-by: Edgar Gabriel <[email protected]>
    @edgargabriel
    edgargabriel committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    4b356be View commit details
    Browse the repository at this point in the history

  3. fcoll/vulcan: add read_all implementation 

    add an implementation of the read_all operation that uses the two-phase I/O algorithm using even partitioning, i.e. the same base idea that is used by the write_all operation of this component.

In addition to using the 'correct' data partitioning approach for the component, the vulcan read_all implementation also adds some other features that were there for the write_all operations, but not for the (generic) read_all algorithm used by all components so far. Specifically, it can overlap the execution of the I/O phase and the communication phase. The algorithm can also use GPU buffers for aggregation.

Signed-off-by: Edgar Gabriel <[email protected]>
    @edgargabriel
    edgargabriel committed Oct 30, 2024
    Configuration menu
    Copy the full SHA
    030ead1 View commit details
    Browse the repository at this point in the history