Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SpectralElement2D with GLL{2} doesn't work on MPI #2098

Open
Sbozzolo opened this issue Dec 13, 2024 · 1 comment
Open

SpectralElement2D with GLL{2} doesn't work on MPI #2098

Sbozzolo opened this issue Dec 13, 2024 · 1 comment
Assignees
@sriharshakandala
Labels
bug Something isn't working

Comments

@Sbozzolo
Copy link
Member

Sbozzolo commented Dec 13, 2024 •
edited
Loading

To replicate

ENV["CLIMACOMMS_CONTEXT"] = "MPI"
import ClimaCore
import ClimaComms
ClimaComms.@import_required_backends
context = ClimaComms.context()
pid, nprocs = ClimaComms.init(context)
FT = Float32
radius = FT(6.3781e6)
nelements = (50, 10)
npolynomial = 1
device = ClimaComms.device()
horzdomain = ClimaCore.Domains.SphereDomain(radius)
horzmesh = ClimaCore.Meshes.EquiangularCubedSphere(horzdomain, nelements[1])
horztopology = ClimaCore.Topologies.Topology2D(context, horzmesh)
quad = ClimaCore.Spaces.Quadratures.GLL{npolynomial + 1}()
horzspace = ClimaCore.Spaces.SpectralElementSpace2D(horztopology, quad)

Run with two MPI processes.

Error:

ERROR: LoadError: ERROR: LoadError: BoundsError: attempt to access BoundsError: attempt to access 300-element Vector{Int64} at index [301]
Stacktrace:
  [1] setindex!
    @ ./array.jl:1021 [inlined]
  [2] compute_ghost_send_recv_idx(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, Nq::Int64)
    @ ClimaCore.Topologies ~/.julia/packages/ClimaCore/rSpyb/src/Topologies/topology2d.jl:609
  [3] create_dss_buffer(data::ClimaCore.DataLayouts.IJFH{Float64, 2, Array{Float64, 4}}, topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ::Type{ClimaCore.DataLayouts.VIFH}, local_geometry::Nothing, local_weights::Nothing)
    @ ClimaCore.Topologies ~/.julia/packages/ClimaCore/rSpyb/src/Topologies/dss.jl:156
  [4] create_dss_buffer (repeats 2 times)
    @ ~/.julia/packages/ClimaCore/rSpyb/src/Topologies/dss.jl:71 [inlined]
  [5] dss!(data::ClimaCore.DataLayouts.IJFH{Float64, 2, Array{Float64, 4}}, topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}})
    @ ClimaCore.Topologies ~/.julia/packages/ClimaCore/rSpyb/src/Topologies/dss.jl:693
  [6] _SpectralElementGrid2D(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, quadrature_style::ClimaCore.Quadratures.GLL{2}, ::Val{7500}, ::Type{ClimaCore.DataLayouts.IJFH}; enable_bubble::Bool)
    @ ClimaCore.Grids ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:421
  [7] _SpectralElementGrid2D(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, quadrature_style::ClimaCore.Quadratures.GLL{2}, horizontal_layout_type::Type; enable_bubble::Bool)
    @ ClimaCore.Grids ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:215
  [8] _SpectralElementGrid2D
    @ ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:215 [inlined]
  [9] #8
    @ ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:196 [inlined]
 [10] get!(default::ClimaCore.Grids.var"#8#9"{UnionAll, Bool, ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ClimaCore.Quadratures.GLL{2}}, h::Dict{Any, Any}, key::Tuple{UnionAll, ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ClimaCore.Quadratures.GLL{2}, Bool, UnionAll})
    @ Base ./dict.jl:479
 [11] 300-element Vector{Int64} at index [301]
Stacktrace:
  [1] setindex!
    @ ./array.jl:1021 [inlined]
  [2] compute_ghost_send_recv_idx(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, Nq::Int64)
    @ ClimaCore.Topologies ~/.julia/packages/ClimaCore/rSpyb/src/Topologies/topology2d.jl:609
  [3] create_dss_buffer(data::ClimaCore.DataLayouts.IJFH{Float64, 2, Array{Float64, 4}}, topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ::Type{ClimaCore.DataLayouts.VIFH}, local_geometry::Nothing, local_weights::Nothing)
    @ ClimaCore.Topologies ~/.julia/packages/ClimaCore/rSpyb/src/Topologies/dss.jl:156
  [4] create_dss_buffer (repeats 2 times)
    @ ~/.julia/packages/ClimaCore/rSpyb/src/Topologies/dss.jl:71 [inlined]
  [5] dss!(data::ClimaCore.DataLayouts.IJFH{Float64, 2, Array{Float64, 4}}, topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}})
    @ ClimaCore.Topologies ~/.julia/packages/ClimaCore/rSpyb/src/Topologies/dss.jl:693
  [6] _SpectralElementGrid2D(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, quadrature_style::ClimaCore.Quadratures.GLL{2}, ::Val{7500}, ::Type{ClimaCore.DataLayouts.IJFH}; enable_bubble::Bool)
    @ ClimaCore.Grids ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:421
  [7] _SpectralElementGrid2D(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, quadrature_style::ClimaCore.Quadratures.GLL{2}, horizontal_layout_type::Type; enable_bubble::Bool)
    @ ClimaCore.Grids ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:215
  [8] _SpectralElementGrid2D
    @ ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:215 [inlined]
  [9] #8
    @ ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:196 [inlined]
 [10] get!(default::ClimaCore.Grids.var"#8#9"{UnionAll, Bool, ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ClimaCore.Quadratures.GLL{2}}, h::Dict{Any, Any}, key::Tuple{UnionAll, ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, ClimaCore.Quadratures.GLL{2}, Bool, UnionAll})
    @ Base ./dict.jl:479
 [11] ClimaCore.Grids.SpectralElementGrid2D(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, quadrature_style::ClimaCore.Quadratures.GLL{2}; horizontal_layout_type::Type, enable_bubble::Bool)
    @ ClimaCore.Grids ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:186
 [12] SpectralElementGrid2D
    @ ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:180 [inlined]
 [13] ClimaCore.Spaces.SpectralElementSpace2D(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, quadrature_style::ClimaCore.Quadratures.GLL{2}; kwargs::@Kwargs{})
    @ ClimaCore.Spaces ~/.julia/packages/ClimaCore/rSpyb/src/Spaces/spectralelement.jl:106
 [14] ClimaCore.Spaces.SpectralElementSpace2D(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, quadrature_style::ClimaCore.Quadratures.GLL{2})
    @ ClimaCore.Spaces ~/.julia/packages/ClimaCore/rSpyb/src/Spaces/spectralelement.jl:101
 [15] top-level scope
    @ ~/repos/ClimaLand.jl/experiments/standalone/Bucket/global_bucket_function.jl:109
in expression starting at /home/sbozzolo/repos/ClimaLand.jl/experiments/standalone/Bucket/global_bucket_function.jl:109
ClimaCore.Grids.SpectralElementGrid2D(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, quadrature_style::ClimaCore.Quadratures.GLL{2}; horizontal_layout_type::Type, enable_bubble::Bool)
    @ ClimaCore.Grids ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:186
 [12] SpectralElementGrid2D
    @ ~/.julia/packages/ClimaCore/rSpyb/src/Grids/spectralelement.jl:180 [inlined]
 [13] ClimaCore.Spaces.SpectralElementSpace2D(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, quadrature_style::ClimaCore.Quadratures.GLL{2}; kwargs::@Kwargs{})
    @ ClimaCore.Spaces ~/.julia/packages/ClimaCore/rSpyb/src/Spaces/spectralelement.jl:106
 [14] ClimaCore.Spaces.SpectralElementSpace2D(topology::ClimaCore.Topologies.Topology2D{ClimaComms.MPICommsContext{ClimaComms.CPUSingleThreaded, MPI.Comm}, ClimaCore.Meshes.EquiangularCubedSphere{ClimaCore.Domains.SphereDomain{Float64}, ClimaCore.Meshes.NormalizedBilinearMap}, CartesianIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, LinearIndices{3, Tuple{Base.OneTo{Int64}, Base.OneTo{Int64}, Base.OneTo{Int64}}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64, Int64, Int64, Bool}}, Vector{Tuple{Int64, Int64}}, Vector{Int64}, Vector{Tuple{Bool, Int64, Int64}}, Vector{Int64}, Vector{Int64}, @NamedTuple{}, Vector{Tuple{Int64, Int64}}}, quadrature_style::ClimaCore.Quadratures.GLL{2})
    @ ClimaCore.Spaces ~/.julia/packages/ClimaCore/rSpyb/src/Spaces/spectralelement.jl:101
 [15] top-level scope
    @ ~/repos/ClimaLand.jl/experiments/standalone/Bucket/global_bucket_function.jl:109
in expression starting at /home/sbozzolo/repos/ClimaLand.jl/experiments/standalone/Bucket/global_bucket_function.jl:109

This also happen with 3 MPI processes
@Sbozzolo Sbozzolo added the bug Something isn't working label Dec 13, 2024
@Sbozzolo
Copy link
Member Author

The issue only happens with npolynomial = 1, possibly due to this line?

ClimaCore.jl/src/Topologies/topology2d.jl

Line 575 in 340603b

comm_lengths = comm_vertex_lengths .+ (comm_face_lengths .* nfacedof)

@Sbozzolo Sbozzolo changed the title BoundsError in constructing space with MPI processes SpectralElement2D with GLL{2} doesn't work on MPI Dec 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@sriharshakandala sriharshakandala

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@Sbozzolo @sriharshakandala