Multiphase stabilization and mass-momentum consistency updates

[wjhorne]

This PR contains all of the multiphase work that was used to successfully run hybrid cases of waves passing over solid bodies. The main changes include

• mass-momentum consistency via modifying the mass flux when vof is used. This was found to be a substantial improvement for moving interfaces over long runs.
• upwinding stabilization with switching in the momentum equation at interfaces. This was a critical addition to ensure stability for high CFLs and harsh flow events that are encountered when waves pass over bodies.

[wjhorne]

@psakievich Do you know anyone that could take a look at this to start getting the ball rolling on getting it in?

[psakievich]

I will try to take a look but I'm not sure how timely I can be. @BumseokLee would you also be willing to take a look? @mbkuhn your input would also be helpful. Also @rcknaus if you have some spare time your input is always appreciated. We have a P/T you can charge for your time if needed.

[BumseokLee]

Hi Phil,
I am on a business trip until July 6. Also, I don’t have experiences with the multiphase stabilization or mass-momentum consistency in Nalu-Wind. I think it would be better to assign someone else to the work.
Bumseok

[psakievich]

@wjhorne will you add some regression tests that utilize the user functions you've added?

[marchdf]

Great work! A bunch of minor comments.

[wjhorne]

@wjhorne will you add some regression tests that utilize the user functions you've added?

We have fully-coupled tests that utilize the user functions. The only exception is the droplet case, which is also covered by VOFDroplet in our regression tests.

[wjhorne]

I pushed in changes addressing all of the review here. Let me know how it looks and I'll keep iterating

[mbkuhn]

I'm getting a weird error with this PR. I'm using it to run these reg tests. When tpetra is used for the velocity solver, the reg tests run fine. But with hypre instead, I get the error "terminating due to uncaught exception of type std::runtime_error: HypreLinearSystem::sumInto not yet implemented for (NON) overset constaint algorithms. Exiting." What makes this strange is the other (single-phase) reg tests use hypre for velocity with no issues, and these two-phase cases run fine with hypre for the volume_of_fluid equation.

[mbkuhn]

I'm getting a weird error with this PR...

This doesn't have to do with any of the changes in the PR. Running one of the cases with master (and turning off the unavailable user function sloshing_tank) still leads to the error. It seems to be a problem with the velocity + volume_of_fluid equations together, not separately (because the zalesak reg test runs fine).

[psakievich]

@mbkuhndo those single phase tests use overset? Here is where the error comes from:

nalu-wind/src/HypreLinearSystem.C

Lines 2399 to 2403 in 53f4939

	} else {
	throw std::runtime_error(
	"HypreLinearSystem::sumInto not yet implemented for (NON) "
	"overset constaint algorithms. Exiting.");
	}

looking at the calls of sumInto it seems it would hit with the computing overset pressure call for this case here:

nalu-wind/src/overset/AssembleOversetPressureAlgorithm.C

Lines 184 to 185 in 53f4939

	eqSystem_->linsys_->sumInto(
	connected_nodes, scratchIds, scratchVals, rhs, lhs, __FILE__);

Also worth checking if the other cases are using a segregated velocity solve. It may use a different path. Looks like the HypreUVW solver has this as a null function.

nalu-wind/include/HypreUVWLinearSystem.h

Lines 171 to 193 in 53f4939

	virtual void sumInto(
	const std::vector<stk::mesh::Entity>& /*sym_meshobj*/,
	std::vector<int>& /*scratchIds*/,
	std::vector<double>& /*scratchVals*/,
	const std::vector<double>& /*rhs*/,
	const std::vector<double>& /*lhs*/,
	const char* /*trace_tag*/)
	{
	}
	/*****************************************/
	/* Legacy methods needed for Compilation */
	/*****************************************/
	virtual void sumInto(
	unsigned /*numEntities*/,
	const stk::mesh::NgpMesh::ConnectedNodes& /*entities*/,
	const SharedMemView<const double*, DeviceShmem>& /*rhs*/,
	const SharedMemView<const double**, DeviceShmem>& /*lhs*/,
	const SharedMemView<int*, DeviceShmem>& /*localIds*/,
	const SharedMemView<int*, DeviceShmem>& /*sortPermutation*/,
	const char* /*trace_tag*/)
	{
	}

I don't have a great mental model of how all these go together though.

[mbkuhn]

Update: the problem is not the VOF algorithm or even overset. It's the buoyancy term.

None of the other overset reg tests contain the buoyancy term, and typical runs with nalu-wind tend to employ the buoyancy_boussinesq term, which has no issue with hypre solvers. To check, I ran the problem exawind-driver PR reg tests with the buoyancy term turned off (no error). I looked into the nalu-wind reg tests to find tests with the plain "buoyancy" source term, I replaced the tpetra solver with hypre for velocity, and these tests generated the same sumInto error. These were the VOFDroplet and periodic3dEdge reg tests. All of these checks were performed with the master branch of nalu-wind, not this PR.

Should I make an issue for this?

[wjhorne]

@psakievich I believe with the latest push it should be good to go for a merge. Let me know what you think

[mbkuhn]

The big difference in gold norms is not because of changing the VOFDroplet setup (i.e., using input arguments instead of hard-coded parameters). It's because of using a much smaller interface thickness, which Wyatt chose -- I'm assuming with the intent of stressing the new stabilization. With the new stabilization but the old thickness, the norms are very close to the originals: 1.768e-5, 6.628e-6, 3.367e-6.

[psakievich]

Thanks for the update. This is good to know.

For better or worse, we don't really track/commit the norms any more due to the wide differences we see between platforms/compilers/architectures. The span is too big for comfort for these general integrated norms. I suppose we could do it with the CI container now as a constant platform for evaluation, but the infrastructure for that is not currently in place.

[mbkuhn]

Good to know! Thanks, I wasn't sure.

[mbkuhn]

We have fully-coupled tests that utilize the user functions. The only exception is the droplet case, which is also covered by VOFDroplet in our regression tests.

SloshingTank is/will be covered by reg tests for exawind-driver; I added a VOFInertialDroplet case so the droplet velocity user function would be covered.