SHMEM_LOCKS: MCS implementation of SHMEM LOCKS #11796

vvenkates27 · 2023-06-30T17:45:24Z

Adding MCS algorithm-based implementation for shmem_locks to improve performance for large scale SHMEM applications using locks. MCS lock is now the default algorithm, use the following MCA parameter to disable.
-- Use MCA parameter --mca oshmem_enable_mcs_lock 0 to disable mcs locks and revert to default ticket locking.
-- Use oshmem_api_verbose 10 for debug information on shmem_locks.
@manjugv @brminich @rakhmets @yosefe

oshmem/shmem/c/shmem_test_lock.c

brminich

👍
have some minor comments

brminich · 2023-07-26T07:02:15Z

oshmem/shmem/c/shmem_clear_lock.c

@@ -27,5 +28,10 @@

 void shmem_clear_lock(volatile long *lock)
 {
-    _shmem_clear_lock((void *)lock, sizeof(long));
+	if (oshmem_shmem_mcs_locks) {


general comment: pls use 4 space tabs as per OMPI Coding Style

I have checked and fixed all files updated in this PR to stick to 4 space tabs.

can you please double check? E.g. looks like 8-width tabs are used in this file (and possibly others)

brminich · 2023-07-26T07:04:36Z

oshmem/shmem/c/shmem_clear_lock.c

@@ -27,5 +28,10 @@

 void shmem_clear_lock(volatile long *lock)
 {
-    _shmem_clear_lock((void *)lock, sizeof(long));
+	if (oshmem_shmem_mcs_locks) {
+		SHMEM_API_VERBOSE(10, "shmem_clear_lock using MCS Lock");


maybe move this print to the very beggining of the func and also print locking scheme being cleaned?

I have moved the print and also printing the locking scheme now with the latest commit

brminich · 2023-07-26T07:05:11Z

oshmem/shmem/c/shmem_fetch.c

+    type prefix##_ctx##type_name##_atomic_fetch(shmem_ctx_t ctx,	\
+						const type *target, int pe) \


unrelated change, pls remove

brminich · 2023-07-26T07:07:00Z

oshmem/shmem/c/shmem_lock.c

+        MCA_ATOMIC_CALL(cswap(oshmem_ctx_default, target, (void*)&prev_value,
+			      cond, value, target_size, pe));


pls consider pushing changes in this file in a separate PR, as they are not relavant for the new locking algo

Removed unwanted changes from this PR.

brminich · 2023-07-26T07:08:01Z

oshmem/shmem/c/shmem_mcs_lock.c

+/*
+ * Copyright (c) 2013      Mellanox Technologies, Inc.
+ *                         All rights reserved.
+ * Copyright (c) 2014      Cisco Systems, Inc.  All rights reserved.
+ * Copyright (c) 2015      Research Organization for Information Science
+ *                         and Technology (RIST). All rights reserved.
+ * $COPYRIGHT$
+ *


as this is newly introduced file, i'd guess only relevant Nvidia copyright notice should be present

Fixed the License

brminich · 2023-07-26T07:26:14Z

oshmem/shmem/c/shmem_mcs_lock.c

+		 *  shmem_atomic_add(next, my_pe - NEXT_MASK,
+				     prev_tailpe);


why need this?

No we can remove this. also

brminich · 2023-07-26T07:26:52Z

oshmem/shmem/c/shmem_mcs_lock.c

+		/**
+		 * Wait for predecessor release lock to this PE
+		 * signal to false.
+		 * int curr = shmem_atomic_fetch(next, my_pe);


why needed?

Same here, artifact from previous implementation removed.

brminich · 2023-07-26T07:33:01Z

oshmem/shmem/c/shmem_mcs_lock.c

+		 * prev_value = shmem_atomic_compare_swap(tail, swap_cond, 0,
+		 *					   mcs_tail_owner);


isn't it almost the same what is used below?
same comment is relevant for other similar comments in this file

I have removed this comment.

brminich · 2023-07-26T07:47:58Z

oshmem/shmem/c/shmem_mcs_lock.c

+	shmem_mcs_internal_set_lock(lock);
+}
+
+void _shmem_mcs_clear_lock(long *lock) {
+	shmem_mcs_internal_clear_lock(lock);
+}
+
+int  _shmem_mcs_test_lock(long *lock) {
+	return shmem_mcs_internal_test_lock(lock);


why do we need these wrappers?

Was intended to keep similar to the existing implementation. Doesn't have purpose for this implementation. Removing it.

brminich · 2023-07-26T07:49:47Z

oshmem/shmem/c/shmem_set_lock.c

@@ -27,5 +28,10 @@

 void shmem_set_lock(volatile long *lock)
 {
-    _shmem_set_lock((void *)lock, sizeof(long));
+	if (oshmem_shmem_mcs_locks) {
+		SHMEM_API_VERBOSE(10, "shmem_set_lock using MCS Lock");


maybe such comments are worth to be common for all type of locks? (i.e. to move them to the very beginning of func)

Agreed and moved in the latest commit.

vvenkates27 · 2023-08-03T14:17:24Z

👍 have some minor comments

Thank you for the review. I have fixed all the issues pointed in the comments and uploaded a new patch. Please review and let me know if you have any other comments.

brminich · 2023-08-04T18:48:31Z

oshmem/shmem/c/shmem_clear_lock.c

@@ -27,5 +28,10 @@

 void shmem_clear_lock(volatile long *lock)
 {
-    _shmem_clear_lock((void *)lock, sizeof(long));
+	if (oshmem_shmem_mcs_locks) {


can you please double check? E.g. looks like 8-width tabs are used in this file (and possibly others)

brminich · 2023-08-04T18:49:11Z

oshmem/shmem/c/shmem_mcs_lock.c

+/*
+ * Copyright (c) 2023      NVIDIA Corporation.
+ *                         All rights reserved.
+ *                         and Technology (RIST). All rights reserved.


Yes it was, removed now.

brminich · 2023-08-04T18:50:52Z

oshmem/shmem/c/shmem_mcs_lock.c

+		RUNTIME_CHECK_RC(retv);
+
+		/**
+		 *	This value to be changed eventually by predecessor


I have redone the alignment for the whole file now, hopefully this patch fixes the alignment issues.

brminich · 2023-08-04T18:51:38Z

oshmem/shmem/c/shmem_mcs_lock.c

+	int	next_value	= 0;
+	int	swap_cond	= 0;
+	int	prev_value	= 0;


no need for extra spaces before =

brminich · 2023-08-04T18:52:12Z

oshmem/shmem/c/shmem_mcs_lock.c

+int
+_shmem_mcs_test_lock(long *lockp)
+{
+	lock_t *lock = (lock_t *) lockp;


pls align by = for consistency with other vars in this func

brminich · 2023-08-04T18:54:19Z

oshmem/shmem/c/shmem_mcs_lock.c

+	if (0 == prev_tail)
+		return 0; /** lock acquired successfully */


can you please use {} for every statement, like

Suggested change

if (0 == prev_tail)

return 0; /** lock acquired successfully */

if (0 == prev_tail) {

return 0; /** lock acquired successfully */

}

(we try to use this style for oshmem and ucx pml code)
also relevant for other one line statements in this PR

brminich · 2023-08-04T18:56:17Z

oshmem/shmem/c/shmem_set_lock.c

@@ -27,5 +29,12 @@

 void shmem_set_lock(volatile long *lock)
 {
-    _shmem_set_lock((void *)lock, sizeof(long));
+	SHMEM_API_VERBOSE(10, "shmem_set_lock");


maybe remove this one now?

brminich · 2023-08-04T18:56:37Z

oshmem/shmem/c/shmem_test_lock.c

@@ -28,5 +31,12 @@

 int shmem_test_lock(volatile long *lock)
 {
-    return _shmem_test_lock((void *)lock, sizeof(long));
+	SHMEM_API_VERBOSE(10, "shmem_test_lock");


github-actions · 2023-08-19T14:15:28Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

94dae7c: Update shmem_mcs_lock.c - minor alignment for "="

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2023-08-19T14:16:49Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

4b2a05a: Update shmem_mcs_lock.c - aligning a commit

check_signed_off: does not contain a valid Signed-off-by line

94dae7c: Update shmem_mcs_lock.c - minor alignment for "="

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2023-08-19T14:18:41Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

923785e: Update shmem_test_lock.c - Remove unwanted prints

check_signed_off: does not contain a valid Signed-off-by line

26b3a73: Update shmem_clear_lock.c remove unwanted print.

check_signed_off: does not contain a valid Signed-off-by line

4b2a05a: Update shmem_mcs_lock.c - aligning a commit

check_signed_off: does not contain a valid Signed-off-by line

94dae7c: Update shmem_mcs_lock.c - minor alignment for "="

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2023-08-19T14:19:15Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

923785e: Update shmem_test_lock.c - Remove unwanted prints

check_signed_off: does not contain a valid Signed-off-by line

26b3a73: Update shmem_clear_lock.c remove unwanted print.

check_signed_off: does not contain a valid Signed-off-by line

4b2a05a: Update shmem_mcs_lock.c - aligning a commit

check_signed_off: does not contain a valid Signed-off-by line

94dae7c: Update shmem_mcs_lock.c - minor alignment for "="

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2023-08-19T14:22:17Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

3786663: Update shmem_mcs_lock.c - Some more minor alignmen...

check_signed_off: does not contain a valid Signed-off-by line

923785e: Update shmem_test_lock.c - Remove unwanted prints

check_signed_off: does not contain a valid Signed-off-by line

26b3a73: Update shmem_clear_lock.c remove unwanted print.

check_signed_off: does not contain a valid Signed-off-by line

4b2a05a: Update shmem_mcs_lock.c - aligning a commit

check_signed_off: does not contain a valid Signed-off-by line

94dae7c: Update shmem_mcs_lock.c - minor alignment for "="

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2023-08-19T14:23:47Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

5b47412: Update shmem_mcs_lock.c - removed unwanted lines/s...

check_signed_off: does not contain a valid Signed-off-by line

3786663: Update shmem_mcs_lock.c - Some more minor alignmen...

check_signed_off: does not contain a valid Signed-off-by line

923785e: Update shmem_test_lock.c - Remove unwanted prints

check_signed_off: does not contain a valid Signed-off-by line

26b3a73: Update shmem_clear_lock.c remove unwanted print.

check_signed_off: does not contain a valid Signed-off-by line

4b2a05a: Update shmem_mcs_lock.c - aligning a commit

check_signed_off: does not contain a valid Signed-off-by line

94dae7c: Update shmem_mcs_lock.c - minor alignment for "="

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

github-actions · 2023-08-19T14:49:32Z

Hello! The Git Commit Checker CI bot found a few problems with this PR:

6936a0e: Commits from github online without signed-off opti...

check_signed_off: does not contain a valid Signed-off-by line

5b47412: Update shmem_mcs_lock.c - removed unwanted lines/s...

check_signed_off: does not contain a valid Signed-off-by line

3786663: Update shmem_mcs_lock.c - Some more minor alignmen...

check_signed_off: does not contain a valid Signed-off-by line

923785e: Update shmem_test_lock.c - Remove unwanted prints

check_signed_off: does not contain a valid Signed-off-by line

26b3a73: Update shmem_clear_lock.c remove unwanted print.

check_signed_off: does not contain a valid Signed-off-by line

4b2a05a: Update shmem_mcs_lock.c - aligning a commit

check_signed_off: does not contain a valid Signed-off-by line

94dae7c: Update shmem_mcs_lock.c - minor alignment for "="

check_signed_off: does not contain a valid Signed-off-by line

Please fix these problems and, if necessary, force-push new commits back up to the PR branch. Thanks!

brminich · 2023-08-23T14:31:22Z

@yosefe, pls review

rakhmets · 2023-08-23T11:55:44Z

oshmem/shmem/c/shmem_mcs_lock.c

+/**
+ * @file
+ */


Seems that the comment is not needed here.

rakhmets · 2023-08-23T11:57:14Z

oshmem/shmem/c/shmem_mcs_lock.c

+    /** has meaning only on MCSQ_TAIL OWNER */
+    int tail;
+    /** It has meaning on all PEs */
+    /** The next pointer is a combination of the PE ID and  wait signal */


Suggested change

/** The next pointer is a combination of the PE ID and wait signal */

/** The next pointer is a combination of the PE ID and wait signal */

rakhmets · 2023-08-23T11:59:29Z

oshmem/shmem/c/shmem_mcs_lock.c

+#define SIGNAL_MASK   0x80000000U //Wait signal mask
+#define NEXT(A)       (A & NEXT_MASK)
+#define GET_PE(P)     (P & NEXT_MASK) // Improve readability


Please use C-style comments.

rakhmets · 2023-08-23T12:01:09Z

oshmem/shmem/c/shmem_mcs_lock.c

+     *  releasing it.
+     */
+    /**
+     * Can make this to be shmem_atomic_set to be safe in non-cc architecutres


Suggested change

* Can make this to be shmem_atomic_set to be safe in non-cc architecutres

* Can make this to be shmem_atomic_set to be safe in non-cc architectures

rakhmets · 2023-08-23T12:03:16Z

oshmem/shmem/c/shmem_mcs_lock.c

+    int retv            = 0;
+
+    /**
+     * Can make atomic fetch to be safe in non-cc architecutres


Suggested change

* Can make atomic fetch to be safe in non-cc architecutres

* Can make atomic fetch to be safe in non-cc architectures

rakhmets · 2023-08-23T12:04:35Z

oshmem/shmem/c/shmem_set_lock.c

@@ -9,7 +11,6 @@
 *
 * $HEADER$
 */
-


Please return back the empty line.

rakhmets · 2023-08-31T10:03:26Z

oshmem/shmem/c/shmem_mcs_lock.c

It is better to replace sizeof(int) by sizeof(variable_name) in swap, add, etc.

Actually, swap, add calls are internal MCA calls of shmem_int__ calls. sizeof(int) can help with readability than the variable name.

oshmem/runtime/oshmem_shmem_params.c

oshmem/shmem/c/shmem_clear_lock.c

oshmem/shmem/c/shmem_mcs_lock.c

oshmem/runtime/oshmem_shmem_params.c

brminich · 2023-10-17T13:15:19Z

@vvenkates27 , can you please squash the commits?

Adding MCS algorithm-based implementation for shmem_locks to improve performance for large scale SHMEM applications using locks. MCS lock is now the default algorithm, use the following MCA parameter to disable. --mca oshmem_enable_mcs_lock 0 to disable mcs locks and revert to default ticket locking. --mca oshmem_api_verbose 10 for debug information on shmem_locks. Signed-off-by: Vishwanath Venkatesan <[email protected]>

brminich · 2023-10-18T05:03:54Z

@vvenkates27 can you please port it to 4.1.x and 5.0 branches?

SHMEM_LOCKS: MCS implementation of SHMEM LOCKS Adding MCS algorithm-based implementation for shmem_locks to improve performance for large scale SHMEM applications using locks. MCS lock is now the default algorithm, use the following MCA parameter to disable. --mca oshmem_enable_mcs_lock 0 to disable mcs locks and revert to default ticket locking. --mca oshmem_api_verbose 10 for debug information on shmem_locks. Signed-off-by: Vishwanath Venkatesan <[email protected]>

vvenkates27 · 2023-10-18T06:18:59Z

@vvenkates27 can you please port it to 4.1.x and 5.0 branches?

Submitted pull requests for both branches.

github-actions bot added the Target: main label Jun 30, 2023

bosilca reviewed Jun 30, 2023

View reviewed changes

oshmem/shmem/c/shmem_test_lock.c Outdated Show resolved Hide resolved

brminich reviewed Jul 26, 2023

View reviewed changes

brminich reviewed Aug 4, 2023

View reviewed changes

vvenkates27 force-pushed the upstream_main branch from d0b7901 to 4203927 Compare August 19, 2023 14:52

brminich approved these changes Aug 23, 2023

View reviewed changes

rakhmets reviewed Aug 31, 2023

View reviewed changes

rakhmets approved these changes Sep 26, 2023

View reviewed changes

yosefe reviewed Oct 3, 2023

View reviewed changes

yosefe reviewed Oct 11, 2023

View reviewed changes

oshmem/runtime/oshmem_shmem_params.c Outdated Show resolved Hide resolved

brminich approved these changes Oct 16, 2023

View reviewed changes

vvenkates27 force-pushed the upstream_main branch from 4a4e214 to 1396585 Compare October 18, 2023 04:40

brminich merged commit 67026ef into open-mpi:main Oct 18, 2023
8 checks passed

github-actions bot mentioned this pull request Oct 18, 2023

SHMEM_LOCKS: MCS implementation of SHMEM LOCKS #12002

Merged

		type prefix##_ctx##type_name##_atomic_fetch(shmem_ctx_t ctx, \
		const type *target, int pe) \

		MCA_ATOMIC_CALL(cswap(oshmem_ctx_default, target, (void*)&prev_value,
		cond, value, target_size, pe));

		* prev_value = shmem_atomic_compare_swap(tail, swap_cond, 0,
		* mcs_tail_owner);

		if (0 == prev_tail)
		return 0; /** lock acquired successfully */

	/** The next pointer is a combination of the PE ID and wait signal */
	/** The next pointer is a combination of the PE ID and wait signal */

	* Can make this to be shmem_atomic_set to be safe in non-cc architecutres
	* Can make this to be shmem_atomic_set to be safe in non-cc architectures

	* Can make atomic fetch to be safe in non-cc architecutres
	* Can make atomic fetch to be safe in non-cc architectures

SHMEM_LOCKS: MCS implementation of SHMEM LOCKS #11796

SHMEM_LOCKS: MCS implementation of SHMEM LOCKS #11796

Conversation

vvenkates27 commented Jun 30, 2023 • edited Loading

brminich left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vvenkates27 commented Aug 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Aug 19, 2023

github-actions bot commented Aug 19, 2023

github-actions bot commented Aug 19, 2023

github-actions bot commented Aug 19, 2023

github-actions bot commented Aug 19, 2023

github-actions bot commented Aug 19, 2023

github-actions bot commented Aug 19, 2023

brminich commented Aug 23, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rakhmets Aug 31, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

brminich commented Oct 17, 2023

brminich commented Oct 18, 2023

vvenkates27 commented Oct 18, 2023

vvenkates27 commented Jun 30, 2023 •

edited

Loading

vvenkates27 commented Aug 3, 2023 •

edited

Loading

rakhmets Aug 31, 2023 •

edited

Loading