Only users with active ALCF accounts can sign in to the Account and Project Management website. If you have an active account, verify that you are using the correct ALCF username. Note that username is case-sensitive. If you forgot your username, contact support@alcf.anl.gov. For passcode token issues, please review the troubleshooting information on this page: Passcode Tokens.
+If your account is inactive, please submit a reactivation request here: https://my.alcf.anl.gov/accounts/#/accountReactivate.
+If you never had an ALCF account, please apply for one here: https://my.alcf.anl.gov/accounts/#/accountRequest. Note that all ALCF accounts must be associated with a project with an active allocation.
+There are 3 allocation opportunities at ALCF. Please see How to Get an Allocation on how to get time on our systems.
+To request an extension of your existing discretionary allocation or to request additional hours, please email support@alcf.anl.gov with answers to the following or fill out the form at request an extension/additional hours: +- What you have accomplished with your original allocation? + - Please include a brief description of any publications or major presentations that were (or will be) generated in full or in part because of this allocation. +- What you will do with the extra time? +- What you are requesting as your new expiration date? +- How many additional hours you are requesting?
+To join a project, please go to https://my.alcf.anl.gov/, then click "join a project". Once there, scroll down to the project you want to join and click on it. At the bottom of the next page, please click on the "Request Membership" button. Once we receive approval from the PI regarding your membership request, we will provide you with access to the necessary resources.
+Reservation requests must include information detailed here:
+Note: All ALCF accounts must be associated with an allocated project.
+Please forward your account expiry email to your Sponsor. As soon as we receive an approval email from your Sponsor, we'll proceed with your account renewal process as needed.
+If you are planning to extend this assignment/computer user account, please let us know, so a new 593 (Foreign Visit & Assignment Request form) will be filed for you using the information from before. In case any other documents are needed from your end, you'll be contacted as necessary. In order to allow sufficient time for an indices check, it is recommended that your response be submitted as soon as possible.
+If you are not planning to extend your account, also let us know so that we may close out your records.
+ + + + + + + + + + + + + + + +Please note: An account can be associated with a single token only (Mobile or Physical token). Please contact accounts@alcf.anl.gov to change your token preference.
+The SafeNet MobilePass+ Mobile Token allows access to ALCF systems. This security mobile token uses one-time passwords combined with your PIN for controlled access to the login systems. The mobile token utilizes an app that is keyed to your user account and for which you are responsible on your Android, iPhone or Windows mobile device. Please safeguard your phone as you would your credit cards or house keys: Do not store username, PIN, or other account-related records with the token. Sharing of mobile tokens is strictly forbidden. A mobile token can be associated with a single device only.
+The SafeNet MobilePASS+ app turns your device into a two-factor authentication device, removing the need to carry an additional hardware token. As a SafeNet MobilePASS+ user, you can generate passcodes on your mobile device and use those passcodes to authenticate on ALCF computing resources. See supported OS and platforms for more information.
+After you’ve been provisioned a mobile token, you will receive a notification email with the subject line "ALCF Mobile Token Self-Enrollment" which you must access from the device on which you wish to install the token.
+Auto-Enrollment (to enroll SafeNet MobilePass+ token automatically):
+Manual Enrollment:
+When prompted for a password, click the SafeNet MobilePASS+ app on your phone. Click on the token name listed within the app, and enter your PIN.
+The app will display your passcode immediately. Enter the passcode as the login password for the system within the SSH session. Please Note: You do NOT have to enter the PIN on the SSH screen when logging into a resource. This only needs to be done to access the passcode within the SafeNet MobilePASS+ app.
+Each generated passcode is valid on the SafeNet MobilePass+ app window until your mobile device screen times out.
+Case 1: Forgotten PIN: If you enter a PIN for your mobile Token and you get an invalid PIN, you will be asked to re-enter your PIN. After 6 failed attempts your token will be deleted and you will need to call the ALCF help desk or send an email to ALCF support to have a new mobile token provisioned.
+Case 2: Account Lockout: If you fail to enter the correct password 6 times, you will get a permission denied error on the SSH screen. Upon 4 more failed attempts, your IP will be blocked. You will need to call the ALCF help desk and submit a ticket to have the IP unblocked.
+Case 3: PIN Change: While logged in to the mobile token, click on token settings then tap change PIN. Enter the current PIN followed by the new PIN and confirm.
+Case 4: Re-Sync: If you are unable to log in to a resource after entering the correct PIN and passcode your token may be out of sync with the server. Please email ALCF Service Desk at accounts at alcf.anl.gov for assistance.
+Case 5: New Mobile Device: If you have a new mobile device, please email the ALCF Service Desk at accounts at alcf.anl.gov to have a new mobile token provisioned.
+The physical token allows access to the ALCF systems. This security token uses one-time passwords combined with your PIN for controlled access to the login systems. The physical token is a tracked asset for which you are responsible and is keyed to your use. Please safeguard your token as you would your credit cards or house keys: Do not store username, PIN, or other account-related records with the token. Sharing of tokens is strictly forbidden. Please do not mark on the token or alter it in any way.
+Upon receipt of CRYPTOCard token, contact accounts@alcf.anl.gov to verify your identity and activate the token. If this step is not performed the CRYPTOCard token will not be able to log on to the ALCF resource.
+ALCF Accounts Service Desk Info +Hours: Monday-Friday 9 a.m. - 5 p.m. (Central time); +Email: accounts@alcf.anl.gov
+When the physical token is activated, an initial PIN will be provided. This will be a four-digit number that will prepend to the one-time password string generated by the token.
+Upon INITIAL login (to one of the ALCF machines), a prompt to change the PIN will appear. PINs must be at least four characters long and must only contain numbers.
+Initiate an SSH session using: +
+A password prompt will be received. At this point, push the button on the physical token once.
+An eight-character, one-time password made up of letters and numbers will appear on the token’s display. This one-time password is case-sensitive.
+Type your PIN followed immediately by the one-time password at the SSH password prompt.
+For example, if your PIN is 1234 and you received the one-time password string ABCD9876, you would type 1234ABCD9876 at the password prompt.
+Case 1: It says "locked": The physical token may be locked due to too many failed attempts. Please contact the ALCF Help Desk to return the defective token and so a replacement can be sent.
+Case 2: You have a PIN for your physical token: Once a PIN has been set for your physical token, you will need to prepend your PIN to the token password. Otherwise you will not be able to log in. If you do not remember your PIN, please email us so we can verify your identity and reset your Initial PIN.
+Case 3: It does not say "locked" but still does not work: It is likely that your token has fallen out of sync with the server. If you have pushed the button on your physical token more than 10 times without successfully logging in, it will fail to authenticate because it has lost synchronization with the server. Please try connecting to Polaris first. If it still fails, please follow the re-sync instructions below.
+If you have pushed the button on your physical token more than 10 +times, it will fail to authenticate because it has lost synchronization +with the server. You can re-synchronize your token using the following procedure:
+1. Have your physical token ready.
+
+2. Obtain a challenge sequence:
+ - Initiate an SSH session to a host that allows token
+ authentication (such as polaris.alcf.anl.gov). At the password
+ prompt, just hit 'Enter'. This will cause the Cryptocard service
+ to produce a challenge string consisting of 8 numbers.
+
+3. Hold down the button on your token for a few seconds until the
+ display says "Init", then let go.
+
+4. The token will scroll through a series of menu options. When it
+ displays "ReSync", hit the button again.
+
+5. The display will say
+
+ Resync?0
+
+6. The number at the end will start cycling from 0 to 9, over and over.
+
+7. Look at the numbers in your challenge string. When the number
+ displayed on your token changes to the first number of the challenge
+ string, press the button. The display will now show this number, and
+ the second digit will start cycling.
+
+8. Enter each of the numbers from your challenge string in the same
+ manner, until the display on your token matches the entire challenge string.
+ Choose the "<" to backspace and re-enter the previous number if
+ necessary.
+
+9. Once you've entered all 8 digits, re-check to make sure they're
+ accurate. Then, while all 8 digits are displayed on the token, press
+ the button to generate a new password.
+
+10. Enter your PIN followed by the new password, and hit 'Enter'.
+ If successful, you will be logged in to the resource. You're now back
+ in sync with the authentication server.
+
+If you are unsuccessful, you will be presented with another challenge string.
+At this point, you may need to perform the re-sync instructions again.
+If there are still problems after completing the re-synchronization procedures, please email us at accounts@alcf.anl.gov so we can run a test on the physical token to determine if it is defective.
+If it is found to be defective we will promptly replace it. Physical tokens are the property of Argonne National Laboratory.
+Please return them to us at:
+ALCF Help Desk
+Argonne National Laboratory
+9700 S. Cass Ave.
+Bldg. 240, Rm. 2129
+Lemont, IL 60439
+Please email us at support at alcf.anl.gov for PIN resets. Once your identity has been verified, we will provide you with a new PIN for your CRYPTOcard token.
+If you no longer need your physical token, please return it to this address:
+ + + + + + + + + + + + + + + + +All computing carried out on the ALCF systems is associated with a user "account." This account is used to log onto the login servers and run jobs on the resources. If someone has a user account, then he or she has a login name that is recorded in the user database. This web page describes the process that users will need to understand to manage account details, including policies and procedures.
+If you need an account, visit the Accounts and Project Management website: Request an account
+If you want to learn how to get started, visit the Get Started Guide: Get Started Guide
+Those who are interested in having an account on a ALCF resource must first request an allocation and provide a detailed description of the work, including computational requirements and coding capabilities for the Blue Gene platform. Another means of acquiring an allocation on the ALCF system is to be part of a project team that already has an active allocation. Once an allocation has been granted, new users should complete an account request. A project’s Principal Investigator (PI) must sponsor these accounts—if the PI is the user, an ALCF staff member must serve as sponsor. Sponsors are asked annually to evaluate the accounts they have sponsored to determine whether or not these accounts should be kept active.
+A user with an active account can login to the ALCF login servers (e.g., polaris.alcf.anl.gov) This account will have some home directory space, where file transfer can occur from that space via the login nodes, and where development activities, such as editing and compiling, can also occur.
+Accounts are classified in one of the following categories:
+Allocations require management – balance checks, resource allocation, requesting more time, etc.
+To determine if there is an active allocation, check Job Submission.
+For information on how to run the query, look at our documentation on our sbank Allocations Accounting System or email support@alcf.anl.gov and ask for all active allocations.
+To determine which platforms have an active balance, check our allocation accounting system sbank.
+Projects and allocations at the ALCF are different. A particular project might have multiple allocations of time. For example, a discretionary project that has been approved for more than 3 times will have 3 allocations (2 are probably expired) but just one project. Projects will not expire, allocations will. If allocations are expired, or have no hours left, jobs will not be able to run. Use the two bullets above (Checking for an active allocation and Determining the balance of an allocation) to determine active allocations.
+To request an extension of your existing discretionary allocation or to request additional hours, please email support@alcf.anl.gov with answers to the following:
+Suballocations let PIs control who in their team can runs jobs, how much they are allowed to consume (allocation amount), and when they are allowed to run jobs (start and end dates)
+Step 1: Create Suballocations (Project PI):
+PI creates suballocations
+sbank new sub <allocationid> **-name <nameofsuballoc>
Tip: see sbank new suballocation -h for all the options.
Step 2: Manage Suballocations (Project PI)
+PI adds users to suballocations
+sbank e sub <projectname>::<nameofsuballoc> --add-user="<username1> <username2> ..."
PI can change the name of a suballocation
+sbank e sub <suballocationID> --name=<new_name_of_suballocation>
By default, the primary suballocation (which is the default suballocation created when the allocation is created by ALCF) is unrestricted .i.e. enabled for all project members. That means all project members can submit jobs against the primary suballocation by default. All other suballocations are restricted by default and users have to be added for each of them.
+To change the default for the primary suballocation to restrict usage, PI must first edit the suballocation:
+sbank-edit-suballocation --restrict <primary suballocation id>
Then add users with this command:
+sbank e sub <primary suballocation id> --add-user="<username1> <username2> ..."
PI changes start and end dates for a suballocation:
+sbank e sub <suallocationID> -S <start_date> -E <end_date>
PI adds hours to a suballocation:
+sbank e sub <projectname>::<nameofsuballoc> --hours-to-move <hours> --to-suballocation <projectname>::<nameofsuballoc2>
Note: hourstomove must be greater than or equal to the available balance for the suballocation nameofsuballoc
Tip: see sbank e suballocation -h for all the options
Step 3: Submit Jobs (Project team)
+Submit jobs to a suballocation. Note that the user should be on the suballocation’s user list
+Eg: qsub -l select=10,walltime=30:00,filesystems=eagle:home -A <suballoctionID> -q demand test.sh
Note: Once submanagement is enabled for a project allocation, all job submissions must specify the suballocationID
Useful commands: +List all suballocations for a project that shows number of jobs run, charges, allocation balance, suballocation name, and list of users
+sbank-list-allocations -r polaris -p <projectname> -f”+subname users_list”
Tip: see sbank l a -h for all the options and sbank –f\? for list of fields that can be displayed
Detail allocation information.
+NOTE:
+ 1. The list of
show program's version number and exit
+show this help message and exit
+filter on allocation id
+filter on event id
+FIELD_INFO is
filter on jobid
+set number of fields to display
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on transaction id
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+"FIELD_INFO" FIELD_INFO is
[OPER1]
Operator Defaults:
+**Date Parsing Precedence: **
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions), ...
+also get inactive allocations
+only inactive allocations
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+transaction types: CHARGE, REFUND, PULLBACK, DEPOSIT, VOID
+filter on award type name
+filter on award category
+filter on Clusterbank reference id
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+SILENT, MUCH_LESS, LESS, MORE, VERBOSE, DEBUG, DEBUG1, DEBUG2
+also get deleted objects
+only deleted objects
+only show list info that have charges regardless of project/user relationship
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+remove commas from comma separated thousands
+do not display the header
+do not show history information
+do not display the row data
+do not display system message
+do not display the totals
+ + + + + + + + + + + + + + + +Detail job information. +NOTE:
+--version
+show program's version number and exit
+-h, --help
+show this help message and exit
+-a ALLOCATION_ID, --allocation-id=ALLOCATION_ID
+filter on allocation id
+-e EVENT_ID, --event-id=EVENT_ID
+filter on event id
+-f FIELD_INFO, --field-to-display=FIELD_INFO
+FIELD_INFO is
-j JOBID, --jobid=JOBID
+filter on jobid
+-n NUM_FIELDS_TO_DISPLAY, --num-fields-to-display=NUM_FIELDS_TO_DISPLAY
+set number of fields to display
+-p PROJECT, --project=PROJECT
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+-r RESOURCE, --resource=RESOURCE
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+-t TRANSACTION_ID, --transaction-id=TRANSACTION_ID
+filter on transaction id
+-u USER, --user=USER
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+-w "FIELD_INFO", --field-width
+"FIELD_INFO" FIELD_INFO is
-E END, --end=END
+[OPER1]
-H, --human-readable
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+-S START, --start=START
+[OPER1]
-T TRANSACTION_TYPE, --transaction-type=TRANSACTION_TYPE
+transaction types: CHARGE, REFUND, PULLBACK, DEPOSIT, VOID
+--created=CREATED_TIMESTAMP
+[OPER1]
--debug=DEBUG_LEVEL
+SILENT, MUCH_LESS, LESS, MORE, VERBOSE, DEBUG, DEBUG1, DEBUG2
+--eligible=ELIGIBLE_TIMESTAMP
+[OPER1]
--get-not-charged
+only un-charged jobs
+--history-date-range=END
+[OPER1]
--last-updated=LAST_UPDATED_TIMESTAMP
+[OPER1]
--no-commas
+remove commas from comma separated thousands
+--no-header
+do not display the header
+--no-history
+do not show history information
+--no-rows
+do not display the row data
+--no-sys-msg
+do not display system message
+--no-totals
+do not display the totals
+--queued=QUEUED_TIMESTAMP
+[OPER1]
Detail project information.
+NOTE:
+ 1. The list of
show program's version number and exit
+show this help message and exit
+filter on allocation id
+FIELD_INFO is
set number of fields to display
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+"FIELD_INFO" FIELD_INFO is
[OPER1]
Operator Defaults:
+Date Parsing Precedence: + - YEAR then MONTH then DAY, i.e., 121101 is parsed as YYMMDD, hence Nov. 1, 2012
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+get inactive allocations
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+SILENT, MUCH_LESS, LESS, MORE, VERBOSE, DEBUG, DEBUG1, DEBUG2
+only show list info that have charges regardless of project/user relationship
+remove commas from comma separated thousands
+do not display the header
+do not display the row data
+do not display system message
+do not display the totals
+ + + + + + + + + + + + + + + +Detail transaction information.
+NOTE:
+ 1. The list of
show program's version number and exit
+show this help message and exit
+filter on allocation id
+display comment
+filter on event id
+FIELD_INFO is
filter on jobid
+set number of fields to display
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on transaction id
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+"FIELD_INFO" FIELD_INFO is
[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+transaction types: CHARGE, REFUND, PULLBACK, DEPOSIT, VOID
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+filter on Clusterbank reference id
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+SILENT, MUCH_LESS, LESS, MORE, VERBOSE, DEBUG, DEBUG1, DEBUG2
+remove commas from comma separated thousands
+do not display the header
+do not display the row data
+do not display system message
+do not display the totals
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+Detail user information.
+**NOTE: **
+ 1. Use -I to include inactive allocations
+ 2. the list of
show program's version number and exit
+show this help message and exit
+filter on allocation id
+FIELD_INFO is
set number of fields to display
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+"FIELD_INFO" FIELD_INFO is
[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+get inactive allocations
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+SILENT, MUCH_LESS, LESS, MORE, VERBOSE, DEBUG, DEBUG1, DEBUG2
+only show list info that have charges regardless of project/user relationship
+remove commas from comma separated thousands
+do not display the header
+do not display the row data
+do not display system message
+do not display the totals
+ + + + + + + + + + + + + + + +Detail Meta Command
+enter allocation id
+enter comment for new or edit commands, display comment for list commands
+enter event db id; event db id is an internal id created by the charging system
+enter
command line help
+enter jobid; jobid is created by the scheduler and is not unique
+enter number of fields to display
+enter name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+enter name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+enter suballocation id
+enter transaction id
+enter name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+enter the field width as follows:
enter end datetime filter
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+include inactive allocations
+get only inactive allocations
+enter start datetime filter
+enter type of transaction
+for list allocations | projects | users, only show info with charges
+enter transaction created datetime filter
+enter allocation award category
+enter allocation award-type name
+enter created datetime filter
+enter debug level
+get deleted objects
+get jobs that have not been charged
+get only deleted objects
+enter history datetime filter
+enter last updated datetime filter
+remove commas from comma-separated thousands
+do not display header
+do not display history information
+do not display rows
+do not display system message
+do not display totals
+enter queued datetime filter
+ + + + + + + + + + + + + + + +Below is a set of helpful commands to help you better manage the projects you have running at the ALCF.
+Use this command to list all of your active allocations for a specific project [Project-X]. This is useful when you need to provide this information in a report. +
> sbank-list-allocations -p ProjectX -r all
+ Id Start End Resource Project Jobs Charged Available Balance
+ --------- ---------- ---------- --------- --------------- ---------- --------------- -----------------
+ 2106 2016-01-04 2017-01-01 cooley ProjectX 1,139 6,032.8 43,967.2
+ 2146 2016-01-14 2017-01-10 theta ProjectX 983 1,084,770.3 25,483,927.5
+ 6438 2020-09-22 2022-01-01 thetagpu ProjectX 3 0.0 2,000.0
+
+
+Totals:
+ Rows: 3
+ Cooley:
+ Available Balance: 43,967.2 node hours
+ Charged : 6,032.8 node hours
+ Jobs : 1,139
+ Theta:
+ Available Balance: 25,483,927.5 node hours
+ Charged : 1,084,770.3 node hours
+ Jobs : 983
+ Thetagpu:
+ Available Balance: 2,000.0 node hours
+ Charged : 0.0 node hours
+ Jobs : 3
+> sbank-list-allocations -p ProjectX -r eagle
+ Allocation Suballocation Start End Resource Project Quota
+ ---------- ------------- ---------- ---------- -------- ----------- -----
+ 6687 6555 2020-12-16 2022-01-01 eagle ProjectX 1.0
+
+Totals:
+ Rows: 1
+ Eagle:
+ Quota: 1.0 TB
+
+> sbank-list-allocations -p ProjectX -r eagle
+ Allocation Suballocation Start End Resource Project Quota
+ ---------- ------------- ---------- ---------- -------- ----------- -----
+ 6688 6556 2020-12-16 2022-01-01 eagle ProjectX 1.0
+
+Totals:
+ Rows: 1
+ Eagle:
+ Quota: 1.0 TB
+> sbank-list-allocations --created "<20150101" -r all -p ProjectX "-f created"
+ Created
+ ----------
+ 2016-01-04
+ 2016-01-14
+ 2016-01-15
+
+Totals:
+ Rows: 3
+Date filters (UTC): created < "2015-01-01 00:00:00",
+shrubbery~ > sbank-list-allocations -r all -p ProjectX -f "+created"
+ Id Start End Resource Project Jobs Charged Available Balance Created
+ --------- ---------- ---------- --------- --------------- ---------- --------------- ----------------- ----------
+ 279 2011-08-30 2020-01-01 theta ProjectX 6,361 12,332,699.9 -12,332,699.9 2013-02-22
+ 2106 2016-01-04 2017-01-01 cooley ProjectX 1,150 6,080.9 43,919.1 2016-01-04
+
+Totals:
+ Rows: 2
+ Theta:
+ Available Balance: -12,332,699.9 node hours
+ Charged : 12,332,699.9 node hours
+ Jobs : 6,361
+ Cooley:
+ Available Balance: 43,919.1 node hours
+ Charged : 6,080.9 node hours
+ Jobs : 1,150
+> sbank-list-allocations -f "?"
+available fields:
+ id
+ start_timestamp
+ end_timestamp
+ resource
+ project_name
+ jobs_count
+ charged_sum
+ available_balance_sum
+ created_timestamp
+ award_category
+ award_type_name
+ admin_name
+ cbank_ref
+ comment
+List all charges for userx on theta on project ProjectX +
> sbank-list-users -p ProjectX -r theta -u userx
+ User Jobs Charged
+ --------------- ---------- ---------------
+ userx 1,814 9,884.5
+
+Totals:
+ Rows: 1
+ Resources: theta
+ Charged: 9,884.5 node hours
+ Jobs : 1,814
+ ```
+
+### List charges for all users in ProjectX on Cooley.
+This works for project leads (i.e. PIs, Co-PIs, Proxies), since they can see everything in their own projects.
+++sbank-list-users -p ProjectX -r theta + User Jobs Charged
+
user1 120 4,243.7 + user2 0 0.0 + user3 0 0.0 + user4 181 1,195.5 + user5 0 0.0 + user6 2,560 10,868.7 + user7 0 0.0 + user8 0 0.0 + user9 0 0.0 + user10 7 3.5 + user11 0 0.0
+Totals: + Rows: 11 + Resources: theta + Charged: 16,311.4 node hours + Jobs : 2,868
+## View your project's jobs
+List jobs for user "userx" for jobs that started in the range 2016-02-15<= started < 2016-02-29 and add the transactions related to the job
+
+### **Command:** sbank-list-jobs
+
+**Note:** The job with the refund ```transaction_ids_list field can be shorten all the way to "t" in the -f "+ t"```
+1013857 730417 theta ProjectX 1740 userx 1:53:07 61,776.8 CHARGE-1011230
+ 1013860 730558 theta ProjectX 1740 userx 1:53:07 61,776.8 CHARGE-1011233
+ 1014168 730668 theta ProjectX 1740 userx 1:53:25 61,940.6 CHARGE-1011541
Totals: + Rows: 3 + Theta: + Charged : 185,494.2 node hours + Duration : 6:44:00 +Date filters (UTC): "2016-02-15 00:00:00" <= start < "2016-02-29 00:00:00", +
### List the nodes used, runtime and start timestamp for Cooley job 744160
+**Note**: To display the date and time we increased the the number of characters of start_timestamp to 19
+## View your project's transactions
+### **Command:** sbank-list-transactions
+
+List of transactions that where at or after 2016-02-29 for ProjectX add fields: job_duration, nodes_used and hosts
+
+**Note**:
+- job_duration, nodes_used and hosts are shorten, but they are still uniquely identified
+- host has the left justified width of 20, specified as "h:-20"
+1025426 theta ProjectX 2147 2016-02-29 userx CHARGE 48,005.1 740587 1:27:54 2048 MIR-00800-33BF1-2048 + 1028046 theta ProjectX 2147 2016-03-01 userx CHARGE 147,647.1 742090 4:30:21 2048 MIR-40000-733F1-2048 + 1028755 theta ProjectX 2147 2016-03-02 userx CHARGE 1,576,068.0 742126 6:00:44 16384 MIR-04000-77FF1-1638
+Totals:
+ Rows: 3
+ Theta:
+ Charges Amount: 1,771,720.2 node hours
+ Job Duration : 11:58:98
+Date filters (UTC) : at >= "2016-02-29 00:00:00",
+```
Generate allocation list report.
+Notes: +1. Use -I to include inactive allocations +2. enter "-r all" to get information for all resources
+show program's version number and exit
+show this help message and exit
+filter on allocation id
+display comment
+filter on event id
+FIELD_INFO is
filter on jobid
+set number of fields to display
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+filter on transaction id
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+"FIELD_INFO" FIELD_INFO is
[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+get inactive allocations
+only inactive allocations
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+transaction types: CHARGE, REFUND, PULLBACK, DEPOSIT, VOID
+filter on award-type name
+filter on award category
+filter on Clusterbank reference id
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+SILENT, MUCH_LESS, LESS, MORE, VERBOSE, DEBUG, DEBUG1, DEBUG2
+get deleted objects
+get only deleted objects
+only show list info that have charges regardless of project/user relationship
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+remove commas from comma-separated thousands
+do not display the header
+do not display the row data
+do not display system message
+do not display the totals
+ + + + + + + + + + + + + + + +Generate job list report Note: To get information for all resources, enter "-r all".
+show program's version number and exit
+show this help message and exit
+filter on allocation id
+filter on event id
+FIELD_INFO is
filter on jobid
+set number of fields to display
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on transaction id
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+"FIELD_INFO" FIELD_INFO is
[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+transaction types: CHARGE, REFUND, PULLBACK, DEPOSIT, VOID
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+SILENT, MUCH_LESS, LESS, MORE, VERBOSE, DEBUG, DEBUG1, DEBUG2
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+get only jobs that have not been charged
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+remove commas from comma-separated thousands
+do not display the header
+do not display the row data
+do not display system message
+do not display the totals
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+Generate project list report.
+Notes:
+show program's version number and exit
+show this help message and exit
+filter on allocation id
+FIELD_INFO is
set number of fields to display
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+"FIELD_INFO" FIELD_INFO is
[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+get inactive allocations
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+SILENT, MUCH_LESS, LESS, MORE, VERBOSE, DEBUG, DEBUG1, DEBUG2
+only show list info that have charges regardless of project/user relationship
+remove commas from comma-separated thousands
+do not display the header
+do not display the row data
+do not display system message
+do not display the totals
+ + + + + + + + + + + + + + + +Generate transaction list report.
+Note: To get information for all resources, enter "-r all".
+show program's version number and exit
+show this help message and exit
+filter on allocation id
+display comment
+filter on event id
+FIELD_INFO is
filter on jobid
+set number of fields to display
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+filter on transaction id
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed but only on names
+"FIELD_INFO" FIELD_INFO is
[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+transaction types: CHARGE, REFUND, PULLBACK, DEPOSIT, VOID
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+filter on Clusterbank reference id
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+SILENT, MUCH_LESS, LESS, MORE, VERBOSE, DEBUG, DEBUG1, DEBUG2
+remove commas from comma-separated thousands
+do not display the header
+do not display the row data
+do not display system message
+do not display the totals
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+Generate user list report.
+Notes:
+show program's version number and exit
+show this help message and exit
+filter on allocation id
+FIELD_INFO is
set number of fields to display
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+filter on name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+"FIELD_INFO" FIELD_INFO is
[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+also get inactive allocations
+[OPER1]
Operator Defaults:
+Date Parsing Precedence:
+SILENT, MUCH_LESS, LESS, MORE, VERBOSE, DEBUG, DEBUG1, DEBUG2
+only show list info that have charges regardless of project/user relationship
+remove commas from comma-separated thousands
+do not display the header
+do not display the row data
+do not display system message
+do not display the totals
+ + + + + + + + + + + + + + + +List Meta Command
+enter allocation id
+enter comment for new or edit commands, display comment for list commands
+enter event db id; event db id is an internal id created by the charging system
+enter
command line help
+enter jobid; jobid is created by the scheduler and is not unique
+enter number of fields to display
+enter name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+enter name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+enter suballocation id
+enter transaction id
+enter name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+enter the field width as follows:
enter end datetime filter
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+include inactive allocations
+include inactive allocations
+enter start datetime filter
+enter type of transaction
+for list allocations | projects | users, only show info with charges
+enter transaction-created datetime filter
+enter allocation award category
+enter allocation award-type name
+enter created datetime filter
+enter debug level
+get deleted objects
+get jobs that have not been charged
+get only deleted objects
+enter history datetime filter
+enter last updated datetime filter
+remove commas from comma-separated thousands
+do not display header
+do not display history information
+do not display rows
+do not display system message
+do not display totals
+enter queued datetime filter
+ + + + + + + + + + + + + + + +HPC Accounting System Command Line Interface
+"detail" meta command displays information in a long format with history updates, where appropriate.
+"list" meta command displays information in a table format, but no history updates are displayed.
+IMPORTANT NOTES + 1. All dates entered shall be interpreted as UTC + 2. non-admin users will only be able to see their content (jobs, charges, etc.) + 3. project admin users will be able to see all of the content for their projects + 4. staff admin users will be able to see all the content + 5. --help and -h are the help options.
+DETAIL COMMANDS
+ * allocations [-a|-e|-f|-j|-n|-p|-r|-t|-u|-w|-E|-H|-I|-O|-S|-T|...] [
LIST COMMANDS + * allocations [-a|-c|-e|-f|-j|-n|-p|-r|-t|-u|-w|-E|-H|-I|-O|-S|-T|...] (DEFAULT) + * jobs [-a|-e|-f|-j|-n|-p|-r|-t|-u|-w|-E|-H|-S|-T|...] projects [-a|-f|-n|-p|-r|-u|-w|-E|-H|-I|-S|...] + * transactions [-a|-c|-e|-f|-j|-n|-p|-r|-t|-u|-w|-E|-H|-S|-T|...] + * users [-a|-f|-n|-p|-r|-u|-w|-E|-H|-S|...]
+enter allocation id
+enter comment for new or edit commands, display comment for list commands
+enter event db id; event db id is an internal id created by the charging system
+enter
command line help
+enter jobid; jobid is created by the scheduler and is not unique
+enter number of fields to display
+enter name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+enter name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+enter suballocation id
+enter transaction id
+enter name or id, DO NOT MIX, enter 'all' to get all, wild cards '*' is allowed, but only on names
+enter the field width as follows:
enter end datetime filter
+abbreviate numbers and use unit suffixes: K (thousands), M (millions), G (billions), T (trillions) ...
+include inactive allocations
+get only inactive allocations
+enter start datetime filter
+enter type of transaction
+for list allocations | projects | users, only show info with charges
+enter transaction-created datetime filter
+enter allocation award category
+enter allocation award-type name
+enter created datetime filter
+enter debug level
+get deleted objects
+get jobs that have not been charged
+get only deleted objects
+enter history datetime filter
+enter the directory to store the pbs meta file
+all new pbs files will be ignored and marked as processed
+enter last updated datetime filter
+remove commas from comma-separated thousands
+do not display header
+do not display history information
+do not display rows
+do not display system message
+do not display totals
+enter queued datetime filter
+For -a, -e, -f, -w, -j, -p, -r, -t, -u, -T, --award-categories, --award_type_names, --cbank_refs options:
+These options can be entered multiple times for different values or entered once for multiple values.
+Examples:
+++sbank-list-allocation -u "pershey rojas allcock" or > sbank-list-allocation -u pershey -u rojas -u allcock
+
++sbank-list-allocation -f "id p avail" or > sbank-list-allocation -f id -f p -f avail For -u, -p and -r the use of wild card "*" is allowed, but only on names, not ids:
+
Examples:
+For -f option: +This option is the display field option.
+To get the available fields enter -f? or -f "?". Default fields columns will be displayed if no field option is specified.
+To replace the current fields to display, enter: +
> sbank-list-allocations ... -f "FIELD[:WIDTH]...FIELD[:WIDTH]" or > sbank-list-allocations ... -f FIELD[:WIDTH] ... -f FIELD[:WIDTH]
+If you wish to add fields to the default fields, enter one + symbol anywhere in the quoted string: +
+The fields will be displayed in table format and in the order entered in the command line. You can specify the field width, where WIDTH can be positive or negative value. Left alignment use -, right alignment use + or nothing.
+For -w option:
+FIELD:WIDTH, if the field is displayed it will change the width for the specified field.
+NOTE: This will not add the field as in -f option, only change the width. To get available fields you can also use -w? or -w "?" as in -f option.
+For -S, -E, --created, --queued, --last-updated, --history-date-range options:
+These are the date filter options. All dates are treated as UTC.
+You can use any reasonable date string that resembles a date Ambiguous dates will be parsed with the following parsing precedence: **YEAR then MONTH then DAY **
+For example, 10-11-12 or 101112 will be the following date: Oct. 11, 2012 Not: Nov. 12, 2010 or Nov. 10, 2012
+Or you can specify a single date as follows: +
"[OPER]UTC_DATE" You can specify a date range as follows:
+"[OPER1]UTC_DATE1...[OPER2]UTC_DATE2" Where OPER can be one of the following operators: "==", ">=", "<=", ">", "<" or "eq", "ge", "le", "gt", "lt"
+Note: The following defaults for OPER, OPER1, OPER2 for the following options: +
Options OPER OPER1 OPER2 ------------------------- ---- ----- ----- -E, < >= < -S, >= >= < --at >= >= < --created >= >= < --eligible >= >= < --last-updated >= >= < --queued >= >= <
+You can also use the following key letters "n", "t", "d", "w", "y" as follows: +
KEY SYNTAX DEFINITIONS ---------- ----------- n[ow] now, where "now" is current-date current-time UTC t[oday] today, where "today" is current-date 00:00:00 UTC [+/-]d specified "number" of +/- days from "today" in UTC [+/-]w specified "number" of +/- weeks from "today" in UTC [+/-]y specified "number" of +/- years from "today" in UTC
+For -T option:
+Transaction type option. The following are the valid transaction types and their explanation: CHARGE filter on job charges PULLBACK filter on allocation pullbacks DEPOSIT filter on allocation deposits REFUND filter on job refunds VOID filter on void transactions
+sbank sbank sbank sbank-detail sbank detail sbank d sbank-detail-allocations sbank detail allocations sbank d a sbank-detail-jobs sbank detail jobs sbank d j sbank-detail-projects sbank detail project sbank d p sbank-detail-transactions sbank detail transactions sbank d t sbank-detail-users sbank detail users sbank d u sbank-list sbank list sbank l sbank-list-allocations sbank list allocations sbank l a sbank-list-jobs sbank list jobs sbank l j sbank-list-projects sbank list projects sbank l p sbank-list-transactions sbank list transactions sbank l t sbank-list-users sbank list users sbank l u
+Command line default options: Define the following environment variables as you would in the command line. Once the environment variable is defined, it will be used as the default options and arguments for the specific command. Command line options will take precedence.
+sbank_DETAIL_ALLOCATIONS_ARGS
+Default arguments and options for sbank-detail-allocations.
+sbank_DETAIL_CATEGORIES_ARGS
+Default arguments and options for sbank-detail-categories.
+sbank_DETAIL_NAMES_ARGS
+Default arguments and options for sbank-detail-names.
+sbank_DETAIL_MESSAGES_ARGS
+Default arguments and options for sbank-detail-messages.
+sbank_DETAIL_JOBS_ARGS
+Default arguments and options for sbank-detail-jobs.
+sbank_DETAIL_PROJECTS_ARGS
+Default arguments and options for sbank-detail-projects.
+sbank_DETAIL_TRANSACTIONS_ARGS
+Default arguments and options for sbank-detail-transactions.
+sbank_DETAIL_USERS_ARGS
+Default arguments and options for sbank-detail-users.
+sbank_LIST_ALLOCATIONS_ARGS
+Default arguments and options for sbank-list-allocations.
+sbank_LIST_JOBS_ARGS
+Default arguments and options for sbank-list-jobs.
+sbank_LIST_PROJECTS_ARGS
+Default arguments and options for sbank-list-projects.
+sbank_LIST_TRANSACTIONS_ARGS
+Default arguments and options for sbank-list-transactions.
+sbank_LIST_USERS_ARGS
+Default arguments and options for sbank-list-users.
+Example 1: -f, --field +
> sbank-list-transactions ... -f field1:-20 -f field2:20 -f field3 or > sbank-list-transactions ... -f "field1:-20 field2:20 field3"
+Example 2: -S, -E, --created, --queued, --last-updated, --history-start, --history-end
+Single date-string examples:
+++sbank-list-allocations -S ">=Oct 11, 2014" start dates that are >= "2014-10-11 00:00:00"
+
++sbank-list-allocations -S "<=2014-11-10" start dates that are <= "2014-11-10 00:00:00"
+
++sbank-list-allocations -E "<20141110" end dates that are < "2014-11-10 00:00:00"
+
++sbank-list-allocations -E "22:30:10" end dates that are < "
+22:30:10"
++sbank-list-allocations -S ">today" start dates that are > "
+00:00:00"
++sbank-list-allocations -E t end dates that are < "
+00:00:00"
++sbank-list-allocations -S gtnow start dates that are > "
+"
++sbank-list-allocations -E len end dates that are <= "
+"
++sbank-list-allocations -S "1d" start dates that are >= "today +1 day"
+
++sbank-list-allocations -E "-2w" end dates that are < "today -2 weeks"
+
++sbank-list-allocations -S ">=1y" start dates that are >= "today +1 year"
+
++sbank-list-allocations -S ">2012" start dates that are > "2012-
+- 00:00:00"
Range date-string examples:
+++sbank-list-allocations -S "2013-01-01...2014-01-01" "2013-01-01" <= DATES < "2014-01-01"
+
++sbank-list-allocations -S "-1y...t" "today -1 year" <= DATES < "today"
+
++sbank-list-allocations -E "2013...t"" "2013-
+- " <= DATES < "today"
++sbank-list-allocations -E ">2013...<=t"" "2013-
+- " < DATES <= "today"
Example 3: Command invocation examples
+++sbank-list-projects list projects full command invocation
+
++sbank list projects list projects meta command invocation
+
++sbank s p list projects partial meta command invocation
+
++sbank p list projects where "list" is the default
+
++sbank list allocations is the default
+
++sbank a list allocations "list" is the default
+
++sbank s a list allocations partial meta command invocation
+
Example 4: -h, --help
+++sbank -h will give you help summary on all of sbank
+
++sbank list --help will give you help on all the "list" commands
+
++sbank list allocations -h will give you help on the "list allocations" command
+
++sbank-list-allocations -h will give you help on the "list allocations" command
+
++sbank l a --help will give you help on the "list allocations" command
+
Researchers gain access to ALCF systems for computational science and engineering projects—typically with awards of millions of core-hours—through competitive, peer-reviewed allocation programs supported by the DOE and Argonne. Our peer-reviewed award programs consist of the INCITE, ALCC, and ADSP programs. More information about the programs, including dates for our CFPs, can be found on their web pages.
+Alternatively, ALCF offers a Director's Discretionary allocation award program to leadership computing preparation, INCITE and ALCC scaling, and application performance to maximize scientific application efficiency and productivity on leadership computing platforms. See the Director's Discretionary (DD) Program page for more information.
+Projects with INCITE, ALCC, and ADSP awards will be contacted directly by the ALCF staff with information on creating accounts.
+Director's Discretionary awards will receive information in the award confirmation email.
+While requesting an allocation, users can choose from:
+Computes: +* Polaris
+File System: +* Eagle (Community Sharing)
+If you are a PI of a Director's Discretionary project that has an active allocation, you can request additional time or an extension using the allocation request form.
+
+
sbank is the accounting system used within the ALCF. It tracks project allocations, usage charges, and refunds. sbank allows queries about the balance and expiration of project allocations, and has replaced the outdated cbank accounting system.
+The sbank accounting system helps users manage their allocations and usage per job. It gives the PIs the ability to monitor their allocation usage by user, job, and machine. It also allows the user to monitor their usage per allocation and provides insight on how many hours are left on the project.
+sbank Example Commands provides a set of example commands on how to use the most common commands.
+Use these sbank man pages to get information on how to use the commands.
+ + + + + + + + + + + + + + + + +TEST
+ + + + + + + + + + + + + + + +The Argonne Leadership Computing Facility (ALCF) is required to report the progress and scientific accomplishments of all peer-reviwed projects.
+PIs of INCITE, ALCC, and ADSP projects are required to complete quarterly reports and a final end-of-project (EOY/EOP) report.
+If a quarterly report is more than 30 days late: +- The ability to submit jobs for the PI and users of the late project will be disabled.
+If a quarterly report is more than 90 days late: +- The PI and users of the late project will have their accounts disabled.
+These penalties will be removed within three business days after the late quarterly or EOY report is submitted.
+A similar penalty will also be applied to new ALCC projects with the same PI or co-PIs that have failed to submit the EOP report for a previous ALCC project. If the EOP report is more than 15 days late:
+A PI or user may appeal a project or account suspension to the ALCF Director by a request to support at alcf.anl.gov.
+Templates for the quarterly and the EOY reports can be found at the links on the bottom of this page.
+Please modify the filename to replace PINAME with the last name of the PI of the INCITE/ALCC project, ALLOCATION to INCITE/ALCC, and YEAR to the corresponding calendar year. For quarterly reports, please replace the X in the filename with the quarter number.
+For example, for a project with PI 'Joe Smith' that is submitting the quarterly report for the first quarter in 2023-2024 cycle for ALCC, the filename will be Smith_ALCC_Q1.docx.
+For an EOY report, replace YEARS with the years associated with your allocation. For example, an ALCC 2023-2024 project with PI 'Joe Smith' would have a filename of Smith_ALCC_2023-2024_EOY.docx.
+The following guide is for PIs and Proxies to get insight into managing projects and teams for ALCF awards. Please submit for questions or trouble tickets to support@alcf.anl.gov.
+To get started using our resources, please visit: +Connect & Login
+We also encourage you to take full advantage of ALCF's training programs and user services. Some useful introductory materials and videos are listed below:
+ +Before your project begins, you will receive an email with the following project information:
+If you do not have an ALCF account: You will need to request one at https://my.alcf.anl.gov/accounts/#/accountRequest. When prompted for project name, please select the project short name you were given in your award email from support@alcf.anl.gov.
+If you have an active ALCF account: Submit a request to join the newly awarded project at https://my.alcf.anl.gov/.
+The U.S. Department of Energy has guidelines and requirements for foreign nationals who access its facilities and sites. This guidance is issued in DOE Order 142.3, which is part of Argonne's contract; therefore, all foreign nationals (non-U.S. Citizens) must obtain authorization prior to using ALCF resources.
+If you are a foreign national and do not have current authorization credentials, you are required to submit a ANL-593 (Foreign National Access Request) form. It is critical that identity documentation requests sent by ALCF staff are completed as early as possible to facilitate timely processing for your account approval.
+Note: This does not apply to Director's Discretionary awards.
+If you are not an employee of Argonne National Laboratory, a user agreement must be signed by your home institution to perform research at Argonne’s user facilities. This policy applies to every member of the project team who will be conducting research on ALCF resources.
+A list of home institutions that have master agreements in place is located on this webpage: https://www.aps.anl.gov/Users-Information/Legal-Financial/Argonne-User-Facility-Agreements
+Note: This does not apply to Director's Discretionary awards.
+Every project team member who requests an ALCF account must sign and return an acknowledgment form, stating that they agree to the terms in the user agreement.
+The form is located at: https://www.alcf.anl.gov/files/Acknowledgement_Form.pdf. Please print, sign, scan and email it to accounts@alcf.anl.gov.
+As a PI, you can add members to your project. You can assign proxies who are project members authorized to add or renew project members on your behalf.
+A project PI or proxy has the authority to:
+During your project setup, the ALCF Support Team will request the following information to establish your project members:
+All project members have the ability to run jobs against your allocation. There is no limit to the number of project members you may authorize. +Project members are automatically added to the project UNIX group giving them the ability to write to the project directory and to access project data. When a project member is added or removed from a project, this automatically be reflected in the project UNIX group membership.
+The PI or a proxy must approve each team member to access ALCF resources and run jobs on their project. PI/proxies can respond to emails from ALCF for account access approval with a "yes" or "no".
+PI/proxies with active ALCF accounts can also approve new account requests, project membership requests, account reactivation requests, add existing active ALCF users to the project by logging into the ALCF Account and Project Management application.
+Note: If PI/proxies need to request an ALCF account, see the section below for instructions on "how to apply" for an account.
+All project members will need an ALCF user account to access project data and to run jobs on ALCF systems.
+Members that do not have an ALCF account should request one at: https://my.alcf.anl.gov/accounts/#/accountRequest. When prompted for project name, they should select your project short name.
+Members with ALCF accounts that are no longer active should submit a reactivation request here: https://my.alcf.anl.gov/accounts/#/accountReactivate. When prompted for project name, they should select your project short name.
+Members with active ALCF accounts but have not been added to your project should submit a request to join your project by going to this page: https://my.alcf.anl.gov/. They should search for your project and click the "Request Membership" button for that project.
+We encourage you to use Globus to move your project data to your ALCF project directory before your allocation begins. For details, see Using Globus.
+Note: PIs that are awarded a Director's Discretionary will not receive weekly status project reports.
+Shortly after your allocation begins, we will begin sending you a weekly project status report via support@alcf.anl.gov to keep you informed or your award progress.
+Look for an email from us with the subject line: ALCF [ALLOCATION PROGRAM] Project Status Report for [PROJECT SHORT NAME]
+Note: PIs that are awarded a Director's Discretionary allocations are not required to submit project reports.
+If you receieved INCITE, ALCC, or ADSP allocation award, quarterly reporting is required to keep DOE informed of progress related to your allocation.
+The ALCF will send you a report template at the end of each quarter. Please complete the report promptly and submit it via email to support@alcf.anl.gov. For more information see the Quarterly Report webpage.
+Please be aware that we will periodically monitor, and could potentially adjust, your project allocation if a large portion of it goes unused. You may view: Pullback Policy
+Please see this page for overburn/overuse eligibility for INCITE projects that have exhausted their allocation in the first 11 months of its allocation year: Allocation Overburn
+Please follow the guidelines provided on the ALCF Acknowledgement Policy page to properly acknowledge the use of ALCF resources in all of your publications, both online and print.
+Facility policies have been established to provide consistent and reliable services. Please read about our [ALCF Facility Policies] (../policies/facility-policies.md).
+We have an allocation management tool called sbank, and below are a few helpful sbank commands.
+You can use the following command to check your project balance on Polaris:
+- sbank-list-allocations -p
For more command examples and details, see sbank.
+We can also help resolve any issues or needs that may be delaying the start of your scientific campaign. +- Are you in need of high-throughput software? +- Are you having difficulty compiling your application? +- Does your code have limited restart capabilities?
+If your project allocation usage is being held back for reasons due to one of our systems, please contact us for assistance by emailing support@alcf.anl.gov.
+ + + + + + + + + + + + + + + +The PI or Proxy must approve each member of the team to gain access and to run project jobs on the ALCF's resources. If you have an active ALCF account, you can manage your project team by logging into the ALCF account and project management website and navigating to https://my.alcf.anl.gov/
+Project members will need to have an active ALCF user account to access project data and to run jobs on ALCF systems. See [Accounts and Access for your Project Members]{https://docs.alcf.anl.gov/account-project-management/project-management/starting-alcf-award/#accounts-and-access-for-your-project-members) for information on how team members can get an account, reactivate an account, or request to join your project.
+Some project information cannot be modified, but as the PI, you can modify the following: project title, institutions, and associated funding.
+Your project can be associated with multiple institutions, but you must specify a primary institution.
+Membership Status
+Proxies are individuals authorized to add or renew user accounts for the project PI. You have the ability to upgrade a user from a member to a Proxy, by clicking on the "Proxy" radio button that corresponds with the desired member.
+mkdir ~/R_2.3.0
+cd ~/R_2.3.0
+# Note: "deactivate" does not actually work in scripts.
+deactivate
+rm -r venv_cerebras_pt
+/software/cerebras/python3.8/bin/python3.8 -m venv venv_cerebras_pt
+source venv_cerebras_pt/bin/activate
+pip install --upgrade pip
+pip install cerebras_pytorch==2.3.0
+To activate a virtual environments
+ +To deactivate a virtual environment,
+ + + + + + + + + + + + + + + + +Make a working directory and a local copy of the Cerebras modelzoo and anl_shared repository, if not previously done, as follows.
+mkdir ~/R_2.3.0
+cd ~/R_2.3.0
+git clone https://github.com/Cerebras/modelzoo.git
+cd modelzoo
+git tag
+git checkout Release_2.3.0
+The modelzoo/modelzoo/transformers/pytorch/bert directory is a PyTorch implementation of BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
+This BERT-large msl128 example uses a single sample dataset for both training and evaluation. See the README.md in the source directory for details on how to build a dataset from text input.
+First, source a Cerebras PyTorch virtual environment and make sure that the requirements are installed:
source ~/R_2.3.0/venv_cerebras_pt/bin/activate
+pip install -r ~/R_2.3.0/modelzoo/requirements.txt
+Then
+cd ~/R_2.3.0/modelzoo/src/cerebras/modelzoo/models/nlp/bert
+cp /software/cerebras/dataset/bert_large/bert_large_MSL128_sampleds.yaml configs/bert_large_MSL128_sampleds.yaml
+export MODEL_DIR=model_dir_bert_large_pytorch
+if [ -d "$MODEL_DIR" ]; then rm -Rf $MODEL_DIR; fi
+python run.py CSX --job_labels name=bert_pt --params configs/bert_large_MSL128_sampleds.yaml --num_workers_per_csx=1 --mode train --model_dir $MODEL_DIR --mount_dirs /home/ /software/ --python_paths /home/$(whoami)/R_2.3.0/modelzoo/src --compile_dir $(whoami) |& tee mytest.log
+/software/cerebras/dataset/bert_large/bert_large_MSL128_sampleds.yaml is the same as the one at /home/$(whoami)/R_2.3.0/modelzoo/modelzoo/transformers/vocab/google_research_uncased_L-12_H-768_A-12.txt.
+The last parts of the output should resemble the following, with messages about cuda that should be ignored and are not shown.
+2023-11-29 20:07:49,284 INFO: Beginning appliance run
+2023-11-29 20:08:14,365 INFO: | Train Device=CSX, Step=100, Loss=9.50000, Rate=4088.28 samples/sec, GlobalRate=4088.26 samples/sec
+2023-11-29 20:08:39,820 INFO: | Train Device=CSX, Step=200, Loss=8.37500, Rate=4048.91 samples/sec, GlobalRate=4055.21 samples/sec
+2023-11-29 20:09:05,356 INFO: | Train Device=CSX, Step=300, Loss=7.96875, Rate=4025.61 samples/sec, GlobalRate=4040.05 samples/sec
+2023-11-29 20:09:30,626 INFO: | Train Device=CSX, Step=400, Loss=7.56250, Rate=4041.61 samples/sec, GlobalRate=4043.10 samples/sec
+2023-11-29 20:09:56,022 INFO: | Train Device=CSX, Step=500, Loss=7.50000, Rate=4035.92 samples/sec, GlobalRate=4040.90 samples/sec
+2023-11-29 20:10:21,410 INFO: | Train Device=CSX, Step=600, Loss=7.37500, Rate=4034.41 samples/sec, GlobalRate=4039.65 samples/sec
+2023-11-29 20:10:46,690 INFO: | Train Device=CSX, Step=700, Loss=7.37500, Rate=4044.10 samples/sec, GlobalRate=4041.20 samples/sec
+2023-11-29 20:11:12,004 INFO: | Train Device=CSX, Step=800, Loss=7.25000, Rate=4044.75 samples/sec, GlobalRate=4041.70 samples/sec
+2023-11-29 20:11:37,196 INFO: | Train Device=CSX, Step=900, Loss=7.21875, Rate=4056.77 samples/sec, GlobalRate=4044.25 samples/sec
+2023-11-29 20:12:02,285 INFO: | Train Device=CSX, Step=1000, Loss=7.12500, Rate=4071.60 samples/sec, GlobalRate=4047.95 samples/sec
+2023-11-29 20:12:02,286 INFO: Saving checkpoint at step 1000
+2023-11-29 20:12:37,079 INFO: Saved checkpoint model_dir_bert_large_pytorch/checkpoint_1000.mdl
+2023-11-29 20:13:25,683 INFO: Heartbeat thread stopped for wsjob-gfi2baioyfduozkmgsc6a7.
+2023-11-29 20:13:25,691 INFO: Training completed successfully!
+2023-11-29 20:13:25,691 INFO: Processed 1024000 sample(s) in 336.373620536 seconds.
+GPT-J [github] is an auto-regressive language model created by EleutherAI. +This PyTorch GPT-J 6B parameter pretraining sample uses 2 CS2s.
+First, source a Cerebras PyTorch virtual environment and make sure that the requirements are installed:
+source ~/R_2.3.0/venv_cerebras_pt/bin/activate
+pip install -r ~/R_2.3.0/modelzoo/requirements.txt
+Then
+cd ~/R_2.3.0/modelzoo/src/cerebras/modelzoo/models/nlp/gptj
+cp /software/cerebras/dataset/gptj/params_gptj_6B_sampleds.yaml configs/params_gptj_6B_sampleds.yaml
+export MODEL_DIR=model_dir_gptj
+if [ -d "$MODEL_DIR" ]; then rm -Rf $MODEL_DIR; fi
+python run.py CSX --job_labels name=gptj_pt --params configs/params_gptj_6B_sampleds.yaml --num_csx=2 --mode train --model_dir $MODEL_DIR --mount_dirs /home/ /software --python_paths /home/$(whoami)/R_2.3.0/modelzoo/src --compile_dir $(whoami) |& tee mytest.log
+The last parts of the output should resemble the following:
+2023-11-29 20:59:19,223 INFO: Beginning appliance run
+2023-11-29 21:03:53,875 INFO: | Train Device=CSX, Step=100, Loss=8.43750, Rate=43.70 samples/sec, GlobalRate=43.70 samples/sec
+2023-11-29 21:08:28,779 INFO: | Train Device=CSX, Step=200, Loss=8.12500, Rate=43.67 samples/sec, GlobalRate=43.67 samples/sec
+2023-11-29 21:08:28,781 INFO: Saving checkpoint at step 200
+2023-11-29 21:13:56,695 INFO: Saved checkpoint model_dir_gptj/checkpoint_200.mdl
+2023-11-29 21:14:30,135 INFO: Heartbeat thread stopped for wsjob-kd4olqkhu6ya8qqzt88utd.
+2023-11-29 21:14:30,142 INFO: Training completed successfully!
+2023-11-29 21:14:30,142 INFO: Processed 24000 sample(s) in 910.883781998 seconds.
+The Cerebras llama7B model implementation can be found at modelzoo/modelzoo/transformers/pytorch/llama and its overview at https://github.com/Cerebras/modelzoo/blob/main/src/cerebras/modelzoo/models/nlp/llama/README.md#configs-included-for-this-model. This set up will use a subset of pile data (preprocessed at path /software/datasets/llama_data_32K/) to train with a 32K vocab size.
+First, source a Cerebras PyTorch virtual environment and make sure that the requirements are installed: +
source ~/R_2.3.0/venv_cerebras_pt/bin/activate
+pip install -r ~/R_2.3.0/modelzoo/requirements.txt
+cd ~/R_2.3.0/modelzoo/src/cerebras/modelzoo/models/nlp/llama
+cp /software/cerebras/dataset/params_llama_7b.yaml configs/params_llama_7b.yaml
+export MODEL_DIR=model_dir_llamma
+if [ -d "$MODEL_DIR" ]; then rm -Rf $MODEL_DIR; fi
+python run.py CSX --job_labels name=llama_7b --params configs/params_llama_7b.yaml --num_csx=1 --mode train --model_dir $MODEL_DIR --mount_dirs /projects /home/ /software --python_paths /home/$(whoami)/R_2.3.0/modelzoo/src --compile_dir $(whoami) |& tee mytest.log
+Please find a sample output +
2024-03-21 14:40:57,949 INFO: Effective batch size is 99.
+2024-03-21 14:40:57,970 INFO: Checkpoint autoloading is enabled. Looking for latest checkpoint in "/srv/projects/datascience/vsastry/model_dir_llama/" directory with the following naming convention: `checkpoint_(step)(_timestamp)?.mdl`.
+2024-03-21 14:40:57,971 INFO: No checkpoints were found in "/srv/projects/datascience/vsastry/model_dir_llama/".
+2024-03-21 14:40:57,971 INFO: No checkpoint was provided. Using randomly initialized model parameters.
+2024-03-21 14:40:59,419 INFO: Saving checkpoint at step 0
+2024-03-21 14:48:46,988 INFO: Saved checkpoint /srv/projects/datascience/vsastry/model_dir_llama/checkpoint_0.mdl
+2024-03-21 14:49:05,547 INFO: Compiling the model. This may take a few minutes.
+2024-03-21 14:49:05,550 INFO: Defaulted to use the job-operator namespace as the usernode config /opt/cerebras/config_v2 only has access to that namespace.
+2024-03-21 14:49:06,819 INFO: Initiating a new image build job against the cluster server.
+2024-03-21 14:49:06,898 INFO: Custom worker image build is disabled from server.
+2024-03-21 14:49:06,911 INFO: Defaulted to use the job-operator namespace as the usernode config /opt/cerebras/config_v2 only has access to that namespace.
+2024-03-21 14:49:07,143 INFO: Initiating a new compile wsjob against the cluster server.
+2024-03-21 14:49:07,226 INFO: compile job id: wsjob-pg4gslxvgsalvh6ppdvydb, remote log path: /n1/wsjob/workdir/job-operator/wsjob-pg4gslxvgsalvh6ppdvydb
+2024-03-21 14:49:17,259 INFO: Poll ingress status: Waiting for job running, current job status: Queueing, msg: job is queueing. Job queue status: current job is top of queue but likely blocked by running jobs, 1 compile job(s) running using 67Gi memory. For more information, please run 'csctl get jobs'.
+2024-03-21 15:02:07,673 INFO: Poll ingress status: Waiting for job running, current job status: Queueing, msg: job is queueing. Job queue status: current job is top of queue but likely blocked by running jobs, 1 execute job(s) running using 1 system(s), 1 compile job(s) running using 67Gi memory. For more information, please run 'csctl get jobs'.
+2024-03-21 15:02:17,683 INFO: Poll ingress status: Waiting for job service readiness.
+2024-03-21 15:02:47,717 INFO: Ingress is ready: Job ingress ready, poll ingress success.
+2024-03-21 15:02:58,509 INFO: Pre-optimization transforms...
+2024-03-21 15:03:14,815 INFO: Optimizing layouts and memory usage...
+2024-03-21 15:03:14,839 INFO: Gradient accumulation enabled
+2024-03-21 15:03:14,840 WARNING: Gradient accumulation will search for an optimal micro batch size based on internal performance models, which can lead to an increased compile time. Specify `micro_batch_size` option in the 'train_input/eval_input' section of your .yaml parameter file to set the gradient accumulation microbatch size, if an optimal microbatch size is known.
+
+2024-03-21 15:03:14,842 INFO: Gradient accumulation trying sub-batch size 3...
+2024-03-21 15:03:21,632 INFO: Exploring floorplans
+2024-03-21 15:03:30,198 INFO: Exploring data layouts
+2024-03-21 15:03:50,589 INFO: Optimizing memory usage
+2024-03-21 15:05:23,008 INFO: Gradient accumulation trying sub-batch size 33...
+2024-03-21 15:05:30,532 INFO: Exploring floorplans
+2024-03-21 15:05:37,304 INFO: Exploring data layouts
+2024-03-21 15:06:11,327 INFO: Optimizing memory usage
+2024-03-21 15:11:37,204 INFO: Gradient accumulation trying sub-batch size 9...
+2024-03-21 15:11:44,383 INFO: Exploring floorplans
+2024-03-21 15:11:50,639 INFO: Exploring data layouts
+2024-03-21 15:12:16,120 INFO: Optimizing memory usage
+2024-03-21 15:15:59,788 INFO: Gradient accumulation trying sub-batch size 11...
+2024-03-21 15:16:06,314 INFO: Exploring floorplans
+2024-03-21 15:16:12,563 INFO: Exploring data layouts
+2024-03-21 15:16:40,965 INFO: Optimizing memory usage
+2024-03-21 15:21:03,938 INFO: Exploring floorplans
+2024-03-21 15:21:10,918 INFO: Exploring data layouts
+2024-03-21 15:22:03,953 INFO: Optimizing memory usage
+2024-03-21 15:30:35,456 INFO: No benefit from gradient accumulation expected. Compile will proceed at original per-box batch size 99 with 9 lanes
+
+2024-03-21 15:30:35,540 INFO: Post-layout optimizations...
+2024-03-21 15:32:11,639 INFO: Allocating buffers...
+2024-03-21 15:32:18,023 INFO: Code generation...
+2024-03-21 15:32:53,573 INFO: Compiling image...
+2024-03-21 15:32:53,578 INFO: Compiling kernels
+2024-03-21 15:34:39,222 INFO: Compiling final image
+2024-03-21 15:36:54,995 INFO: Compile artifacts successfully written to remote compile directory. Compile hash is: cs_2599085507768189065
+2024-03-21 15:36:55,146 INFO: Heartbeat thread stopped for wsjob-pg4gslxvgsalvh6ppdvydb.
+2024-03-21 15:36:55,160 INFO: Compile was successful!
+2024-03-21 15:36:55,171 INFO: Programming Cerebras Wafer Scale Cluster for execution. This may take a few minutes.
+2024-03-21 15:36:56,403 INFO: Defaulted to use the job-operator namespace as the usernode config /opt/cerebras/config_v2 only has access to that namespace.
+2024-03-21 15:36:56,659 INFO: Initiating a new execute wsjob against the cluster server.
+2024-03-21 15:36:56,758 INFO: execute job id: wsjob-bdcvvsrwely3kbfwduefqx, remote log path: /n1/wsjob/workdir/job-operator/wsjob-bdcvvsrwely3kbfwduefqx
+2024-03-21 15:37:06,789 INFO: Poll ingress status: Waiting for job running, current job status: Scheduled, msg: job is scheduled.
+2024-03-21 15:37:16,793 INFO: Poll ingress status: Waiting for job service readiness.
+2024-03-21 15:37:36,838 INFO: Poll ingress status: Waiting for job ingress readiness.
+2024-03-21 15:37:46,861 INFO: Ingress is ready: Job ingress ready, poll ingress success.
+2024-03-21 15:37:47,052 INFO: Preparing to execute using 1 CSX
+2024-03-21 15:38:33,999 INFO: About to send initial weights
+2024-03-21 15:40:01,150 INFO: Finished sending initial weights
+2024-03-21 15:40:01,154 INFO: Finalizing appliance staging for the run
+2024-03-21 15:40:01,203 INFO: Waiting for device programming to complete
+2024-03-21 15:41:26,576 INFO: Device programming is complete
+2024-03-21 15:41:27,888 INFO: Using network type: ROCE
+2024-03-21 15:41:27,890 INFO: Waiting for input workers to prime the data pipeline and begin streaming ...
+2024-03-21 15:41:27,942 INFO: Input workers have begun streaming input data
+2024-03-21 15:41:45,009 INFO: Appliance staging is complete
+2024-03-21 15:41:45,021 INFO: Beginning appliance run
+2024-03-21 15:49:45,474 INFO: | Train Device=CSX, Step=100, Loss=9.84375, Rate=20.61 samples/sec, GlobalRate=20.61 samples/sec
+2024-03-21 15:57:49,616 INFO: | Train Device=CSX, Step=200, Loss=8.35938, Rate=20.51 samples/sec, GlobalRate=20.53 samples/sec
+2024-03-21 16:05:53,769 INFO: | Train Device=CSX, Step=300, Loss=8.26562, Rate=20.47 samples/sec, GlobalRate=20.50 samples/sec
+2024-03-21 16:13:58,078 INFO: | Train Device=CSX, Step=400, Loss=7.02344, Rate=20.45 samples/sec, GlobalRate=20.49 samples/sec
+2024-03-21 16:22:02,644 INFO: | Train Device=CSX, Step=500, Loss=7.07812, Rate=20.44 samples/sec, GlobalRate=20.48 samples/sec
+2024-03-21 16:30:06,513 INFO: | Train Device=CSX, Step=600, Loss=7.34375, Rate=20.45 samples/sec, GlobalRate=20.47 samples/sec
+2024-03-21 16:38:10,737 INFO: | Train Device=CSX, Step=700, Loss=7.19531, Rate=20.45 samples/sec, GlobalRate=20.47 samples/sec
+2024-03-21 16:46:15,052 INFO: | Train Device=CSX, Step=800, Loss=6.52344, Rate=20.44 samples/sec, GlobalRate=20.47 samples/sec
+2024-03-21 16:54:19,448 INFO: | Train Device=CSX, Step=900, Loss=6.46875, Rate=20.44 samples/sec, GlobalRate=20.46 samples/sec
+2024-03-21 17:02:24,111 INFO: | Train Device=CSX, Step=1000, Loss=5.98438, Rate=20.43 samples/sec, GlobalRate=20.46 samples/sec
+2024-03-21 17:10:28,632 INFO: | Train Device=CSX, Step=1100, Loss=6.17188, Rate=20.43 samples/sec, GlobalRate=20.46 samples/sec
+2024-03-21 17:18:32,943 INFO: | Train Device=CSX, Step=1200, Loss=6.04688, Rate=20.44 samples/sec, GlobalRate=20.46 samples/sec
+2024-03-21 17:26:37,241 INFO: | Train Device=CSX, Step=1300, Loss=5.54688, Rate=20.44 samples/sec, GlobalRate=20.45 samples/sec
+2024-03-21 17:34:41,491 INFO: | Train Device=CSX, Step=1400, Loss=5.92188, Rate=20.44 samples/sec, GlobalRate=20.45 samples/sec
+2024-03-21 17:42:45,646 INFO: | Train Device=CSX, Step=1500, Loss=5.68750, Rate=20.45 samples/sec, GlobalRate=20.45 samples/sec
+2024-03-21 17:50:50,110 INFO: | Train Device=CSX, Step=1600, Loss=5.85938, Rate=20.44 samples/sec, GlobalRate=20.45 samples/sec
+Evolutionary Scale Modeling (ESM-2) is a transformer protein language models from the Meta Fundamental AI Research Protein Team (FAIR).
+The Cerebras ESM-2 model implementation can be found at modelzoo/src/cerebras/modelzoo/models/nlp/esm2. Configs available are listed at https://github.com/Cerebras/modelzoo/tree/main/src/cerebras/modelzoo/models/nlp/esm2#configs-included-for-this-model. This example will use the Uniref 50 dataset, preprocessed at path /software/datasets/ESM-2/, to train a small 35M parameter model.
First, source a Cerebras PyTorch virtual environment and make sure that the requirements are installed: +
source ~/R_2.3.0/venv_cerebras_pt/bin/activate
+pip install -r ~/R_2.3.0/modelzoo/requirements.txt
+cd ~/R_2.3.0/modelzoo/src/cerebras/modelzoo/models/nlp/esm2
+export MODEL_DIR=model_dir_esm2
+if [ -d "$MODEL_DIR" ]; then rm -Rf $MODEL_DIR; fi
+cp /software/cerebras/dataset/ESM-2/params_esm2_t12_35M_UR50D_modified.yaml configs/params_esm2_t12_35M_UR50D_modified.yaml
+python run.py CSX --job_labels name=esm2_t12_35m --params configs/params_esm2_t12_35M_UR50D_modified.yaml --num_csx=1 --mode train --model_dir $MODEL_DIR --mount_dirs /home/$(whoami)/ /software --python_paths /home/$(whoami)/R_2.3.0/modelzoo/src --compile_dir /$(whoami) |& tee mytest.log
+Sample output +
2024-08-02 20:53:25,927 INFO: Checkpoint autoloading is enabled. Looking for latest checkpoint in "model_dir_esm2" directory with the following naming convention: `checkpoint_(step)(_timestamp)?.mdl`.
+2024-08-02 20:53:25,928 INFO: No checkpoints were found in "model_dir_esm2".
+2024-08-02 20:53:25,928 INFO: No checkpoint was provided. Using randomly initialized model parameters.
+2024-08-02 20:53:25,930 INFO: Starting training loop 1, from global step 0 to 400
+2024-08-02 20:53:26,257 WARNING: Passing an absolute path as the compile directory may lead to undesirably long paths as the directory is used on the server side, not on the client side. Please consider passing in a relative directory instead.
+2024-08-02 20:53:26,488 INFO: Saving checkpoint at step 0
+2024-08-02 20:53:35,010 INFO: Saved checkpoint model_dir_esm2/checkpoint_0.mdl
+2024-08-02 20:53:45,962 INFO: Compiling the model. This may take a few minutes.
+2024-08-02 20:53:47,482 INFO: Initiating a new image build job against the cluster server.
+2024-08-02 20:53:47,488 INFO: Custom worker image build is disabled from server.
+2024-08-02 20:53:47,492 WARNING: Passing an absolute path as the compile directory may lead to undesirably long paths as the directory is used on the server side, not on the client side. Please consider passing in a relative directory instead.
+2024-08-02 20:53:47,658 INFO: Initiating a new compile wsjob against the cluster server.
+2024-08-02 20:53:47,672 INFO: Compile job id: wsjob-4jm7wrbl6lfnjf9hc2qukx, remote log path: /n1/wsjob/workdir/job-operator/wsjob-4jm7wrbl6lfnjf9hc2qukx
+2024-08-02 20:53:57,709 INFO: Poll ingress status: Waiting for job running, current job status: Initializing, msg: job initializing with config generation.
+2024-08-02 20:54:07,710 INFO: Poll ingress status: Waiting for all Coordinator pods to be running, current running: 0/1.
+2024-08-02 20:54:17,724 INFO: Ingress is ready: Job ingress ready, poll ingress success.
+2024-08-02 20:54:20,821 INFO: Pre-optimization transforms...
+2024-08-02 20:54:24,168 INFO: Optimizing layouts and memory usage...
+2024-08-02 20:54:24,181 INFO: Gradient accumulation enabled
+2024-08-02 20:54:24,188 INFO: Gradient accumulation trying micro batch size 64...
+2024-08-02 21:06:59,403 INFO: Exploring floorplans
+2024-08-02 21:07:13,655 INFO: Exploring data layouts
+2024-08-02 21:08:06,235 INFO: Optimizing memory usage
+2024-08-02 21:09:37,526 INFO: Gradient accumulation showed a benefit
+2024-08-02 21:09:37,658 INFO: Post-layout optimizations for <microbatch=64, lanes=7>...
+2024-08-02 21:09:37,679 INFO: Post-layout optimizations for <microbatch=64, lanes=10>...
+2024-08-02 21:09:37,680 INFO: Post-layout optimizations for <microbatch=64, lanes=8>...
+2024-08-02 21:09:37,681 INFO: Post-layout optimizations for <microbatch=64, lanes=11>...
+2024-08-02 21:09:37,682 INFO: Post-layout optimizations for <microbatch=64, lanes=5>...
+2024-08-02 21:09:40,830 INFO: Allocating buffers for <microbatch=64, lanes=7>...
+2024-08-02 21:09:41,943 INFO: Allocating buffers for <microbatch=64, lanes=8>...
+2024-08-02 21:09:41,977 INFO: Allocating buffers for <microbatch=64, lanes=11>...
+2024-08-02 21:09:42,011 INFO: Allocating buffers for <microbatch=64, lanes=10>...
+2024-08-02 21:09:42,014 INFO: Allocating buffers for <microbatch=64, lanes=5>...
+2024-08-02 21:09:42,657 INFO: Code generation for <microbatch=64, lanes=7>...
+2024-08-02 21:09:43,821 INFO: Code generation for <microbatch=64, lanes=8>...
+2024-08-02 21:09:43,986 INFO: Code generation for <microbatch=64, lanes=5>...
+2024-08-02 21:09:44,039 INFO: Code generation for <microbatch=64, lanes=11>...
+2024-08-02 21:09:44,059 INFO: Code generation for <microbatch=64, lanes=10>...
+2024-08-02 21:09:52,882 INFO: Gradient accumulation picked micro batch size 64
+2024-08-02 21:10:08,744 INFO: Compile artifacts successfully written to remote compile directory. Compile hash is: cs_3997308062121820062
+2024-08-02 21:10:08,780 INFO: Compile was successful!
+2024-08-02 21:10:08,780 INFO: Waiting for weight initialization to complete
+2024-08-02 21:10:08,781 INFO: Programming Cerebras Wafer Scale Cluster for execution. This may take a few minutes.
+2024-08-02 21:10:08,946 INFO: Initiating a new execute wsjob against the cluster server.
+2024-08-02 21:10:08,968 INFO: Execute job id: wsjob-bzlmwdcyywzfu7bttr9gz9, remote log path: /n1/wsjob/workdir/job-operator/wsjob-bzlmwdcyywzfu7bttr9gz9
+2024-08-02 21:10:18,994 INFO: Poll ingress status: Waiting for job running, current job status: Initializing, msg: job initializing with config generation.
+2024-08-02 21:10:29,016 INFO: Poll ingress status: Waiting for all Worker pods to be running, current running: 0/1.
+2024-08-02 21:10:39,024 INFO: Poll ingress status: Waiting for all Activation pods to be running, current running: 48/59.
+2024-08-02 21:10:49,036 INFO: Poll ingress status: Waiting for all Weight pods to be running, current running: 17/20.
+2024-08-02 21:10:59,048 INFO: Poll ingress status: Waiting for all Activation pods to be running, current running: 54/59.
+2024-08-02 21:11:09,060 INFO: Poll ingress status: Waiting for all Weight pods to be running, current running: 18/20.
+2024-08-02 21:11:19,067 INFO: Poll ingress status: Waiting for all Activation pods to be running, current running: 54/59.
+2024-08-02 21:11:49,095 INFO: Poll ingress status: Waiting for all Weight pods to be running, current running: 18/20.
+2024-08-02 21:11:59,105 INFO: Poll ingress status: Waiting for all Activation pods to be running, current running: 54/59.
+2024-08-02 21:12:09,117 INFO: Poll ingress status: Waiting for all Weight pods to be running, current running: 18/20.
+2024-08-02 21:12:19,138 INFO: Ingress is ready: Job ingress ready, poll ingress success.
+2024-08-02 21:12:19,380 INFO: Preparing to execute using 1 CSX
+2024-08-02 21:12:52,993 INFO: About to send initial weights
+2024-08-02 21:12:59,491 INFO: Finished sending initial weights
+2024-08-02 21:12:59,492 INFO: Finalizing appliance staging for the run
+2024-08-02 21:13:20,532 INFO: Waiting for device programming to complete
+2024-08-02 21:14:35,817 INFO: Device programming is complete
+2024-08-02 21:14:36,774 INFO: Using network type: ROCE
+2024-08-02 21:14:36,775 INFO: Waiting for input workers to prime the data pipeline and begin streaming ...
+2024-08-02 21:14:36,792 INFO: Input workers have begun streaming input data
+2024-08-02 21:14:38,044 INFO: Appliance staging is complete
+2024-08-02 21:14:38,044 INFO: Beginning appliance run
+2024-08-02 21:20:22,200 INFO: | Train Device=CSX, Step=100, Loss=4.71332, Rate=595.23 samples/sec, GlobalRate=595.23 samples/sec
+2024-08-02 21:26:13,253 INFO: | Train Device=CSX, Step=200, Loss=10.34700, Rate=588.13 samples/sec, GlobalRate=589.25 samples/sec
+2024-08-02 21:26:13,260 INFO: Saving checkpoint at step 200
+2024-08-02 21:26:29,525 INFO: Saved checkpoint model_dir_esm2/checkpoint_200.mdl
+2024-08-02 21:32:04,197 INFO: | Train Device=CSX, Step=300, Loss=4.17420, Rate=585.39 samples/sec, GlobalRate=587.34 samples/sec
+2024-08-02 21:37:56,370 INFO: | Train Device=CSX, Step=400, Loss=4.12672, Rate=583.08 samples/sec, GlobalRate=585.88 samples/sec
+2024-08-02 21:37:56,377 INFO: Saving checkpoint at step 400
+2024-08-02 21:38:13,224 INFO: Saved checkpoint model_dir_esm2/checkpoint_400.mdl
+2024-08-02 21:38:42,917 INFO: Training completed successfully!
+2024-08-02 21:38:42,923 INFO: Processed 819200 training sample(s) in 2716.994790088 seconds.
+
+Connection to one of the CS-2 cluster login nodes requires an MFA passcode for authentication - either an 8-digit passcode generated by an app on your mobile device (e.g. MobilePASS+) or a CRYPTOCard-generated passcode prefixed by a 4-digit pin. This is the same passcode used to authenticate into other ALCF systems, such as Polaris.
+In the examples below, replace ALCFUserID with your ALCF user id.
+To connect to a CS-2 login:
The CS-2 cluster has its own Kubernetes-based system for job submission and queuing.
Jobs are started automatically through the Python framework in modelzoo.common.pytorch.run_utils +Continuous job status for a job is output to stdout/stderr; redirect the output, or consider using a persistent session started with screen, or tmux, or both.
+Jobs that have not yet completed can be listed as shown. Note: this command can take over a minute to complete.
+(venv_cerebras_pt) $ csctl get jobs
+NAME AGE DURATION PHASE SYSTEMS USER LABELS DASHBOARD
+wsjob-thjj8zticwsylhppkbmjqe 13s 1s RUNNING cer-cs2-01 username name=unet_pt https://grafana.cerebras1.lab.alcf.anl.gov/d/WebHNShVz/wsjob-dashboard?orgId=1&var-wsjob=wsjob-thjj8zticwsylhppkbmjqe&from=1691705374000&to=now
+(venv_cerebras_pt) $
+Jobs can be canceled as shown:
+(venv_cerebras_pt) $ csctl cancel job wsjob-eyjapwgnycahq9tus4w7id
+Job canceled successfully
+(venv_cerebras_pt) $
+Jobs can be labeled in the command line that launches them, if they are written with Cerebras's Python framework for running appliance jobs, by adding a command line option of this form: +
+Jobs can also be labeled after they have been started as shown: +
(venv_cerebras_pt) $ csctl label job wsjob-ez6dyfronnsg2rz7f7fqw4 testlabel=test
+job/wsjob-ez6dyfronnsg2rz7f7fqw4 was patched
+(venv_cerebras_pt) $
+Jobs with a particular label/label value can be listed as shown: +
(venv_cerebras_pt) $ csctl get jobs | grep "testlabel=test"
+wsjob-ez6dyfronnsg2rz7f7fqw4 19m SUCCEEDED cer-cs2-02 username testlabel=test,user=username
+(venv_cerebras_pt) $
+See csctl -h for more options.
+Add -h to a command for help for that command, e.g. csctl get -h or csctl cancel -h.
$ csctl -h
+Cerebras cluster command line tool.
+
+Usage:
+ csctl [command]
+
+Available Commands:
+ cancel Cancel job
+ clear-worker-cache Clear the worker cache
+ config View csctl config files
+ get Get resources
+ label Label resources
+ log-export Gather and download logs.
+ types Display resource types
+
+Flags:
+ -d, --debug int higher debug values will display more fields in output objects
+ -h, --help help for csctl
+ --namespace string configure csctl to talk to different user namespaces
+ -v, --version version for csctl
+
+Use "csctl [command] --help" for more information about a command.
+Cerebras documentation for porting code to run on a Cerebras CS-2 system:
+Ways to port your model
A Grafana dashboard provides support for visualizing, querying, and exploring the CS2 system’s metrics and enables to access system logs and traces. +See the Cerebras documentation for the Job Information Dashboard
+Here is a summary (tested to work on Ubuntu and MacOS)
On your work machine with a web browser, e.g. your laptop,
+edit /etc/hosts, using your editor of choice
+
Download the Grafana certificate present on the Cerebras node at /opt/cerebras/certs/grafana_tls.crt to your local machine. To add this certificate to your browser keychain,
+
On your work machine with a web browser, e.g. your laptop,
+tunnel the grafana https port on the cerebras grafana host through to localhost
+
Point a browser at grafana. (Tested with Firefox and Chrome/Brave)
+Open browser to a job grafana url shown in csctl get jobs, adding :8443 to hostname, e.g.
+
https://grafana.cerebras1.lab.alcf.anl.gov:8443/d/WebHNShVz/wsjob-dashboard?orgId=1&var-wsjob=wsjob-49b7uuojdelvtrcxu3cwbw&from=1684859330000&to=noww
+Login to the dashboard with user admin, and password prom-operator
+ + + + + + + + + + + + + + + + + + + +Cerebras jobs are initiated and tracked automatically within the Python framework in modelzoo.common.pytorch.run_utils. This framework interacts with the Cerebras cluster management node.
+Jobs are launched from login nodes. +If you expect a loss of an internet connection for any reason, for long-running jobs we suggest logging into a specific login node and using either screen or tmux to create persistent command line sessions. For details use:2
+ +Follow these instructions to compile and train the fc_mnist PyTorch sample. This models is a couple of fully connected layers plus dropout and RELU.
First, make a virtual environment for Cerebras for PyTorch.
+See Customizing Environments for the procedures for making PyTorch virtual environments for Cerebras.
+If an environment is made in ~/R_2.3.0/, it would be activated as follows:
+
mkdir ~/R_2.3.0
+cd ~/R_2.3.0
+git clone https://github.com/Cerebras/modelzoo.git
+cd modelzoo
+git tag
+git checkout Release_2.3.0
+source ~/R_2.3.0/venv_cerebras_pt/bin/activate
+pip install -r ~/R_2.3.0/modelzoo/requirements.txt
+cd ~/R_2.3.0/modelzoo/src/cerebras/modelzoo/fc_mnist/pytorch
+Next, edit configs/params.yaml, making the following changes:
+ train_input:
+- data_dir: "./mnist"
++ data_dir: "/software/cerebras/dataset/fc_mnist/data/mnist/train"
+and
+ eval_input:
+- data_dir: "./mnist"
++ data_dir: "/software/cerebras/dataset/fc_mnist/data/mnist/train"
+If you want to have the sample download the dataset, you will need to specify absolute paths for the "data_dir"s.
+To run the sample:
+export MODEL_DIR=model_dir
+# deletion of the model_dir is only needed if sample has been previously run
+if [ -d "$MODEL_DIR" ]; then rm -Rf $MODEL_DIR; fi
+python run.py CSX --job_labels name=fc_mnist --params configs/params.yaml --num_csx=1 --mode train --model_dir $MODEL_DIR --mount_dirs /home/$(whoami)/ /software --python_paths /home/$(whoami)/R_2.3.0/modelzoo/src --compile_dir /$(whoami) |& tee mytest.log
+A successful fc_mnist PyTorch training run should finish with output resembling the following:
+2023-11-29 18:13:13,048 INFO: | Train Device=CSX, Step=1950, Loss=2.28834, Rate=397.31 samples/sec, GlobalRate=433.98 samples/sec
+2023-11-29 18:13:13,555 INFO: | Train Device=CSX, Step=2000, Loss=2.34778, Rate=395.69 samples/sec, GlobalRate=431.83 samples/sec
+2023-11-29 18:13:13,555 INFO: Saving checkpoint at step 2000
+2023-11-29 18:13:17,242 INFO: Saved checkpoint model_dir/checkpoint_2000.mdl
+2023-11-29 18:13:55,517 INFO: Heartbeat thread stopped for wsjob-fpwqt7maq8a5mxvblwwzbu.
+2023-11-29 18:13:55,523 INFO: Training completed successfully!
+2023-11-29 18:13:55,523 INFO: Processed 4000 sample(s) in 51.230697212 seconds.
+The Cerebras CS-2 is a wafer-scale deep learning accelerator comprising 850,000 processing cores, each providing 48KB of dedicated SRAM memory for an on-chip total of 40GB and interconnected to optimize bandwidth and latency. Its software platform integrates the popular machine learning framework PyTorch.
+The ALCF CS-2 systems are configured as a Cerebras Wafer-Scale Cluster, designed to support large-scale models (up to and well beyond 1 billion parameters) and large-scale inputs. The cluster contains two CS-2 systems and can distribute jobs across one or both CS-2 systems in a data-parallel framework. The supporting CPU cluster consists of MemoryX, SwarmX, management, and input worker nodes. The Cerebras Wafer-Scale cluster is run as an appliance: a user submits a job to the appliance, and the appliance manages preprocessing and streaming of the data, IO, and device orchestration within the appliance. It provides programming via PyTorch, with data-parallel distribution when using more than one CS-2. This installation supports both Pipelined execution for models up to 1 billion parameters and Weight Streaming execution for models up to and above 1 billion parameters.
+ + + + +The public Cerebras documentation is available here.
+A typical Cerebras Wafer-Scale Cluster is shown in the figure.
+Users connect (ssh) to one of the three login nodes. Either ssh to cerebras.ai.alcf.anl.gov, which randomly resolves to one of cer-login-0[1-3].ai.alcf.anl.gov, or ssh to a specific node, cer-login-01.ai.alcf.anl.gov, cer-login-02.ai.alcf.anl.gov, cer-login-03.ai.alcf.anl.gov.
+The rest of the nodes in the cluster infrastructure are not directly accessible, except by admins.
+The trees /home, /projects, and /software are shared across all three login nodes, the relevant cluster infrastructure nodes, and all ALCF AI testbed platforms.
+
(Figure from +https://docs.cerebras.net/en/latest/wsc/cerebras-basics/how-cerebras-works.html)
+As indicated in the figures, the CS-2 nodes on the right are responsible only for running and accelerating the computations for training and predictions with the model. The other work, including compilation, is performed by input nodes, and by MemoryX nodes, which are used for weight storage and broadcast, and SwarmX nodes, which are used for gradient accumulation. Some model verification work can be done on login nodes.
+ + + + + + + + + + + + + + + +See ALCF's Jupyter Instructions, and +Tunneling and forwarding ports. The Cerebras login nodes are direct login; tunneling and port forwarding do not involve jump hosts.
+ + + + + + + + + + + + + + + +Users have a shared home filesystem /home shared across the ALCF AI testbed systems, including the login and compute nodes. Default user quota is 1 TB storage and 1,000,000 files. This space is backed up.
The team project/campaign file system /projects is intended to facilitate project collaboration and is accessible to the team members of your project that have an ALCF account. Default group storage quota is 2 TB and 2,000,000 files. Please note that this space isn't backed up. Our policy is that data will be purged from disk 6 months after project completion.
Users can transfer data to and from the AI testbed using Globus or tools such as scp or rsync.
We have a Globus endpoint each to move data to and from the /projects and /home filesystem respectively.
alcf#ai_testbed_projects for the /projects file systemalcf#ai_testbed_home for the /home files system Relevant information regarding using globus can be found here
+ALCF data policies is available here
+Please Note: The basic level of protection provided is UNIX file level permissions; it is the user's responsibility to ensure that file permissions and umasks are set to match their needs.
+ + + + + + + + + + + + + + + +++Note: Conversion of CT to the various machines is meant to be a tutorial as to how +to convert a model.
+
Cerebras cannot support CT and UNets in general as of 4/25/23.
+Alex has been very busy with conferences, etc.
+He ran CT but, it ran on the CPU. He has stated that it may need to be completely written +using, I can't remember which, Poplar or PopArt. If that is necessary, Venkat should +make the call.
+Repo: https://github.com/argonne-lcf/user-guides.git +Branch: feature/Habana002-DNP +File: docs/ai-testbed/habana/CosmicTagger-Conversion.md
+SN has a highly-engineered version of CT.
+They are working to support CT OOB, Out-Of-Box.
+Repo: https://github.com/argonne-lcf/user-guides.git +Branch: Talk to Bill.
+Repo: https://github.com/argonne-lcf/user-guides.git
+When you change back to 3.2, use virtual-environments.md from the commit a4ce3b5598f4d6feee7ca58accde1a6a0ea84244 "virtual-environments.md with 3.2 edits."
+Repo: https://github.com/argonne-lcf/user-guides.git +Branch: feature/Groq001-DNP
+Repo: https://github.com/argonne-lcf/user-guides.git +Branch: feature/Habana002-DNP
+Repo: https://github.com/argonne-lcf/user-guides.git
+ + + + + + + + + + + + + + + +
The ALCF AI Testbed houses some of the most advanced AI accelerators for scientific research.
+The goal of the testbed is to enable explorations into next-generation machine learning applications and workloads, enabling the ALCF and its user community to help define the role of AI accelerators in scientific computing and how to best integrate such technologies with supercomputing resources.
+The AI accelerators complement the ALCF's current and next-generation supercomputers to provide a state-of-the-art computing environment that supports pioneering research at the intersection of AI, big data, and high performance computing (HPC).
+The platforms are equipped with architectural features that support AI and data-centric workloads, making them well suited for research tasks involving the growing deluge of scientific data produced by powerful tools, such as supercomputers, light sources, telescopes, particle accelerators, and sensors. In addition, the testbed will allow researchers to explore novel workflows that combine AI methods with simulation and experimental science to accelerate the pace of discovery.
+Researchers interested in using the AI Testbed’s Cerebras CS-2, SambaNova DataScale SN30, Graphcore Bow Pod64 and GroqRack platforms can now submit project proposals via the ALCF’s Director’s Discretionary program. Access to additional testbed resources, including Habana accelerators, will be announced at a later date.
Submit your proposal requests at: Allocation Request Page
+Request a Director's Discretionary project on SambaNova/Cerebras/Graphcore/Groq.
+Apply for an ALCF account after the project request is approved. Choose the SambaNova/Cerebras/Graphcore/Groq project that your PI has created at ALCF. If you have an active ALCF account, request to join the project after your project is approved.
+Transfer data to ALCF using Globus after your account has been created.
+a. The endpoint for your data in ALCF is alcf#ai_testbed_projects with the path to your project being /<project name>.
b. The endpoint for your home directory on the AI Testbeds in ALCF is alcf#ai_testbed_home.
Add/invite team members to your ALCF project on SambaNova/Cerebras/Graphcore/Groq.
+The documentation is based on MkDocs and source files are +on GitHub. You can contribute to the documentation by creating a pull request.
+ + + + + + + + + + + + + + + + +Poplar SDK
+ PyTorch for the IPU: User Guide
+ Targetting the IPU from Tensorflow 2
+ IPU programming guide
+ Examples
+ Examples Github Repo
+ POD systems
+ POD64 specs
Graphcore provides examples of some well-known AI applications in their repository at https://github.com/graphcore/examples.git. +Clone the examples repository to your personal directory structure: +
+Change directory: +
+Execute the command: +
+The expected output will resemble the following:
+srun: job 10671 queued and waiting for resources
+srun: job 10671 has been allocated resources
+TrainingModelWithLoss(
+ (model): Network(
+ (layer1): Block(
+ (conv): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))
+ (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
+ (relu): ReLU()
+ )
+ (layer2): Block(
+ (conv): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))
+ (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
+ (relu): ReLU()
+ )
+ (layer3): Linear(in_features=1600, out_features=128, bias=True)
+ (layer3_act): ReLU()
+ (layer3_dropout): Dropout(p=0.5, inplace=False)
+ (layer4): Linear(in_features=128, out_features=10, bias=True)
+ (softmax): Softmax(dim=1)
+ )
+ (loss): CrossEntropyLoss()
+)
+Epochs: 0%| | 0/10 [00:00<?,[23:27:06.753] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 10
+Graph compilation: 100%|██████████| 100/100 [00:00<00:00]
+Epochs: 100%|██████████| 10/10 [01:17<00:00, 7.71s/it]
+Graph compilation: 100%|██████████| 100/100 [00:00<00:00]
+Accuracy on test set: 96.85%██████| 100/100 [00:00<00:00]
+Create a TensorFlow2 environment as explained in the tensorflow-2-environment-setup and activate the same. +
+Change directory: +
+Execute the command:
+ +The expected output will resemble the following:
+srun: job 10672 queued and waiting for resources
+srun: job 10672 has been allocated resources
+2023-08-22 23:35:02.925033: I tensorflow/compiler/plugin/poplar/driver/poplar_platform.cc:43] Poplar version: 3.3.0 (de1f8de2a7) Poplar package: b67b751185
+2023-08-22 23:35:06.119772: I tensorflow/compiler/plugin/poplar/driver/poplar_executor.cc:1619] TensorFlow device /device:IPU:0 attached to 1 IPU with Poplar device ID: 0
+2023-08-22 23:35:07.087287: I tensorflow/compiler/mlir/mlir_graph_optimization_pass.cc:185] None of the MLIR Optimization Passes are enabled (registered 2)
+2023-08-22 23:35:07.351132: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:210] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
+2023-08-22T23:35:09.469066Z PL:POPOPS 3545299.3545299 W: createOutputForElementWiseOp 'while/sparse_categorical_crossentropy/SparseSoftmaxCrossEntropyWithLogits/SparseSoftmaxCrossEntropyWithLogits/fusion.3/Op/Equal/Out' ({32,10}): No suitable input found, creating new variable with linear tile mapping
+2023-08-22 23:35:18.532415: I tensorflow/compiler/jit/xla_compilation_cache.cc:376] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
+Epoch 1/4
+2000/2000 [==============================] - 13s 6ms/step - loss: 0.6220
+Epoch 2/4
+2000/2000 [==============================] - 1s 262us/step - loss: 0.3265
+Epoch 3/4
+2000/2000 [==============================] - 1s 273us/step - loss: 0.2781
+Epoch 4/4
+2000/2000 [==============================] - 1s 289us/step - loss: 0.2482
+Create and activate a fresh PopTorch environment poptorch33_resnet50_env as outlined in the virtual environment section, then activate it.
+
Change directory +
+Change directory: +
+Open configs.yml with your favorite editor. +Find in the resnet50 section + +and change it to: + + + +The scripts to train a ResNet50 PyTorch model on Pod4 is located at https://github.com/graphcore/examples/tree/master/vision/cnns/pytorch/train
+Set the following environmental variables. +
+To run 4 replicas (a total for 4 IPUs) of the ResNet50 model: +Make a script with the following contents, called poprun_unet.sh#!/bin/bash
+poprun -vv --vipu-partition=slurm_${SLURM_JOBID} --num-instances=1 --num-replicas=4 --executable-cache-path=$PYTORCH_CACHE_DIR python3 /home/$USER/graphcore/examples/vision/cnns/pytorch/train/train.py --config resnet50-pod4 --imagenet-data-path /mnt/localdata/datasets/imagenet-raw-dataset --epoch 2 --validation-mode none --dataloader-worker 14 --dataloader-rebatch-size 256
+This model is run with the imagenet dataset.
+The expected output starts with this: +
srun: job 10675 queued and waiting for resources
+srun: job 10675 has been allocated resources
+23:48:29.160 3555537 POPRUN [I] V-IPU server address picked up from 'vipu': 10.1.3.101:8090
+23:48:29.160 3555537 POPRUN [D] Connecting to 10.1.3.101:8090
+23:48:29.162 3555537 POPRUN [D] Status for partition slurm_10673: OK (error 0)
+23:48:29.162 3555537 POPRUN [I] Partition slurm_10673 already exists and is in state: PS_ACTIVE
+23:48:29.163 3555537 POPRUN [D] The reconfigurable partition slurm_10673 is OK
+ ===========================
+| poprun topology |
+|===========================|
+| hosts | gc-poplar-02 |
+|-----------|---------------|
+| ILDs | 0 |
+|-----------|---------------|
+| instances | 0 |
+|-----------|---------------|
+| replicas | 0 | 1 | 2 | 3 |
+ ---------------------------
+23:48:29.163 3555537 POPRUN [D] Target options from environment: {}
+23:48:29.163 3555537 POPRUN [D] Target options from V-IPU partition: {"ipuLinkDomainSize":"4","ipuLinkConfiguration":"default","ipuLinkTopology":"mesh","gatewayMode":"true","instanceSize":"4"}
+23:48:29.207 3555537 POPRUN [D] Found 1 devices with 4 IPUs
+23:48:29.777 3555537 POPRUN [D] Attached to device 6
+23:48:29.777 3555537 POPRUN [I] Preparing parent device 6
+23:48:29.777 3555537 POPRUN [D] Device 6 ipuLinkDomainSize=64, ipuLinkConfiguration=Default, ipuLinkTopology=Mesh, gatewayMode=true, instanceSize=4
+23:48:33.631 3555537 POPRUN [D] Target options from Poplar device: {"ipuLinkDomainSize":"64","ipuLinkConfiguration":"default","ipuLinkTopology":"mesh","gatewayMode":"true","instanceSize":"4"}
+23:48:33.631 3555537 POPRUN [D] Using target options: {"ipuLinkDomainSize":"4","ipuLinkConfiguration":"default","ipuLinkTopology":"mesh","gatewayMode":"true","instanceSize":"4"}
+Graph compilation: 100%|██████████| 100/100 [00:04<00:00][1,0]<stderr>:2023-08-22T23:49:40.103248Z PO:ENGINE 3556102.3556102 W: WARNING: The compile time engine option debug.branchRecordTile is set to "5887" when creating the Engine. (At compile time it was set to 1471)
+[1,0]<stderr>:
+Loss:6.7539 [1,0]<stdout>:[INFO] Epoch 1████▌| 75/78 [02:42<00:06, 2.05s/it][1,0]<stderr>:
+[1,0]<stdout>:[INFO] loss: 6.7462,
+[1,0]<stdout>:[INFO] accuracy: 0.62 %
+[1,0]<stdout>:[INFO] throughput: 7599.7 samples/sec
+[1,0]<stdout>:[INFO] Epoch 2/2
+Loss:6.7462 | Accuracy:0.62%: 100%|██████████| 78/78 [02:48<00:00, 2.16s/it][1,0]<stderr>:
+Loss:6.2821 | Accuracy:2.42%: 96%|█████████▌| 75/7[1,0]<stdout>:[INFO] Epoch 2,0]<stderr>:
+[1,0]<stdout>:[INFO] loss: 6.2720,
+[1,0]<stdout>:[INFO] accuracy: 2.48 %
+[1,0]<stdout>:[INFO] throughput: 8125.8 samples/sec
+[1,0]<stdout>:[INFO] Finished training. Time: 2023-08-22 23:54:57.853508. It took: 0:05:26.090631
+Loss:6.2720 | Accuracy:2.48%: 100%|██████████| 78/78 [02:37<00:00, 2.02s/it][1,0]<stderr>:
+[1,0]<stderr>:/usr/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 14 leaked semaphore objects to clean up at shutdown
+[1,0]<stderr>: warnings.warn('resource_tracker: There appear to be %d '
+23:55:02.722 3555537 POPRUN [I] mpirun (PID 3556098) terminated with exit code 0
+The scripts to train a GPT-2 pytorch model on the POD16 are located at https://github.com/graphcore/examples/tree/master/nlp/gpt2/pytorch
+In order to run the GPT-2 Pytorch model, create a new popTorch virtual environment poptorch33_gpt2 as described in the virtual environment section and activate it.
+ +Change directory: +
+The command for the GPT2 model is as follows is as follows. +
/opt/slurm/bin/srun --ipus=16 python /home/$USER/graphcore/examples/nlp/gpt2/pytorch/train_gpt2.py --model gpt2 --ipus-per-replica 4 --replication-factor 4 --gradient-accumulation 2048 --device-iterations 8 --batch-size 1 --layers-per-ipu 0 4 4 4 --matmul-proportion 0.15 0.15 0.15 0.15 --max-len 1024 --optimizer AdamW --learning-rate 0.00015 --lr-schedule cosine --lr-warmup 0.01 --remap-logit True --enable-sequence-serialized True --embedding-serialization-factor 4 --recompute-checkpoint-every-layer True --enable-half-partials True --replicated-tensor-sharding True --dataset 'generated' --epochs 1
+gpt2 model that fits on 4 IPUS indicated by --ipus-per-replica. The --replication-factor indicates how many times the model is replicated in a data parallel manner (4 in the above example). Hence the total number of IPUs used in this example is 16.
+The effective global batch size in this example is (micro)batch-size * gradient-accumulation * replication-factor = 1 x 2048 x 4 = 8192. The device iterations indicates the total number samples loaded in 1 training step = global batch size * device iterations = 8192*8 = 65536. To learn more about these parameters and in general batching of IPUs refer IPU batching .
+The above example is running with generated or synthetic data. To use the same example with a real world dataset, refer to data setup.
Expected output starts with the following: +
srun: job 10697 queued and waiting for resources
+srun: job 10697 has been allocated resources
+Building (if necessary) and loading remap_tensor_ce.
+Failed to find compiled extension; rebuilding.
+Building (if necessary) and loading residual_add_inplace_pattern.
+Model initializing
+-------------------- Device Allocation --------------------
+Embedding --> IPU 0
+Layer 0 --> IPU 1
+Layer 1 --> IPU 1
+Layer 2 --> IPU 1
+Layer 3 --> IPU 1
+Layer 4 --> IPU 2
+Layer 5 --> IPU 2
+Layer 6 --> IPU 2
+Layer 7 --> IPU 2
+Layer 8 --> IPU 3
+Layer 9 --> IPU 3
+Layer 10 --> IPU 3
+Layer 11 --> IPU 3
+LM_head --> IPU 0
+step 0 of epoch 0, loss: 10.913220405578613, acc: 2.0071864128112793e-05, lr: 0.00012803300858899104, throughput: 646.8439205981404 samples/sec
+step 1 of epoch 0, loss: 10.836345672607422, acc: 1.9788742065429688e-05, lr: 7.5e-05, throughput: 1058.0979097185766 samples/sec
+step 2 of epoch 0, loss: 10.831247329711914, acc: 2.0518898963928223e-05, lr: 2.1966991411008938e-05, throughput: 1058.7595523807183 samples/sec
+step 3 of epoch 0, loss: 10.829034805297852, acc: 1.990795135498047e-05, lr: 0.0, throughput: 1059.6762623043378 samples/sec
+++ + + + + + + + + + + + + + + +Note: The graph compilation for a large model like GPT-2 takes about half an hour.
+
Connection to a Graphcore node is a two-step process.
+The first step is to ssh from a local machine to the login node.
+The second step is to log in to a Graphcore node from the login node.
+
Login to the Graphcore login node from your local machine using the below command. This uses the ALCF account ID that uses the password generated from the MobilePASS+.
+++ +Note: In the examples below, replace ALCFUserID with your ALCF user id.
+
++Note: Use the ssh "-v" option in order to debug any ssh problems.
+
Once you are on the login node, ssh to one of the Graphcore nodes.
+ssh gc-poplar-02.ai.alcf.anl.gov
+# or
+ssh gc-poplar-03.ai.alcf.anl.gov
+# or
+ssh gc-poplar-04.ai.alcf.anl.gov
+++ + + + + + + + + + + + + + + +**Note:
+ssh gc-poplar-01.ai.alcf.anl.govis not accessible to users. However, its IPU resources are assigned by the slurm tasks.
ALCF's Graphcore POD64 system uses Slurm for job submission and queueing. Below are some of the important commands for using Slurm. For more information refer to Slurm Documentation.
+++NOTE: Jobs that require IPUs will fail unless launched with
+srunorsbatch. +NOTE: There is a single Slurm scheduler for the Graphcore POD64.
The Slurm command srun can be used to run individual Python scripts (or other programs) in parallel with other scripts on a cluster managed by Slurm. An example of srun usage is shown below. Use the --ipus= option to specify the number of IPUs required for the run.
Example:
+ +Alternatively, these jobs can be submitted to the Slurm workload manager through a batch script by using the sbatch command. To do this, create a bash script (submit-mnist-poptorch-job.sh here as an example) with the commands that you want to execute.
Then pass the bash script as an input to the sbatch command as shown below, requesting the number of IPUs required:
The squeue command provides information about jobs located in the Slurm scheduling queue.
$ squeue
+ JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
+ 2572 p64 Graphcor username R 1:12 1 gc-poplar-02
+SInfo is used to view partition and node information for a system running Slurm.
+$ sinfo
+PARTITION AVAIL TIMELIMIT NODES STATE NODELIST
+p64* up infinite 3 idle gc-poplar-[02-04]
+For more information, see SInfo.
+SCancel is used to signal or cancel jobs, job arrays, or job steps.
+ + + + + + + + + + + + + + + + +The command gc-monitor is Graphcore's device usage monitor. Run it as follows for ordinary monitoring. See gc-monitor --help for other options.
export IPUOF_VIPU_API_HOST=10.1.3.101
+gc-monitor --no-card-info --all-partitions
+# or watch gc-monitor --no-card-info --all-partitions
+++Note: if there are no partitions active, gc-monitor will core dump:
+Segmentation fault (core dumped)
The output will look something like:
++--------------------------------------------------------------+-----------------------+
+| IPUs in slurm_2616 attached from other namespaces | Board |
++----+------------------------------+--------------+-----------+-----------+-----------+
+| ID | Application host | Clock | Temp | Temp | Power |
++----+------------------------------+--------------+-----------+-----------+-----------+
+| 0 | gc-poplar-02 | 1850MHz | 24.2 C | 21.1 C | 92.3 W |
++----+------------------------------+--------------+-----------+-----------+-----------+
+The command gc-info is used to display device information. See gc-info --help for more options.
To list devices, +
+The command gc-info lists the partition and different IPU Id's along with the multi-IPU configuration IDs.
-+- Id: [0], target: [Fabric], IPU-M host: [10.1.5.1], IPU#: [3]
+-+- Id: [1], target: [Fabric], IPU-M host: [10.1.5.1], IPU#: [2]
+-+- Id: [2], target: [Fabric], IPU-M host: [10.1.5.1], IPU#: [1]
+One may also display detailed information for a specific device. The devices are numbered 0-63. For example,
+ +See gc-info --help for more information.
Use one of
+ + + + + + + + + + + + + + + + +++Note: Please be mindful of how you are using the system. +For example, consider running larger jobs in the evening or on weekends.
+
Running of any model or application includes graph compilation of the model that is then deployed on the IPUs. Below is the description of training a neural network for classification on the MNIST dataset using the PopTorch (pytorch framework optimized for IPU).
+Graphcore provides examples of some well-known AI applications in their repository at https://github.com/graphcore/examples.git.
+Clone the examples repository to your personal directory structure, and checkout the v3.3.0 release:
+mkdir ~/graphcore
+cd ~/graphcore
+git clone https://github.com/graphcore/examples.git
+cd examples
+git checkout v3.3.0
+Follows the steps at Poptorch environment setup to enable the Poplar SDK.
+ +Change directory and install packages specific to the MNIST model:
+ +Execute the command:
+ +All models are run using Slurm, with the --ipus indicating how many IPUs are need to be allocated for the model being run. This example uses a batchsize of 8, and run for 10 epochs. It also set the device iteration to 50 which is the number of iterations the device should run over the data before returning to the user. The dataset used in the example is derived from the TorchVision and the PopTorch dataloader is used to load the data required for the 50 device iterations from the host to the device in a single step.
The model used here is a simple CNN based model with an output from a classifier (softmax layer).
+A simple Pytorch model is translated to a PopTorch model using poptorch.Options().
+poptorch.trainingModel is the model wrapping function on the Pytorch model. The first call to trainingModel will compile the model for the IPU. You can observe the compilation process as part of output of the above command.
Graph compilation: 3%|▎ | 3/100 [00:00<00:03]2023-04-26T16:53:21.225944Z PL:POPLIN 3680893.3680893 W: poplin::preplanMatMuls() is deprecated! Use poplin::preplan() instead
+Graph compilation: 100%|██████████| 100/100 [00:20<00:00]2023-04-26T16:53:38.241395Z popart:session 3680893.3680893
+The artifacts from the graph compilations is cached in the location set by the flag POPTORCH_CACHE_DIR, where the .popef file corresponding to the model under consideration is cached.
The expected output will start with downloads followed by and we can observe the model used by the model, the progress bar of the compilation process, and the training progress bar.
+srun: job 10671 queued and waiting for resources
+srun: job 10671 has been allocated resources
+TrainingModelWithLoss(
+ (model): Network(
+ (layer1): Block(
+ (conv): Conv2d(1, 32, kernel_size=(3, 3), stride=(1, 1))
+ (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
+ (relu): ReLU()
+ )
+ (layer2): Block(
+ (conv): Conv2d(32, 64, kernel_size=(3, 3), stride=(1, 1))
+ (pool): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
+ (relu): ReLU()
+ )
+ (layer3): Linear(in_features=1600, out_features=128, bias=True)
+ (layer3_act): ReLU()
+ (layer3_dropout): Dropout(p=0.5, inplace=False)
+ (layer4): Linear(in_features=128, out_features=10, bias=True)
+ (softmax): Softmax(dim=1)
+ )
+ (loss): CrossEntropyLoss()
+)
+Epochs: 0%| | 0/10 [00:00<?,[23:27:06.753] [poptorch:cpp] [warning] [DISPATCHER] Type coerced from Long to Int for tensor id 10
+Graph compilation: 100%|██████████| 100/100 [00:00<00:00]
+Epochs: 100%|██████████| 10/10 [01:17<00:00, 7.71s/it]
+Graph compilation: 100%|██████████| 100/100 [00:00<00:00]
+Accuracy on test set: 96.85%██████| 100/100 [00:00<00:00]
+Refer to the script to learn more about this example.
+Example Programs lists the different example applications with corresponding commands for each of the above steps.
+ + + + + + + + + + + + + + + +The Graphcore Bow-Pod64 system is the latest-generation AI accelerator from Graphcore. This is a one-rack system consisting of 64 Bow-class Intelligence Processing Units (IPU) with a custom interconnect. The system provides for an aggregate 22 Petaflops/s of performance in half precision. It has a total of 57.6 GB In-Processor-Memory with a total of 94,208 IPU cores. The system consists of four servers for data-processing.
+For more details refer to the POD64 spec
+
+(Figure from
+https://www.graphcore.ai/products/poplar)
The Graphcore software stack includes support for TensorFlow and PyTorch using the Poplar SDK. The Poplar® SDK is t is the toolchain specifically designed for creating graph software for ML applications. It integrates with the traditional ML frameworks like PyTorch and TensorFlow allowing users to port their existing code to the IPU hardware-specific code. The various components of the poplar SDK stack are shown in the figure. It includes the PopTorch framework which is a wrapper over the PyTorch framework optimized to the IPU hardware. It also enlists the different PopLibs libraries supported, which enables to construct graphs, define tensor data and control how the code and data are mapped onto the IPU for execution.
+ + + + + + + + + + + + + + + +Follow all the instructions in Getting Started to log into a Graphcore node.
+Graphcore provides examples of some well-known AI applications in their repository at https://github.com/graphcore/examples.git.
+Clone the examples repository to your personal directory structure:
+ +Establish a virtual environment.
+mkdir -p ~/venvs/graphcore
+rm -rf ~/venvs/graphcore/poptorch31_rn50_env
+virtualenv ~/venvs/graphcore/poptorch31_rn50_env
+source ~/venvs/graphcore/poptorch31_rn50_env/bin/activate
+Install PopTorch.
+POPLAR_SDK_ROOT=/software/graphcore/poplar_sdk/3.1.0
+export POPLAR_SDK_ROOT=$POPLAR_SDK_ROOT
+pip install $POPLAR_SDK_ROOT/poptorch-3.1.0+98660_0a383de63f_ubuntu_20_04-cp38-cp38-linux_x86_64.whl
+Establish the following environment variables.
+mkdir ${HOME}/tmp
+export TF_POPLAR_FLAGS=--executable_cache_path=${HOME}/tmp
+export POPTORCH_CACHE_DIR=${HOME}/tmp
+export POPART_LOG_LEVEL=WARN
+export POPLAR_LOG_LEVEL=WARN
+export POPLIBS_LOG_LEVEL=WARN
+export PYTHONPATH=/software/graphcore/poplar_sdk/3.1.0/poplar-ubuntu_20_04-3.1.0+6824-9c103dc348/python:$PYTHONPATH
+Set up the ssh key on gc-poplar-01.
+On gc-poplar-01:
+mkdir ~/.ssh
+cd ~/.ssh
+ssh-keygen -t rsa -b 4096
+#Accecpt default filename of id_rsa
+#Enter passphrase (empty for no passphrase):
+#Enter same passphrase again:
+cat id_rsa.pub >> authorized_keys
+You should see:
+# gc-poplar-01:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-01:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-01:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-01:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-01:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+You should see:
+# gc-poplar-02:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-02:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-02:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-02:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-02:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+You should see:
+# gc-poplar-03:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-03:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-03:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-03:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-03:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+You should see:
+# gc-poplar-04:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-04:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-04:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-04:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+# gc-poplar-04:22 SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.5
+benchmarks.ymlUpdate ${HOME}/graphcore/examples/vision/cnns/pytorch/train/benchmarks.yml +with your favorite editor to match benchmarks.yml.
+configs.ymlUpdate ${HOME}/graphcore/examples/vision/cnns/pytorch/train/configs.yml +with your favorite editor. At about line 30, change use_bbox_info: true to +use_bbox_info: false.
+Scale and benchmark ResNet50.
+++Note: The number at the end of each line indicates the number of IPUs.
+Note: Use screen because every run is long.
+
"PopRun exposes this control with the --process-placement flag and provides multiple pre-defined strategies. By default (and with --process-placement spreadnuma), PopRun is designed to be NUMA-aware. On each host, all the available NUMA nodes are divided among the instances. This means that each instance is bound to execute on and allocate memory from its assigned NUMA nodes, ensuring memory access locality. This strategy maximises memory bandwidth and is likely to yield optimal performance for most of the data loading workloads in machine learning." [Multi-Instance Multi-Host(https://docs.graphcore.ai/projects/poprun-user-guide/en/latest/launching.html#multi-instance-multi-host)
+Move to the correct directory and establish the datasets directory.
+cd ${HOME}/graphcore/examples/vision/cnns/pytorch/train
+export DATASETS_DIR=/mnt/localdata/datasets/
+One may use any of the following commands to run ResNet50 on one to sixteen IPUs.
+python3 -m examples_utils benchmark --spec benchmarks.yml --benchmark pytorch_resnet50_train_real_1
+python3 -m examples_utils benchmark --spec benchmarks.yml --benchmark pytorch_resnet50_train_real_2
+python3 -m examples_utils benchmark --spec benchmarks.yml --benchmark pytorch_resnet50_train_real_4
+python3 -m examples_utils benchmark --spec benchmarks.yml --benchmark pytorch_resnet50_train_real_8
+python3 -m examples_utils benchmark --spec benchmarks.yml --benchmark pytorch_resnet50_train_real_pod16
+++Note: One must complete the instructions on Multi-node Setup before running this example.
+
HOST1=`ifconfig eno1 | grep "inet " | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' | head -1`
+OCT123=`echo "$HOST1" | cut -d "." -f 1,2,3`
+OCT4=`echo "$HOST1" | cut -d "." -f 4`
+HOST2=$OCT123.`expr $OCT4 + 1`
+HOST3=$OCT123.`expr $OCT4 + 2`
+HOST4=$OCT123.`expr $OCT4 + 3`
+export HOSTS=$HOST1,$HOST2,$HOST3,$HOST4
+export CLUSTER=c16
+export IPUOF_VIPU_API_PARTITION_ID=p64
+export TCP_IF_INCLUDE=$OCT123.0/8
+export IPUOF_VIPU_API_HOST=$HOST1
+This runs to convergence. It uses all 64 IPUs for more than 12 hours.
+++Note: This should only be used if absolutely required.
+
Execute:
+python3 -m examples_utils benchmark --spec benchmarks.yml --benchmark pytorch_resnet50_train_real_pod64
+python3 -m examples_utils benchmark --spec benchmarks.yml --benchmark pytorch_resnet50_train_real_pod64_conv
+[INFO] 2022-12-16 17:07:32: Total runtime: 3956.836479 seconds
+[INFO] 2022-12-16 17:07:32: throughput = '7527.626315789474'
+[INFO] 2022-12-16 17:07:32: accuracy = '57.41'
+[INFO] 2022-12-16 17:07:32: loss = '2.8153'
+[INFO] 2022-12-16 17:07:33: Total compile time: 429.59 seconds
+[INFO] 2022-12-16 15:56:23: Total runtime: 5866.494071 seconds
+[INFO] 2022-12-16 15:56:23: throughput = '4798.778947368421'
+[INFO] 2022-12-16 15:56:23: accuracy = '68.23'
+[INFO] 2022-12-16 15:56:23: loss = '2.3148'
+[INFO] 2022-12-16 15:56:24: Total compile time: 418.75 seconds
+[INFO] 2022-12-16 04:05:28: Total runtime: 3070.994553 seconds
+[INFO] 2022-12-16 04:05:28: throughput = '9959.821052631578'
+[INFO] 2022-12-16 04:05:28: accuracy = '67.76'
+[INFO] 2022-12-16 04:05:28: loss = '2.338'
+[INFO] 2022-12-16 04:05:29: Total compile time: 377.4 seconds
+[INFO] 2022-12-16 02:46:45: Total runtime: 1831.437598 seconds
+[INFO] 2022-12-16 02:46:45: throughput = '19865.263157894733'
+[INFO] 2022-12-16 02:46:45: accuracy = '64.94'
+[INFO] 2022-12-16 02:46:45: loss = '2.4649'
+[INFO] 2022-12-16 02:46:46: Total compile time: 386.27 seconds
+Epochs: 20
+[INFO] 2022-12-15 22:01:14: Total runtime: 1297.274336 seconds
+[INFO] 2022-62:01:14: throughput = '39057.447368421046'
+[INFO] 2022-12-15 22:01:14: accuracy = '57.43'
+[INFO] 2022-12-15 22:01:14: loss = '2.8162'
+[INFO] 2022-12-15 22:01:16: Total compile time: 397.08 seconds
+The intent of this page is to show conceptually how to convert a model to run on the Graphcore system. +It is not necessary to convert CosmicTagger because it has already been converted and is +located at CosmicTagger on the Graphcore branch. +The original is located at CosmicTagger.
+The first step to converting a model is to verify that it runs on the CPU. This step has been verified for CosmicTagger.
+CosmicTagger can run on multiple machines. As such, it is necessary to specify the architecture +that one is using. For example, CPU or GPU. The architecture is stored in the +ComputeMode class.
+Edit src/config/config.py. Add IPU to the ComputeMode class.
+ +Edit src/utils/torch/trainer.py.
+PopTorch is Graphcore's extension of PyTorch.
+Import poptorch at the top of the file.
+ +Wrap the model using poptorch.trainingModel() so that it may be ran on IPUs for training.
+Wrap the model using poptorch.inferenceModel() when not training.
+Find the following code around line 90 in the init_network method.
+ # Foregoing any fusions as to not disturb the existing ingestion pipeline
+ if self.is_training() and self.args.mode.quantization_aware:
+ self._raw_net.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
+ self._net = torch.quantization.prepare_qat(self._raw_net)
+ else:
+ self._net = self._raw_net
+After the above code, add:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ if self.is_training():
+ opts = poptorch.Options()
+ self._net = poptorch.trainingModel(self._net, opts, optimizer=torch.optim.SGD(self._net.parameters(), lr=1e-3))
+ else:
+ self._net = poptorch.inferenceModel(self._net)
+See poptorch.trainingModel() and poptorch.inferenceModel() for more information.
+There is also a Build the Model tutorial.
+Update init_optimizer() to use the poptorch class instead of the torch class as needed.
+Change:
+ if self.args.mode.optimizer.name == OptimizerKind.rmsprop:
+ self._opt = torch.optim.RMSprop(self._net.parameters(), 1.0, eps=1e-4)
+ else:
+ self._opt = torch.optim.Adam(self._net.parameters(), 1.0)
+to:
+ if self.args.mode.optimizer.name == OptimizerKind.rmsprop:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ self._opt = poptorch.optim.RMSprop(self._net.parameters(), 1.0, eps=1e-4)
+ else:
+ self._opt = torch.optim.RMSprop(self._net.parameters(), 1.0, eps=1e-4)
+ else:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ self._opt = poptorch.optim.Adam(self._net.parameters(), 1.0)
+ else:
+ self._opt = torch.optim.Adam(self._net.parameters(), 1.0)
+Putting the loss calculation in forward_pass() allows the loss computation to be performed on the IPUs. +This will be faster because the data will not need to be transfered round-trip to the CPU.
+Change forward_pass():
+ if net is None:
+ logits_image = self._net(minibatch_data['image'])
+ else:
+ logits_image = net(minibatch_data['image'])
+The following code changes are to account for the loss function, i.e., self.loss_calculator, and the +image labels, i.e., labels_image, to be passed to the model's forward_pass method. Additionally, the calculated +loss is returned from the forward_pass method.
+ if net is None:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ logits_image, labels_image, loss = self._net(minibatch_data['image'], self.loss_calculator, labels_image)
+ return logits_image, labels_image, loss
+ else:
+ logits_image = self._net(minibatch_data['image'])
+ else:
+ if self.args.run.compute_mode == ComputeMode.IPU and self.args.mode.name != ModeKind.inference:
+ logits_image, labels_image, loss = net(minibatch_data['image'], self.loss_calculator, labels_image)
+ return logits_image, labels_image, loss
+ else:
+ logits_image = net(minibatch_data['image'])
+Receive the extra loss variable from the forward_pass method.
+Update the train_step method.
+ with self.timing_context("forward"):
+ if self.args.run.precision == Precision.mixed and self.args.run.compute_mode == ComputeMode.GPU:
+ with torch.cuda.amp.autocast():
+ logits_image, labels_image = self.forward_pass(minibatch_data)
+ else:
+ logits_image, labels_image = self.forward_pass(minibatch_data)
+
+ verbose = False
+
+ # Compute the loss based on the logits
+ with self.timing_context("loss"):
+ loss = self.loss_calculator(labels_image, logits_image)
+The forward_pass() method was changed to return the extra variable loss in the previous section. It is now +received conditionally when using an IPU(s).
+In the with self.timing_context("loss"): section, only calculate loss if not using an IPU(s).
+ with self.timing_context("forward"):
+ if self.args.run.precision == Precision.mixed and self.args.run.compute_mode == ComputeMode.GPU:
+ with torch.cuda.amp.autocast():
+ logits_image, labels_image = self.forward_pass(minibatch_data)
+ else:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ logits_image, labels_image, loss = self.forward_pass(minibatch_data)
+ else:
+ logits_image, labels_image = self.forward_pass(minibatch_data)
+
+ verbose = False
+
+
+ # Compute the loss based on the logits
+ with self.timing_context("loss"):
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ loss = loss
+ else:
+ loss = self.loss_calculator(labels_image, logits_image)
+Update the val_step method.
+Find this code.
+ if self.args.run.precision == Precision.mixed and self.args.run.compute_mode == ComputeMode.GPU:
+ with torch.cuda.amp.autocast():
+ logits_image, labels_image = self.forward_pass(minibatch_data, net=val_net)
+ else:
+ logits_image, labels_image = self.forward_pass(minibatch_data, net=val_net)
+
+ # Compute the loss based on the logits
+ loss = self.loss_calculator(labels_image, logits_image)
+Change the code to the following.
+ if self.args.run.precision == Precision.mixed and self.args.run.compute_mode == ComputeMode.GPU:
+ with torch.cuda.amp.autocast():
+ logits_image, labels_image = self.forward_pass(minibatch_data, net=val_net)
+
+ # Compute the loss based on the logits
+ loss = self.loss_calculator(labels_image, logits_image)
+ else:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ logits_image, labels_image, loss = self.forward_pass(minibatch_data, net=val_net)
+ else:
+ logits_image, labels_image = self.forward_pass(minibatch_data, net=val_net)
+
+ # Compute the loss based on the logits
+ loss = self.loss_calculator(labels_image, logits_image)
+The Graphcore system is more computationally efficient if the loss function is on the +IPU. This is accomplished by using the loss function within the model's forward method.
+Edit src/networks/torch/uresnet2D.py.
+Find the forward method.
+ +Update the argument list to include the loss function, i.e., loss_calculator +and the image labels, i.e., labels_image.
+ +Add the loss calculation just before the forward method returns.
+ if loss_calculator is not None:
+
+ labels_image = labels_image.long()
+ labels_image = torch.chunk(labels_image, chunks=3, dim=1)
+ shape = labels_image[0].shape
+ labels_image = [ _label.view([shape[0], shape[-2], shape[-1]]) for _label in labels_image ]
+
+ loss = loss_calculator(labels_image, x)
+ import poptorch
+ loss = poptorch.identity_loss(loss , reduction="mean")
+ return x, labels_image, loss
+
+ # This return already exists.
+ return x
+The poptorch.identity_loss method takes a single PyTorch tensor and will backpropagate a gradient of ones through it. You may find an example at here
+The following is included for completeness. One will not likely find this in other code.
+Open bin/exec.py in your favorite editor. Change:
+ +to
+ + + + + + + + + + + + + + + + +The intent of this page is to show conceptually how to convert a Graphcore model to run on Distributed Data Parallel +using PopDist. +It is not necessary to convert CosmicTagger because it has already been converted and is +located at CosmicTagger on the GraphcoreDDP branch. +The original is located at CosmicTagger.
+The first step to converting a model is to verify that it runs on the CPU. This step has been verified for CosmicTagger.
+You may use the code at CosmicTagger on the Graphcore branch.
+Edit src/utils/torch/trainer.py.
+PopTorch is Graphcore's extension of PyTorch.
+PopDist is Graphcore's distributed processing package.
+Import poptorch and popdist at the top of the file.
+ +Initialize popdist for distributed computing.
+Establish a class variable name instance. This is used to differentiate between different +model instances that will be saved.
+Add the following line at the bottom of init().
+ if self.args.run.compute_mode == ComputeMode.IPU and popdist.isPopdistEnvSet():
+ popdist.init()
+ self._instance = popdist.getInstanceIndex()
+ else:
+ self._instance = 0
+Use the instance variable for the model file name.
+Find def get_model_filepath.
+Change:
+ +To:
+ +Add a helper function to log data at the bottom of the file.
+ def log_in_single_instance(self, string):
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ if not popdist.isPopdistEnvSet() or popdist.getInstanceIndex() == 0:
+ logging.info(string)
+ else:
+ logging.info(string)
+PopTorch has an Option() method which returns values that get passed to poptorch.trainingModel. +The returned values are stored in opts in this example.
+Find:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ if self.is_training():
+ opts = poptorch.Options()
+ self._net = poptorch.trainingModel(self._net, opts, optimizer=torch.optim.SGD(self._net.parameters(), lr=1e-3))
+ else:
+ self._net = poptorch.inferenceModel(self._net)
+Replace it with:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ if popdist.isPopdistEnvSet():
+ opts = popdist.poptorch.Options()
+ # When using the dataloader with 'auto_distributed_partitioning=True'
+ # and 'shuffle=True' we must set the random seed to ensure that tensors
+ # are in the same order in all processes.
+ opts.randomSeed(42)
+ # Replication factor is already set via PopRun so
+ # we ignore 'args.num_replicas'.
+ logging.info(f"Num of local replicas: {popdist.getNumLocalReplicas()}")
+ else:
+ opts = poptorch.Options()
+ opts.replicationFactor(self.args.num_replicas)
+
+ if self.is_training():
+ self._net = poptorch.trainingModel(self._net, opts, optimizer=torch.optim.SGD(self._net.parameters(), lr=1e-3))
+ else:
+ self._net = poptorch.inferenceModel(self._net)
+See instructions in README_GRAPHCORE.md.
+ + + + + + + + + + + + + + + +These steps only need to be executed once per user.
+Running on multiple nodes is a three step process.
+Create a Key
+ +Put Key into Authorized_keys File
+ +Add Node IP Addresses to Known_hosts File
+ +Follow all the instructions in Getting Started to log into a Graphcore node.
+Follow the instructions in Virtual Environments up to and including PopART Environment Setup.
+Following the instructions in Example Programs up to and including +MNIST, Install Requirements.
+Set the option to generate all reports, i.e., "autoReport.all":"true".
+Set the reports directory, i.e., "autoReport.directory":"./reports".
+Do so by running the following commands:
+ +Do so by running the following command:
+ +When MNIST has finished running, see Profiling to use Graph Analyser.
+ + + + + + + + + + + + + + + +Follow all the instructions in Getting Started to log into a Graphcore node.
+Follow the instructions in Virtual Environments up to and including PopART Environment Setup.
+Graphcore provides examples of some well-known AI applications in their repository at https://github.com/graphcore/examples.git.
+Clone the examples repository to your personal directory structure:
+ +Change directory
+ +Export the datasets directory.
+export POPLAR_ENGINE_OPTIONS='{"autoReport.all":"true", "autoReport.directory":"./reports"}'
+export DATASETS_DIR=/software/datasets
+HOST1=`ifconfig eno1 | grep "inet " | grep -o '[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}\.[0-9]\{1,3\}' | head -1`
+OCT123=`echo "$HOST1" | cut -d "." -f 1,2,3`
+OCT4=`echo "$HOST1" | cut -d "." -f 4`
+HOST2=$OCT123.`expr $OCT4 + 1`
+HOST3=$OCT123.`expr $OCT4 + 2`
+HOST4=$OCT123.`expr $OCT4 + 3`
+export HOSTS=$HOST1,$HOST2,$HOST3,$HOST4
+export CLUSTER=c16
+VIPU_SERVER=${VIPU_SERVER:=$HOST1}
+FIRST_PARTITION=`vipu-admin list partitions --api-host $VIPU_SERVER| grep ACTIVE | cut -d '|' -f 3 | cut -d ' ' -f 2 | head -1`
+PARTITON=${PARTITION:=$FIRST_PARTITION}
+Profile ResNet50.
+++Note: Use screen because every run is long.
+
cd train
+python3 -m examples_utils benchmark --spec benchmarks.yml --benchmark pytorch_resnet50_train_real_pod16
+When ResNet50 has finished running, see Profiling to use Graph Analyser.
+ + + + + + + + + + + + + + + +This is an adaptation of Capturing IPU Reports.
+See Capturing IPU Reports for more information.
+This section describes how to generate the files that the Graph Analyser can analyze. The Graph Analyser uses report files generated during compilation and execution by the Poplar SDK.
+Because of all these extra memory requirements, a model with high memory consumption may go out of memory when profiling is enabled. Depending on the model, you can adjust its parameters to leave space for the instrumentation. For example, you can try decreasing the batch size. In TensorFlow BERT you can adjust the micro batch-size.
+It is essential that you also try to reduce the iterations on each run. For instance, by reducing the number of steps or the number of batches per step you can get a lighter execution profile. This will not only reduce the host computation overhead but will also speed up visualization in the Graph Analyser.
+Download PopVision Tools.
+Click Download Now button.
+In the Graph Analyser section, select you operating system.
+Install per selected operating system.
+Use ssh from your development system.
+The ssh command will use a jumphost and port forwarding. The format is as follows:
+ssh -J ALCFUserID@gc-login-dd.ai.alcf.anl.gov ALCFUserID@gc-poplar-DD -L 8090:127.0.0.1:22
+ssh -J wilsonb@gc-login-01.ai.alcf.anl.gov wilsonb@gc-poplar-02.ai.alcf.anl.gov -L 8090:127.0.0.1:22
+Where:
+| Argument | +Help | +
|---|---|
| ALCFUserID | +Is your ALCF user identification. | +
| dd | +Is the Graphcore login node to use, i.e., 01 or 02 | +
| DD | +Is the Graphcore node to use, i.e., 01, 02, 03, or 04. | +
| 8090 | +Is the port on your local machine. | +
| 127.0.0.1:22 | +Is the local IP address and port on the remote machine. | +
| + | + |
You will receive a prompt.
+Continue on your development machine.
+
The Summary Report will be displayed.
+ + + + + + + + + + + + + + + +The Poplar SDK is downloaded onto the graphcore systems at the /software/graphcore/poplar_sdk/ location. The default poplar
+version (3.3.0) is enabled automatically upon logging into a graphcore node.
Check if Poplar is setup correctly:
+ +One should see:
+POPLAR version 3.3.0 (de1f8de2a7)
+clang version 16.0.0 (2fce0648f3c328b23a6cbc664fc0dd0630122212)
+If the Poplar SDK is not enabled, it can be enabled with +
+To disable the current Poplar SDK, e.g. if one wants to use a different Poplar SDK, follow the steps below. (Otherwise, skip to section Miscellaneous Environment Variables.) +This example assumes that the current installed SDK is 3.1.0 and you want to move to 3.3.0
+Set SDK env variable +
+Create a new virtual environment with this SDK and install popTorch and or other frameworks as needed. +
+mkdir ~/tmp
+export TF_POPLAR_FLAGS=--executable_cache_path=~/tmp
+export POPTORCH_CACHE_DIR=~/tmp
+
+export POPART_LOG_LEVEL=WARN
+export POPLAR_LOG_LEVEL=WARN
+export POPLIBS_LOG_LEVEL=WARN
+
+export PYTHONPATH=/software/graphcore/poplar_sdk/3.3.0/poplar-ubuntu_20_04-3.3.0+7857-b67b751185/python:$PYTHONPATH
+PopTorch is an extension of the Pytorch framework that is optimized for the IPU specific functionality. To activate the PopTorch environment, first create a virtual environment and activate it.
+mkdir -p ~/venvs/graphcore
+virtualenv ~/venvs/graphcore/poptorch33_env
+source ~/venvs/graphcore/poptorch33_env/bin/activate
+Use the following commands to install the PopTorch environment.
+POPLAR_SDK_ROOT=/software/graphcore/poplar_sdk/3.3.0
+export POPLAR_SDK_ROOT=$POPLAR_SDK_ROOT
+pip install $POPLAR_SDK_ROOT/poptorch-3.3.0+113432_960e9c294b_ubuntu_20_04-cp38-cp38-linux_x86_64.whl
+The Poplar SDK provides TensorFlow and Keras wheels built on 2.6 that includes the IPU specific functionality and optimized for the AMD processors. It can be installed as follows.
+Create virtual environment.
+virtualenv ~/venvs/graphcore/tensorflow2_33_env
+source ~/venvs/graphcore/tensorflow2_33_env/bin/activate
+Install the TensorFlow and Keras wheels.
+POPLAR_SDK_ROOT=/software/graphcore/poplar_sdk/3.3.0
+export POPLAR_SDK_ROOT=$POPLAR_SDK_ROOT
+pip install $POPLAR_SDK_ROOT/tensorflow-2.6.3+gc3.3.0+251580+08d96978c7f+amd_znver1-cp38-cp38-linux_x86_64.whl
+pip install $POPLAR_SDK_ROOT/keras-2.6.0+gc3.3.0+251582+a3785372-py2.py3-none-any.whl
+You should see:
+2023-08-22 21:53:26.109934: I tensorflow/compiler/plugin/poplar/driver/poplar_platform.cc:43] Poplar version: 3.3.0 (de1f8de2a7) Poplar package: b67b751185
+Install packages in the normal manner such as:
+ +For more details see Use pip for installing.
+To install a different version of a package that is already installed in one's environment, one can use:
+ +++ + + + + + + + + + + + + + + +Note: Conda is not supported on the Graphcore system.
+
If you do not already have an allocation, you will need to request one here: +Discretionary Allocation Request (New & Renewal)
+If you do not have an ALCF account (but have an allocation), request one here: ALCF Account and Project Management
+Connection to a GroqRack node is a two-step process.
+The first step is to ssh from a local machine to a login node. +The second, optional step is to ssh from a login node to a GroqRack node. Jobs may also be started and tracked from login nodes.
+
Connect to a groq login node, editing this command line to use your ALCF user id. You will be prompted for a password; use the 8-digit code provided by MobilePASS+. +
+This randomly selects one of the login nodes, namelygroq-login-01.ai.alcf.anl.gov or groq-login-02.ai.alcf.anl.gov. You can alternatively ssh to the specific login nodes directly.
+Once you are on a login node, optionally ssh to one of the GroqRack nodes, which are numbered 1-9.
+ + + + + + + + + + + + + + + + +This section covers how to remotely use the GroqView profiler and visualizer tool.
+Groq compiles produce an accurate and detailed model of the performance of a model's execution on groq cards. There is no need to run a model on groqcards to use GroqView.
+The GroqView example adds the "groqview=True" parameter to the groqit call, then calls the groqview() method on the model returned by groqit.
+This is the relevant code when using GroqFlow. It tries to retrieve the compiled model from the cache, compiles the model on a cache miss, then calls groqview().
+From groqflow/examples/pytorch/groqview.py:
+
# Build model
+gmodel = groqit(pytorch_model, inputs, groqview=True)
+# Open GroqView
+gmodel.groqview()
+On a groq node, run the groqview.py sample (or any script that includes similar code). Note the port number chosen by GroqView.
+
conda activate groqflow
+cd ~/groqflow/examples/pytorch
+python groqview.py
+# You will see something like the following.
+# The port number may be different.
+...
+Open your web browser:
+ http://localhost:8439
+On your laptop/user machine with a display, set up a 2-hop ssh tunnel.
+Set $GN_HOSTNAME to the name of the host where job is running
+
export GN_HOSTNAME=groq-r01-gn-09
+# Modify the port number if GroqView has chosen a different port.
+# This might happen if another user is also using GroqView.
+# Also, another user may be using the port on the login host.
+# `groq-login-01.ai.alcf.anl.gov` can be used as well.
+ssh -L 8439:localhost:8439 arnoldw@groq-login-02.ai.alcf.anl.gov -t ssh -L 8439:localhost:8439 -N $GN_HOSTNAME
+# When complete, "ctrl-c" or equivalent in the console where the ssh tunnel
+# was started will terminate both parts of a ssh tunnel set up this way.
+Point a Google Chrome-family web browser at this url, adjusting the port number if necessary. +(Chrome, Brave, Vivaldi, Opera tested.) +
+ + + + + + + + + + + + + + + +Groq jobs in the AI Testbed's groqrack are managed by the PBS job scheduler.
+Overview: PBS
+For additional information, see
+https://docs.alcf.anl.gov/running-jobs/job-and-queue-scheduling/
+Man pages are available. These are the key commands:
+
Jobs are launched from any GroqRack node, or from login nodes.
+If you expect a loss of an internet connection for any reason, for long-running jobs we suggest logging into a specific node and using either screen or tmux to create persistent command line sessions. For details use:
GroqFlow is the simplest way to port applications running inference to groq. The groqflow github repo includes many sample applications. +See GroqFlow.
+Clone the groqflow github repo and change current directory to the clone: +
+Create a groqflow conda environment, and activate it.
+Follow the instructions in the Virtual Environments
section.
+Note: Similar install instructions are in ~/groqflow/docs/install.md or GroqFlow™ Installation Guide
+The conda enviroment should be reinstalled whenever new groqflow code is pulled from the groqflow github; with a groqflow conda environment activated, redo just the pip install steps.
Each groqflow sample directory in the ~/groqflow/proof_points tree has a README.md describing the sample and how to run it.
See Job Queueing and Submission for more information about the PBS job scheduler.
+Create a script run_minilmv2.sh with the following contents. It assumes that conda was installed in the default location. The conda initialize section can also be copied from your .bashrc if the conda installer was allowed to add it.
+
#!/bin/bash
+# >>> conda initialize >>>
+# !! Contents within this block are managed by 'conda init' !!
+__conda_setup="$(${HOME}'/miniconda3/bin/conda' 'shell.bash' 'hook' 2> /dev/null)"
+if [ $? -eq 0 ]; then
+ eval "$__conda_setup"
+else
+ if [ -f "${HOME}/miniconda3/etc/profile.d/conda.sh" ]; then
+ . "${HOME}/miniconda3/etc/profile.d/conda.sh"
+ else
+ export PATH="${HOME}/miniconda3/bin:$PATH"
+ fi
+fi
+unset __conda_setup
+# <<< conda initialize <<<
+conda activate groqflow
+cd ~/groqflow/proof_points/natural_language_processing/minilm
+pip install -r requirements.txt
+python minilmv2.py
+Then run the script as a batch job with PBS. This will reserve a full eight-card(chip) node. +
+Note: the number of chips used by a model can be found in the compile cache dir for the model after it is compiled. E.g. +
+The groqflow proofpoints models use 1, 2 or 4 chips. +If your ~/.bashrc initializes conda, an alternative to copying the conda initilization script into your execution scripts is to comment out this section in your "~/.bashrc":
+
#!/bin/bash
+conda activate groqflow
+cd ~/groqflow/proof_points/natural_language_processing/minilm
+pip install -r requirements.txt
+python minilmv2.py
+$ qstat
+Job id Name User Time Use S Queue
+---------------- ---------------- ---------------- -------- - -----
+3084.groq-r01-co* run_minilmv2 user 0 R workq
+$
+Output will by default go to two files with names like the following, where the suffix is the job id. One standard output for the job. The other is the standard error for the job. +
$ ls -la run_minilmv2.sh.*
+-rw------- 1 user users 448 Oct 16 18:40 run_minilmv2.sh.e3082
+-rw------- 1 user users 50473 Oct 16 18:42 run_minilmv2.sh.o3082
+An alternative is to use an interactive PBS job. This may be useful when debugging new or changed code. Here is an example that starts a 24 hour interactive job. It reserves a full eight-card(chip) node. +
+Then activate your groqflow environment, and run python scripts with + + + + + + + + + + + + + + + + +ALCF's Groq system consists of a single GroqRackTM compute cluster that provides an extensible accelerator network consisting of 9 GroqNodeTM [ groq-r01-gn-01 through groq-r01-gn-09 ] nodes with a rotational multi-node network topology. Each of these GroqNodes consists of 8 GroqCardTM accelerators in them with integrated chip-to-chip connections with a dragonfly multi-chip topology.
GroqCardTM accelerator is a dual-width, full-height, three-quarter length PCI-Express Gen4 x16 adapter that includes a single GroqChipTM processor with 230 MB of on-chip memory. Based on the proprietary Tensor Streaming Processor (TSP) architecture, the GroqChip processor is a low latency and high throughput single core SIMD compute engine capable of 750 TOPS (INT8) and 188 TFLOPS (FP16) @ 900 MHz that includes advanced vector and matrix mathematical acceleration units. The GroqChip processor is deterministic, providing predictable and repeatable performance.
The GroqWare suite SDK uses a API based programming model and enables users to develop, compile, and run models on the GroqCard accelerator in a host server system. The SDK uses a ONNX/MLIR enabled DAG compiler and it consists of Groq Compiler, Groq API, and utility tools like GroqView™ profiler and groq-runtime.
+For more information refer to the following links:
+GroqRack spec sheet
+GroqNode spec sheet
+GroqCard spec sheet
+GroqChip spec sheet
+(via)
If conda is not already installed: +
rm Miniconda3-latest-Linux-x86_64.sh*
+wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh
+bash Miniconda3-latest-Linux-x86_64.sh
+# answer y/yes to all prompts
+# exit ssh session, then start a new ssh session
+exit
+Create a groqflow conda environment and activate it +
export PYTHON_VERSION=3.10.12
+conda create -n groqflow python=$PYTHON_VERSION -y
+conda activate groqflow
+Execute the following commands to install groqflow into the activated groqflow conda environment
+# Alter this if you have cloned groqflow to some other location.
+cd ~/groqflow
+if [ -d "groqflow.egg-info" ]; then rm -r groqflow.egg-info; fi
+pip install --upgrade pip
+pip list --format=freeze > frozen.txt
+pip install -r frozen.txt -e .
+pushd .
+cd demo_helpers
+if [ -d "groqflow_demo_helpers.egg-info" ]; then rm -r groqflow_demo_helpers.egg-info; fi
+pip install -e .
+popd
+pip install soundfile
+pip install datasets==2.21.0
+Note: if you encounter problems trying to update an existing groqflow conda environment, consider removing the existing environment with the following command, and recreating it. Make sure you deactivate the environment before removing it. +
+To use groqfloq, +
+Note: Always use a personal conda environment when installing packages on groq nodes; otherwise they can get installed into~/.local and can cause problems when your shared home directory is used on other systems. If you encounter mysterious package dependency/version issues, check your ~/.local/lib and ~/.local/bin for mistakenly installed packages.
+Note: The conda enviroment should be reinstalled whenever new groqflow code is pulled from the groqflow github; with a groqflow conda environment activated, redo just the pip install steps, including the removal of the egg-info directories.
+ + + + + + + + + + + + + + + +Using /data/ANL/results/sn30-r1-h1/wilsonb/032223.18/GPT1.5B.out for output +Using /data/ANL/results/sn30-r2-h1/wilsonb/032223.19/GPT1.5B.out for output
+Using /data/ANL/results/sn30-r2-h1/wilsonb/032223.19/BertLarge.out for output
+ + + + + + + + + + + + + + + +The SambaNova documentation is now available online SambaNova Documentation.
+The documentation for the SambaTune (a profiling and performance tuning tool for SambaNova systems) is now available at SambaTune Documentation.
+ + + + + + + + + + + + + + + +The SambaNova Model Zoo is SambaNova's new github repository for delivering RDU-compatible source code, including example applications for compiling and running models on SambaNova hardware.
+In the ALCF SN30 cluster, the Model Zoo samples run inside of Singularity containers. The Singularity image includes support for compiling and running models.
+The procedures in this section are drawn from Walkthrough—Inference and Fine-tuning with Llama2 7B for Chat.
+The Model Zoo inference sample used as an example in this section is described in more detail here About the Generation Example Apps. This readme (on GitHub) also describes the changes made to a CPU mode sample to run on an RDU.
+The original python scripts and scripts converted to run on an RDU are also supplied in the modelzoo.
+cpu_generate_text.py
+rdu_generate_text.py
+and
+cpu_train_llm.py
+rdu_train_llm.py
+rdu_train_llm_dp.py
Clone the repo in your usual location. +
+Note: your home directory is mounted by default in the singularity containers. +Change directory to your Model Zoo clone, and set an environment variable to be host SambaNova runts version, then start the container. This example binds a directory containing an OpenWebText dataset. +
cd ~/sambanova/modelzoo
+export TARGET_SAMBAFLOW_VERSION=$((rpm -q sambanova-runtime 2>/dev/null || dpkg -s sambanova-runtime 2>/dev/null) | egrep -m 1 -o "[0-9]+\.[0-9]+\.[0-9]+")
+echo $TARGET_SAMBAFLOW_VERSION
+# should be of the form 1.19.1
+./start_container.sh -b /data/ANL/openwebtext/hdf5/hdf5:/opt/datasets/openweb_hdf54096/ -b /software:/software / /software/sambanova/singularity/images/llm-modelzoo/Modelzoo/ModelzooDevbox_1.sif
+APP_ROOT: /home/arnoldw/sambanova/modelzoo
+Using singularity with image /software/sambanova/singularity/images/llm-modelzoo/Modelzoo/ModelzooDevbox_1.sif
+
+Running singularity instance with name: devbox_arnoldw_1724873417
+Singularity start command: singularity instance start --writable-tmpfs --bind /home/arnoldw/github.com/sambanova/modelzoo:/opt/modelzoo --bind /tmp:/tmp --bind /data/ANL/openwebtext/hdf5/hdf5:/opt/datasets/openweb_hdf54096/ --bind /software/models/:/opt/ckpts/ --bind /dev/hugepages:/dev/hugepages --bind /opt/sambaflow/pef/:/opt/sambaflow/pef/ --bind /opt/sambaflow/runtime/:/opt/sambaflow/runtime/ --bind /var/lib/sambaflow/ccl/ccl_config.db:/var/lib/sambaflow/ccl/ccl_config.db --bind /var/snml.sock:/var/snml.sock --bind /opt/sambanova/lib/python3.8/site-packages/pysnml:/opt/sambanova/lib/python3.8/site-packages/pysnml --bind /opt/sambanova/lib/python3.8/site-packages/pysnrdureset:/opt/sambanova/lib/python3.8/site-packages/pysnrdureset --bind /opt/sambanova/lib/python3.8/site-packages/pysnrdutools:/opt/sambanova/lib/python3.8/site-packages/pysnrdutools --bind /opt/sambanova/lib/python3.8/site-packages/sambaruntime:/opt/sambanova/lib/python3.8/site-packages/sambaruntime /software/sambanova/singularity/images/llm-modelzoo/Modelzoo/ModelzooDevbox_1.sif devbox_arnoldw_1724873417
+INFO: instance started successfully
+Singularity instance devbox_arnoldw_1724873417 started
+Run command: singularity exec instance://devbox_arnoldw_1724873417 /bin/bash
+Singularity>
+To list all running containers (while outside a container, e.g. a different SSH session): +
$ singularity instance list
+INSTANCE NAME PID IP IMAGE
+devbox_arnoldw_1724873417 1649294 /software/sambanova/singularity/images/llm-modelzoo/Modelzoo/ModelzooDevbox_1.sif
+To stop all your running containers (while outside a container): +
+cd ~/sambanova/modelzoo/
+pip install -r requirements/requirements.txt
+pip install --upgrade pip
+pip install -e .
+This model is also avaiable in /software/models/Llama-2-7b-hf/
+First, create a Hugging Face account at https://huggingface.co/join if you do not already have one.
+Go to meta-llama/Llama-2-7b-hf and accept the terms of use for Llama2 7B.
+You will need to wait (minutes at least) until the request is proccessed.
+In your Hugging Face account settings, generate a user access token. A read-only token works. Record the token such that it can easily be copy-pasted in the future.
+
# if working in an environment (e.g. laptop) where git-lfs is not installed,
+# sudo apt install git-lfs
+git lfs install # Only needs to be done once
+cd ~/sambanova
+git clone https://huggingface.co/meta-llama/Llama-2-7b-hf
+# Enter your HF user name and user access token (copy;paste) when prompted.
+Compile a LLaMA-7b text generation sample (using the Hugging Face model). This will take 20 minutes
+cd ~/sambanova
+# or ./Llama-2-7b-hf if downloaded
+python ./modelzoo/examples/nlp/text_generation/rdu_generate_text.py \
+command=compile \
+checkpoint.model_name_or_path=/software/models/Llama-2-7b-hf/ \
+samba_compile.output_folder=/home/$(whoami)/sambanova/out_generation \
++samba_compile.target_sambaflow_version=$TARGET_SAMBAFLOW_VERSION # =1.19.1
+Note: each compile will add a new subdirectory to the ouput folder (/home/$(whoami)/sambanova/out_generation), containing compile artifacts. The folder can be deleted when testing is complete;
Run the sample, using the .pef binary created by the compile.
+Note: The expression in the command line finds the most recent pef file.
cd ~/sambanova
+export PEF=$(find /home/$(whoami)/sambanova/out_generation -type f -name "*.pef" -printf "%T@ %p\n" | sort -n | tail -n1 | awk '{print $2}')
+# or ./Llama-2-7b-hf if downloaded
+python ./modelzoo/examples/nlp/text_generation/rdu_generate_text.py \
+ command=run \
+ checkpoint.model_name_or_path=/software/models/Llama-2-7b-hf/ \
+ samba_run.pef=${PEF}
+The end of the console output should resemble the following: +
Generating 32 tokens ...
+Decoding ...
+Completion:
+[', there was a little boy who lived in a small town.\nHe was a good boy, but sometimes he had a hard time following the rules.\n']
+
+latencies
+ time to first token 1.1981s
+ tokens, excluding first token 0.3330s
+ tokens, overall 0.3600s
+ Total Latency 1.5310s
+throughputs
+ tokens/second excluding first token 3.0032
+ tokens/second overall 2.7777
+Singularity>
+Fine-tune the Llama2 7B model using a chat dataset.
+NOTE: These data preparation steps should be performed on a SambaNova node, and not in a singularity container.
+cd ~/sambanova
+git clone https://github.com/sambanova/generative_data_prep.git
+cd generative_data_prep
+python -m venv gdp_venv
+source gdp_venv/bin/activate
+pip install .
+cd ~/sambanova
+Make sure that you have git lfs installed, with git lfs install
+
.jsonl formatcd ~/sambanova
+source generative_data_prep/gdp_venv/bin/activate
+# This step makes a single jsonl file
+python ./modelzoo/examples/nlp/training/utils/convert_ultrachat.py -src ultrachat/ -dest ultrachat_processed.jsonl
+# get a small subset to keep the 1 epoch runtime down.
+mv ~/sambanova/ultrachat_processed.jsonl ~/sambanova/ultrachat_processed_full.jsonl
+head -1000 ~/sambanova/ultrachat_processed_full.jsonl > ~/sambanova/ultrachat_processed.jsonl
+# This step makes a directory of hdf5 files from the single jsonl file
+export TOKENIZER="./Llama-2-7b-hf"
+export MAX_SEQ_LENGTH=4096
+python -m generative_data_prep pipeline --input_file_path=./ultrachat_processed.jsonl --output_path=./ultrachat_dialogue --pretrained_tokenizer=${TOKENIZER} --max_seq_length=${MAX_SEQ_LEN}
+deactivate
+If you are not already in a Singularity container (with the pre-reqs installed),
+start a new Model Zoo Singularity container with
+
cd ~/sambanova/modelzoo
+export TARGET_SAMBAFLOW_VERSION=$((rpm -q sambanova-runtime 2>/dev/null || dpkg -s sambanova-runtime 2>/dev/null) | egrep -m 1 -o "[0-9]+\.[0-9]+\.[0-9]+")
+echo $TARGET_SAMBAFLOW_VERSION
+# should be of the form 1.19.1
+./start_container.sh -b /data/ANL/openwebtext/hdf5/hdf5:/opt/datasets/openweb_hdf54096/ -b /software:/software /software/sambanova/singularity/images/llm-modelzoo/Modelzoo/ModelzooDevbox_1.sif
+Then install the pre-reqs into the container with +
cd ~/sambanova/modelzoo/
+pip install -r requirements/requirements.txt
+pip install --upgrade pip
+pip install -e .
+cd ~/sambanova
+export CHECKPOINT=/software/models/Llama-2-7b-hf/ # or ./Llama-2-7b-hf
+export MAX_SEQ_LENGTH=4096
+export BATCH_SIZE=8
+export ARCH=sn30
+python modelzoo/examples/nlp/training/rdu_train_llm.py \
+ command=compile \
+ checkpoint.config_name=${CHECKPOINT} \
+ model.max_seq_length=${MAX_SEQ_LENGTH} \
+ training.batch_size=${BATCH_SIZE} \
+ samba_compile.arch=${ARCH} \
+ samba_compile.output_folder=/home/$(whoami)/sambanova/out_train \
+ +samba_compile.target_sambaflow_version=$TARGET_SAMBAFLOW_VERSION
+Note: each compile will add a new subdirectory to the ouput folder (/home/$(whoami)/sambanova/out_train), containing compile artifacts. The folder can be deleted when testing is complete;
This will run for 1 full epoch and takes 1 hour to execute, using a single RDU.
+It uses the config file modelzoo/examples/nlp/training/config/base_config_rdu.yaml
cd ~/sambanova
+export CHECKPOINT=/software/models/Llama-2-7b-hf/ # or ./Llama-2-7b-hf
+export MAX_SEQ_LENGTH=4096
+export DATASET=./ultrachat_dialogue; # or container path to dataset
+# Finds most recent pef file in tree
+export PEF=$(find /home/$(whoami)/sambanova/out_train -type f -name "*.pef" -printf "%T@ %p\n" | sort -n | tail -n1 | awk '{print $2}')
+python -u modelzoo/examples/nlp/training/rdu_train_llm.py \
+ command=run \
+ checkpoint.model_name_or_path=${CHECKPOINT} \
+ model.max_seq_length=${MAX_SEQ_LENGTH} \
+ samba_run.pef=${PEF} \
+ training.dataset=${DATASET}
+The end of the console output should resemble the following if run for a full epoch: +
Targeting samba-runtime v4.2.5. Samba is running with --target-runtime-version=1.3.10 on a system with installed runtime None.
+
+Log ID initialized to: [arnoldw][python][1003] at /var/log/sambaflow/runtime/sn.log
+Loading dataset for epoch 1...
+
+Number of epochs: 1
+Batch size: 8
+Number of batches (steps): 1,143
+
+Starting training for epoch 1...
+Epoch [1/1], Step [1/1143], Loss: 0.8184
+Epoch [1/1], Step [2/1143], Loss: 0.2452
+Epoch [1/1], Step [3/1143], Loss: 0.3727
+Epoch [1/1], Step [4/1143], Loss: 0.2945
+...
+Epoch [1/1], Step [1134/1143], Loss: 0.2529
+Epoch [1/1], Step [1135/1143], Loss: 0.2713
+Epoch [1/1], Step [1136/1143], Loss: 0.2669
+Epoch [1/1], Step [1137/1143], Loss: 0.2144
+Epoch [1/1], Step [1138/1143], Loss: 0.2129
+Epoch [1/1], Step [1139/1143], Loss: 0.2229
+Epoch [1/1], Step [1140/1143], Loss: 0.2263
+Epoch [1/1], Step [1141/1143], Loss: 0.2434
+Epoch [1/1], Step [1142/1143], Loss: 0.2131
+Epoch [1/1], Step [1143/1143], Loss: 0.1626
+Finished training.
+Saving checkpoint...
+Checkpoint saved at finetuned_model/
+Saving summary...
+Summary saved at finetuned_model/summary.txt
+Singularity>
+In this section we will learn how to extend the UNet2d and Gpt1.5B applications scripts that we introduced in the Example Programs to compile and run multiple instances of the model in a data parallel fashion across multiple tiles or across multiple nodes.
+Create the following directory and change to it if you have not already done so.
+ +Create the file Unet2d.sh and unet_batch.sh in the current directory using your favorite editor. +Copy and paste the contents of Unet2d.sh and unet_batch.sh +to files with the same name into the current directory using your favorite editor.
+ +Run these commands for training (compile + train): +The compile and run scripts have the following input arguments.
+image size: The images are square. Valid sizes include 256, 512, and 1024.
+Batch size: local batch size. The global batch size is local batch size * Num of instances.
+num of instances: Total number of instances of Unet2d run in data parallel framework.
+RunID: A unique Id for the compile or run process.
+The script uses the arguments pcompile and prun for the data parallel compile and run.
./Unet2d.sh pcompile <image size> <batch_size> <num of instances> <RunID>
+./Unet2d.sh prun <image size> <batch_size> <num of instances> <RunID>
+For a image size of 256x256 and local batch size of 256 when running 8 instance, the commands are provided as follows.
+./Unet2d.sh pcompile 256 256 8 unet2d_8inst_pcompile
+./Unet2d.sh prun 256 256 8 unet2d_8inst_prun
+The above commands displays the file that contains the output for the execution of the above scripts, usually /data/ANL/results/<hostname>/<userId>/<RunID>/Unet2d.out
You can inspect the compile command that contains --data-parallel -ws 2 arguments to ensure that the pef file is compatible for data parallel runs. The pef generated from the compilation process for the above compile command is placed under out/Unet2d/unet_train_256_256_NP_4 inside the current working directory.
python /opt/sambaflow/apps/image/segmentation/compile.py compile --mac-v2 --in-channels=3 --in-width=${2} --in-height=${2} --batch-size=${BS} --enable-conv-tiling --num-tiles=4 --pef-name=unet_train_${BS}_${2}_NP_${NUM_TILES} --data-parallel -ws 2 --output-folder=${OUTDIR}
+Once the model is compiled, sbatch is used to launch the multiple instances. The below example shows that a total of 8 tasks or instances are launched over the host on which the script is launched.
+sbatch --gres=rdu:1 --tasks-per-node ${NP} --nodes 1 --nodelist $(hostname) --cpus-per-task=${cpus} $(pwd)/unet_batch.sh ${NP} ${NUM_WORKERS} ${BS} ${2} ${5}
+The run command has --data-parallel --reduce-on-rdu arguments that is compatible with data parallel run.
srun --mpi=pmi2 python /opt/sambaflow/apps/image/segmentation//hook.py run --data-cache=${CACHE_DIR} --data-in-memory --num-workers=${NUM_WORKERS} --enable-tiling --min-throughput 395 --in-channels=3 --in-width=${IM} --in-height=${IM} --init-features 32 --batch-size=${BS} --epochs 10 --data-dir ${DS} --log-dir log_dir_unet_${IM}_${BS}_${NP} --data-parallel --reduce-on-rdu --pef=${OUTDIR}/unet_train_${BS}_${IM}_NP_4/unet_train_${BS}_${IM}_NP_4.pef
+The throughput is calculated by averaging the e2e samples_per_sec over the different instances.
inner train loop time : 36.314290046691895 for 10 epochs, number of global steps: 10, e2e samples_per_sec: 563.9653143065
+inner train loop time : 33.36756229400635 for 10 epochs, number of global steps: 10, e2e samples_per_sec: 613.7697389922524
+inner train loop time : 33.94625234603882 for 10 epochs, number of global steps: 10, e2e samples_per_sec: 603.3066563941279
+inner train loop time : 32.309499979019165 for 10 epochs, number of global steps: 10, e2e samples_per_sec: 633.8692958200872
+inner train loop time : 31.418426036834717 for 10 epochs, number of global steps: 10, e2e samples_per_sec: 651.8467849404489
+inner train loop time : 28.164129495620728 for 10 epochs, number of global steps: 10, e2e samples_per_sec: 727.1660927132315
+inner train loop time : 30.29698896408081 for 10 epochs, number of global steps: 10, e2e samples_per_sec: 675.9747651583616
+inner train loop time : 25.332663536071777 for 10 epochs, number of global steps: 10, e2e samples_per_sec: 808.442427336472
+Create the files Gpt1.5B_compile.sh and Gpt1.5B_run.sh in the current directory.
+Copy the contents of Gpt1.5B_compile.sh and Gpt1.5B_run.sh. Alternatively, the files can be accessed at /data/ANL/scripts/Gpt1.5B_compile.sh and /data/ANL/scripts/Gpt1.5B_run.sh on any of the compute node and can be copied over to the working directory.
This script consists of commands to compile and run multiple instances of Gpt1.5B model across multiple nodes. Run the Gpt1.5B_compile.sh to first compile and generate the pef file for the model and it in turn launches the Gpt1.5B_run.sh script to run multiple instances of the model over the different nodes.
You can see the log file path displayed on the screen as seen in the example below. You can use the tail command to check the progress of the run.
vsastry@sn30-r1-h1:~/nlp-multiNodetest$ ./Gpt1.5B_compile.sh
+Using /data/ANL/results/sn30-r1-h1/vsastry/041823.19/GPT1.5B.out for output
+The artifacts of the compile process is produced in the path : /data/scratch/<userId>.
Inspect the compile command in the script to see that it includes additional arguments --data-parallel and -ws 2 to generate a pef that is compatible for data parallel runs.
python /opt/sambaflow/apps/nlp/transformers_on_rdu/transformers_hook.py compile --module_name gpt2_pretrain --task_name clm --max_seq_length 1024 -b 16 --output_dir=${OUTDIR}/hf_output --overwrite_output_dir --do_train --per_device_train_batch_size 16 --cache ${OUTDIR}/cache/ --tokenizer_name gpt2 --model_name gpt2 --mac-v2 --non_split_head --mac-human-decision /opt/sambaflow/apps/nlp/transformers_on_rdu/human_decisions_gm/mac_v2_overrides/gpt2_48_enc_full_recompute_training_spatialmapping_tiling16_clmerge_gm_nonpardp_lnsd.json --compiler-configs-file /opt/sambaflow/apps/nlp/transformers_on_rdu/human_decisions_gm/compiler_configs/compiler_configs_gpt2_sc_recompute_spatialmapping_tiling16_clsmerge_withcls_nonpardp_norc_e2e.json --skip_broadcast_patch --config_name /opt/sambaflow/apps/nlp/transformers_on_rdu/customer_specific/mv/configs/gpt2_config_xl_50260.json --no_index_select_patch --data-parallel -ws 2 --weight_decay 0.1 --max_grad_norm_clip 1.0 --num-tiles 4 --pef-name=gpt15 --output-folder=${OUTDIR}
+Once the model is compiled, sbatch is used to launch the multiple instances across the nodes. The below example shows that a total of 32 tasks or instances are launched over 2 nodes with each node having a maximum of 16 tasks. Slurm allocates any 2 of the available nodes in this example.
/usr/local/bin/sbatch --output=${HOME}/slurm-%A.out --ntasks 32 --gres=rdu:1 --ntasks-per-node 16 --nodes 2 --cpus-per-task=8 Gpt1.5B_run.sh ${1} >> ${OUTPUT_PATH} 2>&1
+The run command for each of this instance is present in the Gpt1.5B_run.sh script. You can inspect the command in the script to see that --data-parallel --reduce-on-rdu arguments are present to ensure that the model is run in a data parallel fashion and that the gradient accumulation takes place on the RDU.
/usr/local/bin/srun --mpi=pmi2 python /opt/sambaflow/apps/nlp/transformers_on_rdu/transformers_hook.py run -b 16 --module_name gpt2_pretrain --task_name clm --max_seq_length 1024 --overwrite_output_dir --do_train --per_device_train_batch_size 16 --cache ${OUTDIR}/cache/ --tokenizer_name gpt2 --model_name gpt2 --non_split_head --skip_broadcast_patch --no_index_select_patch --output_dir=${OUTDIR}/hf_output --config_name /opt/sambaflow/apps/nlp/transformers_on_rdu/customer_specific/mv/configs/gpt2_config_xl_50260.json --max_grad_norm_clip 1.0 --skip_checkpoint --data-parallel --reduce-on-rdu --data_dir /data/ANL/ss1024 --data_dir /data/ANL/ss1024 --logging_steps 1 --max_steps 900000 --learning_rate 0.00025 --steps_this_run 800 --min_throughput 299000 --max_throughput 600000 --pef=${OUTDIR}/gpt15/gpt15.pef >> ${OUTPUT_PATH} 2>&1
+squeue shows that the model is run on 2 nodes sn30-r1-h1 and sn30-r2-h2.
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
+10191 sambanova Gpt1.5B_run.sh vsastry R 23:18 2 sn30-r1-h1,sn30-r2-h2
+sntilestat can also be used to check the total numbers of tiles used for the runs.
TILE %idle %exec %pload %aload %chkpt %quiesce PID USER COMMAND
+/XRDU_0/RDU_0/TILE_0 8.0 91.6 0.3 0.1 0.0 0.0 2750333 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_0/TILE_1 8.0 91.6 0.3 0.1 0.0 0.0 2750333 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_0/TILE_2 7.9 91.6 0.3 0.3 0.0 0.0 2750333 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_0/TILE_3 7.7 91.8 0.3 0.3 0.0 0.0 2750333 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_0/TILE_4 7.6 91.9 0.4 0.1 0.0 0.0 2750339 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_0/TILE_5 7.5 91.9 0.5 0.1 0.0 0.0 2750339 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_0/TILE_6 7.5 91.8 0.5 0.3 0.0 0.0 2750339 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_0/TILE_7 7.3 92.0 0.6 0.0 0.0 0.0 2750339 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_1/TILE_0 8.9 89.9 1.0 0.1 0.0 0.0 2750338 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_1/TILE_1 9.0 89.9 0.9 0.1 0.0 0.0 2750338 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_1/TILE_2 8.6 89.8 1.4 0.1 0.0 0.0 2750338 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_1/TILE_3 8.5 89.9 1.4 0.1 0.0 0.0 2750338 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_1/TILE_4 7.9 90.9 0.9 0.4 0.0 0.0 2750343 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_1/TILE_5 7.7 90.9 0.9 0.5 0.0 0.0 2750343 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_1/TILE_6 7.7 91.0 0.9 0.4 0.0 0.0 2750343 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_0/RDU_1/TILE_7 8.0 91.0 0.6 0.4 0.0 0.0 2750343 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_0/TILE_0 7.6 92.0 0.3 0.1 0.0 0.0 2750345 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_0/TILE_1 7.6 92.0 0.3 0.1 0.0 0.0 2750345 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_0/TILE_2 7.5 92.1 0.3 0.1 0.0 0.0 2750345 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_0/TILE_3 7.5 92.1 0.3 0.1 0.0 0.0 2750345 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_0/TILE_4 7.5 92.1 0.3 0.1 0.0 0.0 2750335 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_0/TILE_5 7.5 92.1 0.3 0.1 0.0 0.0 2750335 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_0/TILE_6 7.5 92.1 0.3 0.1 0.0 0.0 2750335 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_0/TILE_7 7.5 92.1 0.3 0.1 0.0 0.0 2750335 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_1/TILE_0 7.7 91.5 0.4 0.4 0.0 0.0 2750330 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_1/TILE_1 7.9 91.5 0.3 0.4 0.0 0.0 2750330 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_1/TILE_2 7.9 91.5 0.3 0.4 0.0 0.0 2750330 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_1/TILE_3 7.6 91.8 0.4 0.3 0.0 0.0 2750330 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_1/TILE_4 7.7 91.9 0.4 0.0 0.0 0.0 2750334 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_1/TILE_5 7.7 91.9 0.4 0.0 0.0 0.0 2750334 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_1/TILE_6 7.9 91.9 0.3 0.0 0.0 0.0 2750334 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_1/RDU_1/TILE_7 7.9 91.9 0.3 0.0 0.0 0.0 2750334 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_0/TILE_0 8.0 91.8 0.1 0.1 0.0 0.0 2750346 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_0/TILE_1 8.0 91.8 0.1 0.1 0.0 0.0 2750346 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_0/TILE_2 8.0 91.8 0.1 0.1 0.0 0.0 2750346 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_0/TILE_3 7.7 91.9 0.1 0.3 0.0 0.0 2750346 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_0/TILE_4 7.5 92.0 0.5 0.0 0.0 0.0 2750336 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_0/TILE_5 7.6 91.9 0.5 0.0 0.0 0.0 2750336 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_0/TILE_6 7.6 91.9 0.4 0.1 0.0 0.0 2750336 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_0/TILE_7 7.5 91.9 0.4 0.3 0.0 0.0 2750336 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_1/TILE_0 7.5 91.8 0.6 0.1 0.0 0.0 2750331 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_1/TILE_1 7.5 91.8 0.6 0.1 0.0 0.0 2750331 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_1/TILE_2 7.7 91.6 0.5 0.1 0.0 0.0 2750331 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_1/TILE_3 7.7 91.6 0.5 0.1 0.0 0.0 2750331 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_1/TILE_4 7.9 91.4 0.8 0.0 0.0 0.0 2750329 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_1/TILE_5 7.9 91.4 0.8 0.0 0.0 0.0 2750329 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_1/TILE_6 8.1 91.4 0.5 0.0 0.0 0.0 2750329 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_2/RDU_1/TILE_7 8.2 91.4 0.4 0.0 0.0 0.0 2750329 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_0/TILE_0 7.5 91.8 0.4 0.4 0.0 0.0 2750344 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_0/TILE_1 7.5 91.8 0.4 0.4 0.0 0.0 2750344 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_0/TILE_2 7.5 91.8 0.4 0.4 0.0 0.0 2750344 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_0/TILE_3 7.5 91.8 0.4 0.4 0.0 0.0 2750344 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_0/TILE_4 7.6 91.8 0.3 0.4 0.0 0.0 2750337 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_0/TILE_5 7.7 91.8 0.1 0.4 0.0 0.0 2750337 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_0/TILE_6 7.7 91.8 0.3 0.3 0.0 0.0 2750337 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_0/TILE_7 7.7 91.9 0.3 0.1 0.0 0.0 2750337 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_1/TILE_0 7.7 92.0 0.1 0.1 0.0 0.0 2750347 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_1/TILE_1 7.7 92.0 0.1 0.1 0.0 0.0 2750347 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_1/TILE_2 7.7 92.1 0.1 0.0 0.0 0.0 2750347 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_1/TILE_3 7.7 92.1 0.1 0.0 0.0 0.0 2750347 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_1/TILE_4 7.3 91.9 0.5 0.3 0.0 0.0 2750332 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_1/TILE_5 7.3 91.9 0.5 0.3 0.0 0.0 2750332 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_1/TILE_6 7.3 91.9 0.5 0.3 0.0 0.0 2750332 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+/XRDU_3/RDU_1/TILE_7 7.3 92.0 0.5 0.1 0.0 0.0 2750332 vsastry /opt/sambaflow/apps/nlp/transformers_on_rdu/venv/b
+The Slurm log associated with the JOBID (10191 in the above example) is located in the home directory. You can use the tail command to check the progress of the training.
vsastry@sn30-r1-h1:~$ tail -f ~/slurm-10191.out
+Using /data/ANL/results/sn30-r1-h1/vsastry/041823.03/Gpt1.5B.out for output
+Once the run is completed, check the log file for the performance results.
+ + + + + + + + + + + + + + + + +You can use the link to the tutorials on the SambaNova GitHub site or the examples on the compute node (as explained below).
+/opt/sambaflow/apps/starters, on all SambaNova compute nodes, as discussed on this page. Make a copy of this to your home directory:
+ +Deactivate any active conda environment. If you have conda installed and a conda environment is active, you will see something like (base) at the beginning of the command prompt. If so, you will need to deactivate it with conda deactivate. Conda is not used on the SambaNova SN30 cluster.
Change directory
+ +Below are some of the common arguments used across most of the models in the example code.
+| Argument | +Default | +Help | +
|---|---|---|
| -b | +1 | +Batch size for training | +
| + | + | + |
| -n, | +100 | +Number of iterations to run | +
| --num-iterations | ++ | the pef for | +
| + | + | + |
| -e, | +1 | +Number epochs for training | +
| --num-epochs | ++ | + |
| + | + | + |
| --log-path | +'check | +Log path | +
| + | points' | ++ |
| + | + | + |
| --num-workers | +0 | +Number of workers | +
| + | + | + |
| --measure-train- | +None | +Measure training performance | +
| performance | ++ | + |
| + | + | + |
| Argument | +Default | +Help | +
|---|---|---|
| --lr | +0.01 | +Learning rate for training | +
| + | + | + |
| --momentum | +0.0 | +Momentum value for training | +
| + | + | + |
| --weight-decay | +0.01 | +Weight decay for training | +
| + | + | + |
| --data-path | +'./data' | +Data path | +
| + | + | + |
| --data-folder | +'mnist_ | +Folder containing mnist data | +
| + | data' | ++ |
| + | + | + |
++Note: If you receive an \"HTTP error\" message on any of the +following commands, run the command again. Such errors (e.g 503) are +commonly an intermittent failure to download a dataset.
+
Run these commands to compile and train the LeNet model:
+srun python lenet.py compile -b=1 --pef-name="lenet" --output-folder="pef"
+srun python lenet.py run --pef="pef/lenet/lenet.pef"
+Alternatively to use Slurm sbatch, create submit-lenet-job.sh with the following +contents:
+#!/bin/sh
+
+python lenet.py compile -b=1 --pef-name="lenet" --output-folder="pef"
+python lenet.py run --pef="pef/lenet/lenet.pef"
+Then
+ +Squeue will give you the queue status.
+ +One may see the run log using:
+ +Change directory
+ +Commands to run MNIST example:
+srun python ffn_mnist.py compile -b 1 --pef-name="ffn_mnist" --mac-v2
+srun python ffn_mnist.py run -b 1 -p out/ffn_mnist/ffn_mnist.pef
+To run the same using Slurm sbatch, create and run the submit-ffn_mnist-job.sh with the following contents.
+#!/bin/sh
+python ffn_mnist.py compile -b 1 --pef-name="ffn_mnist" --mac-v2
+python ffn_mnist.py run -b 1 -p out/ffn_mnist/ffn_mnist.pef
+Change directory
+ +This is not an exhaustive list of arguments.
+Arguments
+| Argument | +Default | +Help | +Step | +
|---|---|---|---|
| --lr | +0.001 | +Learning rate for training | +Compile | +
| + | + | + | + |
| --momentum | +0.0 | +Momentum value for training | +Compile | +
| + | + | + | + |
| --weight-decay | +1e-4 | +Weight decay for training | +Compile | +
| + | + | + | + |
| --num-features | +784 | +Number features for training | +Compile | +
| + | + | + | + |
| --num-classes | +10 | +Number classes for training | +Compile | +
| + | + | + | + |
| --weight-norm | +na | +Enable weight normalization | +Compile | +
| + | + | + | + |
Run these commands:
+srun python logreg.py compile --pef-name="logreg" --output-folder="pef"
+srun python logreg.py run --pef="pef/logreg/logreg.pef"
+To use Slurm, create submit-logreg-job.sh with the following contents:
+#!/bin/sh
+python logreg.py compile --pef-name="logreg" --output-folder="pef"
+python logreg.py run --pef="pef/logreg/logreg.pef"
+Then
+ +The output, pef/logreg/output.log, will look something like this:
+2023-03-08 21:18:25.168190: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 FMA
+To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
+2023-03-08 21:18:25.334389: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
+2023-03-08 21:18:25.334430: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
+2023-03-08 21:18:26.422458: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
+2023-03-08 21:18:26.422701: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
+2023-03-08 21:18:26.422709: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
+[Info][SAMBA]# Placing log files in /home/wilsonb/apps/starters/logreg/pef/logreg/logreg.samba.log
+[Info][MAC]# Placing log files in /home/wilsonb/apps/starters/logreg/pef/logreg/logreg.mac.log
+...
+
+Epoch [1/1], Step [10000/60000], Loss: 0.4642
+Epoch [1/1], Step [20000/60000], Loss: 0.4090
+Epoch [1/1], Step [30000/60000], Loss: 0.3863
+Epoch [1/1], Step [40000/60000], Loss: 0.3703
+Epoch [1/1], Step [50000/60000], Loss: 0.3633
+Epoch [1/1], Step [60000/60000], Loss: 0.3553
+Test Accuracy: 91.40 Loss: 0.3014
+2023-03-08T21:19:08 : [INFO][LIB][2688517]: sn_create_session: PEF File: pef/logreg/logreg.pef
+The UNet application example is provided in the the path : /opt/sambaflow/apps/image/segmentation/. As any other application, we first compile and then train the model using compile and run arguments respectively.
+The scripts containing the compile and run commands for UNet2D model can be accessed at Unet2d.sh or at /data/ANL/scripts/Unet2d.sh on any SN30 compute node.
Change directory and copy files.
+ +Copy and paste the contents of +Unet2d.sh +to a file with the same name into the current directory using your favorite editor.
+ +Run these commands for training (compile + train):
+./Unet2d.sh compile <image size> <batch_size> <num of instances> <RunID>
+./Unet2d.sh run <image size> <batch_size> <num of instances> <RunID>
+The compile and run arguments of the script can only be run with number of instances equal to 1, indicating that this is a simple 4 tile run without data parallel framework.
+For a image size of 256x256 and batch size 256 when running just 1 instance, the commands are provided as follows.
The above commands displays the file that contains the output for the execution of the above scripts, usually /data/ANL/results/<hostname>/<userid>/<RunID>/Unet2d.out
If we inspect the compile and run commands for the UNet application provided in the script, we see that the application is compiled with --num-tiles 4, which means that the entire application fits on 4 tiles or half of a RDU.
+The pef generated from the compilation process of the above command is placed under out/Unet2d/unet_train_256_256_single_4 inside the current working directory.
python ${UNET}/compile.py compile --mac-v2 --in-channels=3 --in-width=${2} --in-height=${2} --batch-size=${BS} --enable-conv-tiling --num-tiles=4 --pef-name=unet_train_${BS}_${2}_single_${NUM_TILES} --output-folder=${OUTDIR}
+srun --nodelist $(hostname) python /opt/sambaflow/apps/image/segmentation//hook.py run --data-cache=${CACHE_DIR} --data-in-memory --num-workers=${NUM_WORKERS} --enable-tiling --min-throughput 395 --in-channels=3 --in-width=${2} --in-height=${2} --init-features 32 --batch-size=${BS} --epochs 10 --data-dir ${DS} --log-dir log_dir_unet_${2}_${BS}_single_${NUM_TILES} --pef=${OUTDIR}/unet_train_${BS}_${2}_single_${NUM_TILES}/unet_train_${BS}_${2}_single_${NUM_TILES}.pef
+The performance data is located at the bottom of log file.
+inner train loop time : 374.6789753437042 for 10 epochs, number of global steps: 130, e2e samples_per_sec: 88.82270474202953
+The Gpt 1.5B application example is provided in the the path : /opt/sambaflow/apps/nlp/transformers_on_rdu/.
+The scripts containing the compile and run commands for Gpt1.5B model can be accessed at the path /data/ANL/scripts/Gpt1.5B_base_single_compile.sh and /data/ANL/scripts/Gpt1.5B_base_single_run.sh on any SN30 compute node. This script is compiled and run for only 1 instance and the model fits on 4 tiles or half of a RDU. The scripts are provided for reference.
Change directory and copy files.
+ +Copy and paste the contents of +Gpt1.5B_base_single_compile.sh and Gpt1.5B_base_single_run.sh +to a file with the same names into the current directory using your favorite editor.
+or copy the contents from /data/ANL/scripts/Gpt1.5B_base_single_compile.sh and /data/ANL/scripts/Gpt1.5B_base_single_run.sh.
cp /data/ANL/scripts/Gpt1.5B_base_single_compile.sh ~/apps/nlp/Gpt1.5B_single/
+cp /data/ANL/scripts/Gpt1.5B_base_single_run.sh ~/apps/nlp/Gpt1.5B_single/
+Run the script with batch size as an argument(shown below with an example of 32).
+ +The Gpt1.5B_base_single_compile.sh script will internally call the Gpt1.5B_base_single_run.sh to perform the training. You can inspect the compile and run commands in the scripts to learn that this model trains with a batch size of 32 for 1 instance over 4 tiles. The human decision file and the compiler config file helps to optimize the compute and memory resources specific to this Gpt 1.5B model run.
python /opt/sambaflow/apps/nlp/transformers_on_rdu/transformers_hook.py compile --pef-name=GPT1.5B_base_single_32 --output-folder=/data/scratch/user/GPT1.5B_base_single_32 --module_name gpt2_pretrain --task_name clm --max_seq_length 1024 -b 32 --output_dir=/data/scratch/user/GPT1.5B_base_single_32/hf_gpt1dot5b_ss1k_gas_1_bs32 --overwrite_output_dir --do_train --per_device_train_batch_size 32 --tokenizer_name gpt2 --model_name gpt2 --mac-v2 --non_split_head --mac-human-decision /opt/sambaflow/apps/nlp/transformers_on_rdu/human_decisions_gm/mac_v2_overrides/gpt2_48_enc_full_recompute_training_spatialmapping_tiling16_clmerge_gm_pardp2_lnsd.json --compiler-configs-file /opt/sambaflow/apps/nlp/transformers_on_rdu/human_decisions_gm/compiler_configs/compiler_configs_gpt1dot5b_perf.json --skip_broadcast_patch --config_name /opt/sambaflow/apps/nlp/transformers_on_rdu/customer_specific/mv/configs/gpt2_config_xl_50260.json --no_index_select_patch --weight_decay 0.1 --max_grad_norm_clip 1.0 --num-tiles 4 --enable-stochastic-rounding
+COMMAND= /usr/local/bin/srun --mpi=pmi2 python /opt/sambaflow/apps/nlp/transformers_on_rdu/transformers_hook.py run -b 32 --data_dir /data/ANL/ss1024 --pef=/data/scratch/user/GPT1.5B_base_single_32/GPT1.5B_base_single_32/GPT1.5B_base_single_32.pef --output_dir=/data/scratch/user/GPT1.5B_base_single_32/hf_gpt1dot5b_ss1k_gas_1_bs16 --module_name gpt2_pretrain --task_name clm --max_seq_length 1024 --overwrite_output_dir --do_train --per_device_train_batch_size 32 --tokenizer_name gpt2 --model_name gpt2 --non_split_head --skip_broadcast_patch --no_index_select_patch --config_name /opt/sambaflow/apps/nlp/transformers_on_rdu/customer_specific/mv/configs/gpt2_config_xl_50260.json --max_grad_norm_clip 1.0 --skip_checkpoint --logging_steps 1 --max_steps 75000 --learning_rate 0.00025 --steps_this_run 100
+The sntilestat command shows that the application runs on 4 tiles as shown below.
/XRDU_0/RDU_0/TILE_0 2.1 96.9 0.8 0.1 0.0 0.0 796481 user python /opt/sambaflow/apps/nlp/transformers_on_rdu/
+/XRDU_0/RDU_0/TILE_1 2.1 96.9 0.8 0.1 0.0 0.0 796481 user python /opt/sambaflow/apps/nlp/transformers_on_rdu/
+/XRDU_0/RDU_0/TILE_2 2.5 96.9 0.4 0.1 0.0 0.0 796481 user python /opt/sambaflow/apps/nlp/transformers_on_rdu/
+/XRDU_0/RDU_0/TILE_3 2.5 96.9 0.4 0.1 0.0 0.0 796481 user python /opt/sambaflow/apps/nlp/transformers_on_rdu/
+/XRDU_0/RDU_0/TILE_4 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_0/TILE_5 100.0 0.0 0.0 0.0 0.0 0.0
+...
+SambaNova SN30 can be accessed using your ALCF account. See Get Started +to request an account and for additional information.
+Connection to a SambaNova node is a two-step process. The first step is to ssh to the login node. +This step requires an MFA passcode for authentication - an +eight-digit passcode generated by an app on your mobile device, e.g., MobilePASS+. +The second step is to log in to a SambaNova node from the login node.
+
Log in to the SambaNova login node from your local machine using the below command. This uses the MobilePASS+ token generated every time you log in to the system. This is the same passcode used to authenticate into other ALCF systems, such as Polaris.
+In the examples below, replace ALCFUserID with your ALCF user id.
+ +++Note: Use the ssh "-v" option in order to debug any ssh problems.
+
Once you are on the login node, a SambaNova node can be accessed using an alias, sn30-r[1-4]-h[1-2] where 'r' stands for the rack number, and 'h' stands for host. sn30-r1-h1 is the first host of the first rack.
+The 8 nodes are aliased as : sn30-r1-h1 , sn30-r1-h2, sn30-r2-h1, sn30-r2-h2, sn30-r3-h1, sn30-r3-h2, sn30-r4-h1, sn30-r4-h2.
+sn30-r1-h1 can be accessed as below.
+ +The required software environment (SambaFlow software stack and the associated environmental variables) for a SN30 node is set up automatically at login. This is unlike the SN10 where the environment had to be set up by each user.
+ + + + + + + + + + + + + + + +SambaNova uses Slurm for job submission and queueing. Below are some of the important commands for using Slurm. For more information refer to Slurm Documentation.
+++Note: Run the Python scripts using 'srun' or 'sbatch', to ensure that concurrent jobs do not interfere with each other.
+Note: There is just one scheduler for all of the SambaNova nodes.
+
The Slurm command srun can be used to run individual Python scripts in parallel with other scripts on a cluster managed by Slurm. Examples of srun usage are shown below.
Slurm will assign a nodelist/host to run a job if a host is not specified.
+Example:
+srun python lenet.py compile -b=1 --pef-name="lenet" --output-folder="pef"
+srun python lenet.py run --pef="pef/lenet/lenet.pef"
+You may specify which node/host on which to run a job.
+Reasons to specify a node list:
+Example:
+ +Alternatively, these jobs can be submitted to the Slurm workload manager through a batch script by using the sbatch command. To do this, create a bash script (submit-lenet-job.sh here as an example) with the commands that you want to execute.
#!/bin/sh
+
+python lenet.py compile -b=1 --pef-name="lenet" --output-folder="pef"
+python lenet.py run --pef="pef/lenet/lenet.pef"
+Then pass the bash script as an input to the sbatch command as shown below.
In case of the need to use multiple RDUs (2 in the example shown below), the sbatch command would be altered as:
The squeue command provides information about jobs located in the Slurm scheduling queue.
SInfo is used to view partition and node information for a system running Slurm.
+Here is a suggested command:
+ +For more information, see SInfo.
+SCancel is used to signal or cancel jobs, job arrays, or job steps.
+ + + + + + + + + + + + + + + + +To find the SDK version, run the following commands
+# TODO
+(venv) ALCFUserID@sn30-r1-h1:~$ python
+Python 3.7.6 (default, Feb 18 2020, 21:28:31)
+[GCC 9.3.0] on linux
+Type "help", "copyright", "credits" or "license" for more information.
+>>> import sambaflow
+>>> sambaflow.__version__
+'1.11.5'
+>>>
+The OMP_NUM_THREADS environment variable sets the number of threads to use for parallel regions.
+The value of this environment variable must be a list of positive integer values. The values of the list set the number of threads to use for parallel regions at the corresponding nested levels.
+For the SambaNova system it, is usually set to one.
+ +Two copies of the model are maintained. One in host CPU memory and one in RDU +memory. They do not interfere with each other unless you explicitly sync +the model/parameter in between using:
+SambaTensor.rdu() # Moves the CPU model to the RDU
+SambaTensor.cpu() # Moves the RDU model to the CPU
+In order to run the model on the CPU, you can simply use the PyTorch model +as if there is no RDU. +In order to run the model on RDU, you would need to use session.run().
+The snconfig utility shows the static configuration of the system. The configuration for the first node is as follows:
+======================================================
+======= NODE Info =======
+======================================================
+======= Static Info =======
+Timestamp: 2023-03-16 17:00:04
+Platform Name: DataScale SN30-8
+Node Name: NODE
+ Number of XRDUS: 4
+ XRDU Name: XRDU_0
+ Number of RDUS: 2
+ RDU name: RDU_0
+ Serial Number : 205057B469B35895
+ Number of TILES: 8
+ TILE Name: TILE_0
+ Serial Number : N/A
+ TILE Name: TILE_1
+ Serial Number : N/A
+
+
+...
+
+
+ Size : 128.0 GB
+ Serial Number : 1F5BC22
+ DDR CH Name: DDRCH_6
+ Number of DIMMS: 1
+ DIMM Name: DIMM_L0
+ Size : 128.0 GB
+ Serial Number : 1F5BC99
+ DDR CH Name: DDRCH_7
+ Number of DIMMS: 1
+ DIMM Name: DIMM_M0
+ Size : 128.0 GB
+ Serial Number : 1F5BB68
+ Total XRDU_3 memory size (GB): 2048.0
+The following command checks if the SambaNova daemon service is running.
+ +The output should look something like this:
+● snd.service - SN Devices Service
+ Loaded: loaded (/lib/systemd/system/snd.service; enabled; vendor preset: enabled)
+ Drop-In: /etc/systemd/system/snd.service.d
+ └─override.conf
+ Active: active (running) since Fri 2023-01-27 04:03:14 UTC; 1 months 18 days ago
+ Main PID: 5635 (snd)
+ Tasks: 9 (limit: 629145)
+ Memory: 156.8M
+ CGroup: /system.slice/snd.service
+ └─5635 /opt/sambaflow/bin/snd
+
+Warning: some journal files were not opened due to insufficient permissions.
+The output shown below is when the system is completely idle.
+TILE %idle %exec %pload %aload %chkpt %quiesce PID USER COMMAND
+/XRDU_0/RDU_0/TILE_0 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_0/TILE_1 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_0/TILE_2 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_0/TILE_3 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_0/TILE_4 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_0/TILE_5 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_0/TILE_6 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_0/TILE_7 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_1/TILE_0 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_1/TILE_1 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_1/TILE_2 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_1/TILE_3 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_1/TILE_4 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_1/TILE_5 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_1/TILE_6 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_0/RDU_1/TILE_7 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_0/TILE_0 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_0/TILE_1 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_0/TILE_2 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_0/TILE_3 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_0/TILE_4 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_0/TILE_5 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_0/TILE_6 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_0/TILE_7 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_1/TILE_0 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_1/TILE_1 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_1/TILE_2 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_1/TILE_3 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_1/TILE_4 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_1/TILE_5 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_1/TILE_6 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_1/RDU_1/TILE_7 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_0/TILE_0 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_0/TILE_1 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_0/TILE_2 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_0/TILE_3 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_0/TILE_4 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_0/TILE_5 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_0/TILE_6 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_0/TILE_7 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_1/TILE_0 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_1/TILE_1 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_1/TILE_2 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_1/TILE_3 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_1/TILE_4 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_1/TILE_5 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_1/TILE_6 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_2/RDU_1/TILE_7 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_0/TILE_0 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_0/TILE_1 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_0/TILE_2 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_0/TILE_3 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_0/TILE_4 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_0/TILE_5 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_0/TILE_6 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_0/TILE_7 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_1/TILE_0 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_1/TILE_1 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_1/TILE_2 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_1/TILE_3 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_1/TILE_4 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_1/TILE_5 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_1/TILE_6 100.0 0.0 0.0 0.0 0.0 0.0
+/XRDU_3/RDU_1/TILE_7 100.0 0.0 0.0 0.0 0.0 0.0
+Use one of
+ + + + + + + + + + + + + + + + +++Note: Please be mindful of how you are using the system. +For example, consider running larger jobs in the evening or on weekends
+Note: Please use only Slurm commands, i.e., srun and sbatch, to run your code. +If you run your code directly using the 'python' command, it may cause conflicts +on the system.
+Note: If you have conda installed and a conda environment is active, you will see something like
+(base)at the beginning of the command prompt. If so, you will need to deactivate it withconda deactivate. Conda is not used on the SambaNova SN30 cluster.
The SambaNova workflow includes the following main steps to run a model.
+The system uses the Slurm job +scheduler to schedule the jobs and manage the workload on the system. For more information on Slurm, see Job Queueing and Submission.
+Example Programs lists the different example applications with corresponding commands for each of the above steps.
+Compiles the model and generates a .pef file. This file contains +information on how to reconfigure the hardware, and map the compute and +memory resources required to run an application on RDUs. +The pef files are by default saved in the 'out' directory; the +SambaNova documentation advises saving pef files in separate +directories with the '--output-folder' option.
+It is necessary to re-compile only when the model changes, or parameters specific to the model graph change, including the batch size.
+Compile times can be significant. Compiling the UNet sample, for example, when using images of size 32x32 pixels, takes 358(s), and 1844(s) for images of size 256x256.
+The entire compile process is executed on the host and no RDUs are involved in the compile step.
+Example of compiling the LeNet application:
+ +where
+| Argument | +Default | +Help | +
|---|---|---|
| -b | +1 | +Batch size for training | +
| + | + | + |
As part of this step, the model is trained on the RDUs by passing in the PEF file and the training dataset. The location of the pef file generated in the compile step is passed as an argument to the run command. Below is the example of the run command that trains a LeNet model.
The location of the pef file generated in the compile step is passed as an argument to the run command.
+This command is used to run the model on both the host CPU and a SambaNova RDU. It compares the results from the CPU and RDU and will report if any discrepancies are found. Pass the pef file generated as part of the compile step as the input to this command.
+ + + + + + + + + + + + + + + + +This section covers how to use the SambaTune profiling performance tuning tool, and the SambaTune UI for viewing the results.
+ +SambaTune uses a yaml file that describes how to profile an application.
+There are samples in /opt/sambaflow/sambatune/configs.
+This section shows how to run the simplest sample, a linear net.
First, ssh into one of the nodes in the SN30 cluster.
+Next, start a slurm interative job reserving a full node (8 RDUs), for 8 hours (480 minutes):
+
Next, set an environment variable indicating where the profiling information should be stored: +
+If running a large model, the profiling information can be hundreds of gigabytes or more, and the DUMP_ROOT should be set to some location with more storage than your home directory (which has a quota).
+E.g. somewhere that you have write access to in /projects
Optionally, examine the sample yaml file. You will see that it has 5 top-level sections: app:, model-args:, compile-args:, run-args:, env:
Next, run sambatune using a sample sambatune yaml configuration file. This sample command line requests profiling with the benchmark, instrument, and run modes.
+
This will take a while to run, particularly if the yaml for a larger model is used.
+Then, run sambatune_ui:
+
$ export ST_PORT=8576
+$ sambatune_ui --directory $DUMP_ROOT/artifact_root/sambatune_gen --port $ST_PORT
+Copy the password shown (e.g. to your clipboard). The userid is always admin. The password is different for every sambatune_ui run.
+In a fresh console on your working machine where you will run the browser, set up a two-hop ssh tunnel to the target node. Replace the ALCFUserID in the ssh command line with your ALCF userid.
+
$ export ST_PORT=8576
+$ ssh -L $ST_PORT:localhost:$ST_PORT ALCFUserID@sambanova.alcf.anl.gov -t ssh -L $ST_PORT:localhost:$ST_PORT -N sn30-r1-h1
+Put localhost:8576 in the url bar of a Chrome-family browser. (Chrome, Brave, Vivaldi, Opera tested.)
+A login prompt for the sambatune ui should show.
+Enter admin and the password copied previously.
+You should now see the SambaTune UI.
If the browser does not show a login prompt, or if any previous step complains about a port conflict, try another value for ST_PORT on both the target node and for the ssh tunnel command, e.g. 8577.
+See SambaNova's SambaTune documentation for more information about using SambaTune and the SambaTune UI.
+This section is a good starting point: Workflow overview
When finished:
+- Break the ssh tunnel with ctrl-c or equivalent.
+- Stop the sambatune_ui server on the target node with ctrl-c or equivalent.
+- Exit the interactive slurm job to release the reserved resources.
A disconnected job can be canceled by determining its job id with squeue -a and canceling the job with scancel <jobid>
The SambaNova DataScale SN30 system is architected around the next-generation Reconfigurable Dataflow Unit (RDU) processor for optimal dataflow processing and acceleration. The AI Testbed's SambaNova SN30 system consists of eight nodes in 4 full racks, each node featuring eight RDUs interconnected to enable model and data parallelism. SambaFlow, Sambanova's software stack, extracts, optimizes, and maps the dataflow graphs to the RDUs from standard machine learning frameworks like PyTorch.
+Below are some of the links to SambaNova documentation.
+SambaNova white paper: Accelerated Computing with a Reconfigurable Dataflow Architecture
+SN30 documentation: SambaNova Documentation
+ + + + + + + + + + + + + + + +Port forwarding is covered here. This is specifically for TensorBoard.
+This section describes the steps to be followed to set up port forwarding for applications, +like TensorBoard, which runs on the SambaNova system and binds to one or more ports. +This example uses 6006 and 16006 as port numbers. Using port numbers other than these may +avoid collisions with other users.
+Replace ALCFUserID with your ALCF User ID.
+Run
+# Forward a port number from sambanova.alcf.anl.gov to your local machine.
+ssh -v -N -f -L localhost:16006:localhost:16006 ALCFUserID@sambanova.alcf.anl.gov
+...
+Password: < MobilePass+ code >
+
+# Connect to sambanova.alcf.anl.gov
+ssh ALCFUserID@sambanova.alcf.anl.gov
+...
+Password: < MobilePass+ code >
+Below are the commands specific to sn30-r1-h1. You may replace sn30-r1-h1 with any other node when using the appropriate system.
+Run
+++Note: The full name is sn30-r1-h1.ai.alcf.anl.gov and it may also be used.
+
# Forward the port.
+ssh -N -f -L localhost:16006:localhost:6006 ALCFUserID@sn30-r1-h1
+# Connect to the system.
+ssh ALCFUserID@sn30-r1-h1
+Activate the venv appropriate to your project.
+Navigate to the appropriate directory for your model. +Launch your model using srun or sbatch.
+ +The SambaNova system has a bash shell script to setup the required software environment. +This sets up the SambaFlow software stack, the associated environmental variables and activates +a pre-configured virtual environment.
+Use the command appropriate for your environment.
+For example, if you are using LogReg:
+ALCFUserID@sn30-r1-h1:~$ source /opt/sambaflow/apps/starters/logreg/venv/bin/activate
+(venv) ALCFUserID@sn30-r1-h1:~$
+Navigate to the appropriate directory for your model.
+ +Then, navigate in your browser to, in this example, http://localhost:16006 on your local machine.
+Explanation of ssh command:
+-N : no remote commands
+
+-f : put ssh in the background
+
+-L <machine1>:<portA>:<machine2>:<portB> :
+
+The full command line will forward <machine2>:<portB> (remote scope) to <machine1>:<portA> (local scope)
+Adapted from: How can I run Tensorboard on a remote server?
+ + + + + + + + + + + + + + + +The intent of this page is to show conceptually how to convert a model to run on the SambaNova system. +It is not necessary to convert CosmicTagger because it has already been converted and is +located at CosmicTagger on the SambaNova branch. +The original is located at CosmicTagger.
+The first step to converting a model is to verify that it runs on the CPU. This step has been verified for CosmicTagger.
+CosmicTagger can run on multiple machines. As such, it is necessary to specify the architecture +that one is using. For example, CPU or GPU. The architecture is stored in the +ComputeMode class.
+Edit src/config/config.py. Add RDU to the ComputeMode class.
+ +Edit src/utils/torch/trainer.py.
+Insert the imports at the top of the file.
+SambaFlow is a complete software stack designed to take input from standard machine learning frameworks such as PyTorch and TensorFlow. SambaFlow automatically extracts, optimizes, and maps dataflow graphs onto RDUs.
+try:
+ from sambaflow import samba
+
+ import sambaflow.samba.utils as utils
+ from sambaflow.samba.utils.argparser import parse_app_args
+ from sambaflow.samba.utils.common import common_app_driver
+except:
+ pass
+Wrap the model using poptorch.trainingModel() so that it may be ran on IPUs for training.
+Wrap the model using poptorch.inferenceModel() when not training.
+Find the following code around line 90 in the init_network method.
+ # Foregoing any fusions as to not disturb the existing ingestion pipeline
+ if self.is_training() and self.args.mode.quantization_aware:
+ self._raw_net.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm')
+ self._net = torch.quantization.prepare_qat(self._raw_net)
+ else:
+ self._net = self._raw_net
+After the above code, add:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ if self.is_training():
+ opts = poptorch.Options()
+ self._net = poptorch.trainingModel(self._net, opts, optimizer=torch.optim.SGD(self._net.parameters(), lr=1e-3))
+ else:
+ self._net = poptorch.inferenceModel(self._net)
+See poptorch.trainingModel() and poptorch.inferenceModel() for more information.
+There is also a Build the Model tutorial.
+Update init_optimizer() to use the poptorch class instead of the torch class as needed.
+Change:
+ if self.args.mode.optimizer.name == OptimizerKind.rmsprop:
+ self._opt = torch.optim.RMSprop(self._net.parameters(), 1.0, eps=1e-4)
+ else:
+ self._opt = torch.optim.Adam(self._net.parameters(), 1.0)
+to:
+ if self.args.mode.optimizer.name == OptimizerKind.rmsprop:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ self._opt = poptorch.optim.RMSprop(self._net.parameters(), 1.0, eps=1e-4)
+ else:
+ self._opt = torch.optim.RMSprop(self._net.parameters(), 1.0, eps=1e-4)
+ else:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ self._opt = poptorch.optim.Adam(self._net.parameters(), 1.0)
+ else:
+ self._opt = torch.optim.Adam(self._net.parameters(), 1.0)
+Putting the loss calculation in forward_pass() allows the loss computation to be performed on the IPUs. +This will be faster because the data will not need to be transfered round-trip to the CPU.
+Change forward_pass():
+ if net is None:
+ logits_image = self._net(minibatch_data['image'])
+ else:
+ logits_image = net(minibatch_data['image'])
+The following code changes are to account for the loss function, i.e., self.loss_calculator, and the +image labels, i.e., labels_image, to be passed to the model's forward_pass method. Additionally, the calculated +loss is returned from the forward_pass method.
+ if net is None:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ logits_image, labels_image, loss = self._net(minibatch_data['image'], self.loss_calculator, labels_image)
+ return logits_image, labels_image, loss
+ else:
+ logits_image = self._net(minibatch_data['image'])
+ else:
+ if self.args.run.compute_mode == ComputeMode.IPU and self.args.mode.name != ModeKind.inference:
+ logits_image, labels_image, loss = net(minibatch_data['image'], self.loss_calculator, labels_image)
+ return logits_image, labels_image, loss
+ else:
+ logits_image = net(minibatch_data['image'])
+Receive the extra loss variable from the forward_pass method.
+Update the train_step method.
+ with self.timing_context("forward"):
+ if self.args.run.precision == Precision.mixed and self.args.run.compute_mode == ComputeMode.GPU:
+ with torch.cuda.amp.autocast():
+ logits_image, labels_image = self.forward_pass(minibatch_data)
+ else:
+ logits_image, labels_image = self.forward_pass(minibatch_data)
+
+ verbose = False
+
+ # Compute the loss based on the logits
+ with self.timing_context("loss"):
+ loss = self.loss_calculator(labels_image, logits_image)
+The forward_pass() method was changed to return the extra variable loss in the previous section. It is now +received conditionally when using an IPU(s).
+In the with self.timing_context("loss"): section, only calculate loss if not using an IPU(s).
+ with self.timing_context("forward"):
+ if self.args.run.precision == Precision.mixed and self.args.run.compute_mode == ComputeMode.GPU:
+ with torch.cuda.amp.autocast():
+ logits_image, labels_image = self.forward_pass(minibatch_data)
+ else:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ logits_image, labels_image, loss = self.forward_pass(minibatch_data)
+ else:
+ logits_image, labels_image = self.forward_pass(minibatch_data)
+
+ verbose = False
+
+
+ # Compute the loss based on the logits
+ with self.timing_context("loss"):
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ loss = loss
+ else:
+ loss = self.loss_calculator(labels_image, logits_image)
+Update the val_step method.
+Find this code.
+ if self.args.run.precision == Precision.mixed and self.args.run.compute_mode == ComputeMode.GPU:
+ with torch.cuda.amp.autocast():
+ logits_image, labels_image = self.forward_pass(minibatch_data, net=val_net)
+ else:
+ logits_image, labels_image = self.forward_pass(minibatch_data, net=val_net)
+
+ # Compute the loss based on the logits
+ loss = self.loss_calculator(labels_image, logits_image)
+Change the code to the following.
+ if self.args.run.precision == Precision.mixed and self.args.run.compute_mode == ComputeMode.GPU:
+ with torch.cuda.amp.autocast():
+ logits_image, labels_image = self.forward_pass(minibatch_data, net=val_net)
+
+ # Compute the loss based on the logits
+ loss = self.loss_calculator(labels_image, logits_image)
+ else:
+ if self.args.run.compute_mode == ComputeMode.IPU:
+ logits_image, labels_image, loss = self.forward_pass(minibatch_data, net=val_net)
+ else:
+ logits_image, labels_image = self.forward_pass(minibatch_data, net=val_net)
+
+ # Compute the loss based on the logits
+ loss = self.loss_calculator(labels_image, logits_image)
+The Graphcore system is more computationally efficient if the loss function is on the +IPU. This is accomplished by using the loss function within the model's forward method.
+Edit src/networks/torch/uresnet2D.py.
+Find the forward method.
+ +Update the argument list to include the loss function, i.e., loss_calculator +and the image labels, i.e., labels_image.
+ +Add the loss calculation just before the forward method returns.
+ if loss_calculator is not None:
+
+ labels_image = labels_image.long()
+ labels_image = torch.chunk(labels_image, chunks=3, dim=1)
+ shape = labels_image[0].shape
+ labels_image = [ _label.view([shape[0], shape[-2], shape[-1]]) for _label in labels_image ]
+
+ loss = loss_calculator(labels_image, x)
+ import poptorch
+ loss = poptorch.identity_loss(loss , reduction="mean")
+ return x, labels_image, loss
+
+ # This return already exists.
+ return x
+The poptorch.identity_loss method takes a single PyTorch tensor and will backpropagate a gradient of ones through it. You may find an example at here
+The following is included for completeness. One will not likely find this in other code.
+Open bin/exec.py in your favorite editor. Change:
+ +to
+ + + + + + + + + + + + + + + + +This is an example for measuring TFLOPs for Conv2D forward pass.
+elif args.command == 'run':
+ samba.session.run(inputs, section_types=['fwd'])
+ #samba.session.run(inputs, section_types=['bckwd'])
+ n_iters = 100
+ forward_pass_time = []
+ print("run starts")
+ start_time_forward = time.time()
+ for loop in range(n_iters):
+ samba.session.run(inputs, section_types=['fwd'])
+ #samba.session.run(inputs, section_types=['bckwd'])
+ #samba.session.run(inputs, section_types=['fwd', 'bckwd'])
+ end_time_forward = time.time()
+ forward_pass_time.append(end_time_forward - start_time_forward)
+ print("run ends")
+
+ w_0 = (args.w + 2*args.pad_w - args.s)/args.wstride + 1
+ h_0 = (args.h + 2*args.pad_h - args.r)/args.hstride + 1
+ tflops = 2 * (w_0*h_0) * args.s * args.r * args.c * args.k * args.n
+ tflops_forw = tflops/(sum(forward_pass_time)/n_iters/5)/(10**12) #tflops
+ print(tflops)
+ print(sum(forward_pass_time))
+ print("tflops: %f"%tflops_forw)
+ print("SN,Training,%s,Conv2d_fwd,%d,100,1,%d,%d,%d,%d,%d,%d,%d,0.0,%f,None,%f,%f,%f" % ("dtype", args.n, args.w, args.h, args.c, args.k, args.s, args.pad_w, args.wstride, (sum(forward_pass_time)/n_iters)/args.n, args.n/(sum(forward_pass_time)/n_iters), tflops_forw, (sum(forward_pass_time)/n_iters)/args.n))
+This GPT-2 example is for 1.5B parameters on two (2) nodes. +Each node has eight (8) RDUs for a total of sixteen (16) RDUs.
+Using your favorite editor, create the file 'Gpt1.5B.sh'.
+Copy the contents of Gpt1.5B.sh.
+Make the script executable:
+ +Gpt1.5B.sh contains the sbatch command:
+/usr/local/bin/sbatch --output=${HOME}/slurm-%A.out --ntasks 32 --gres=rdu:1 --ntasks-per-node 16 --nodes 2 --cpus-per-task=8 /data/ANL/scripts/Gpt1.5B_run.sh ${1} >> ${OUTPUT_PATH} 2>&1
+The sbatch nodes argument specifies the number of nodes to use.
+nodes 2 Nodes to use.
+Additionally, here are the other sbatch arguments.
+--ntasks 32: This option specifies the number of tasks to be used in the job.
+ntasks-per-node 16: This option specifies the number of tasks per node.
+gres=rdu:1 Indicates the model fits on a single RDU.
+cpus-per-task=8 CPUs per task.
+The script accepts an optional first parameter to specify the log directory.
+Run the script:
+ +The output can be found at /data/ANL/results/$(hostname)/${USER}/${LOGDIR}/${MODEL_NAME}.out. +The actual path will be displayed on the screen.
+ + + + + + + + + + + + + + + +Establish a test directory from which to work.
+ +Copy BertLarge.sh into your current directory.
+ +Let's cover several options for executing the script.
+This is helpful if doing multiple runs and one wishes to specify a run ID. + This bash script argument is optional. Place it at the very end of the command.
+Example:
+sbatch --output=${HOME}/app-test/slurm-%A.out --cpus-per-task=128 --gres=rdu:16 BertLarge.sh my_runID
+One may optionally specify a nodelist for sbatch. An example is to use hostname.
+sbatch --nodelist $(hostname) --output=${HOME}/app-test/slurm-%A.out --cpus-per-task=128 --gres=rdu:16 BertLarge.sh
+Let's specify the log file and the nodelist.
+Run
+sbatch --nodelist $(hostname) --output=${HOME}/app-test/slurm-%A.out --cpus-per-task=128 --gres=rdu:16 BertLarge.sh
+Display the slurm output. For example:
+ +The output will look something like:
+ +You may display that file. You may want to use less to do so because it is quite long.
+ +The organization of the file is:
+Rick 4/16/2023 [10:16 AM] +/home/rweisner/sambatune_ui_dir contains the 1.15.3 version which is the latest released version. It should work on your experimental. You will need browser access to wherever you install it.
+ +#TODOBRW
+ssh wilsonb@homes.cels.anl.gov
+ssh sm-02
+MobilePass+ password
+On sm-02
+source /opt/sambaflow/venv/bin/activate
+export PATH=/opt/sambaflow/bin:$PATH
+sambatune linear_net.yaml --artifact-root $(pwd)/artifact_root --modes benchmark instrument run
+sambatune_ui --directory /home/wilsonb/tmp/sambatune_gen --port 8580
+#There will be a username and password displayed that you will use in your browser on your laptop.
+Command used on laptop for port forward
+ssh -XL 8580:127.0.0.1:8580 wilsonb@sm-02.cels.anl.gov
+MobilePass+ password
+# You will be logged into sm-02 but, you do not need to do anything.
+address used in browser on laptop localhost:8580
+#Use username and password from sambatune_ui.
+Username
+Password
+
+#TODOBRW
+/home/wilsonb/DL/Sambanova/apps_1.12/private/anl/2022-09-21T19-21-05.html
+SambaTune is a tool for profiling, debugging, and tuning the performance of applications +running on SN hardware.
+The tool automates the collection of hardware performance counters, metrics aggregation, +report generation, and visualization. It also automates benchmarking of the application +to compute average throughput over a sufficient number of runs. The tool is designed to +aid the user with performance bottleneck analysis and tuning.
+SambaTune is currently used by SN engineers involved in performance tuning efforts. +SambaTune is also planned for release to external customers to aid with performance +bottleneck analysis and resolution.
+First, enter the virtual environment on sm-01 or sm-02:
+ +Update path:
+ +usage: sambatune [-h] [--artifact-root ARTIFACT_ROOT] [--disable-override]
+ [--compile-only | -m MODES [MODES ...]] [--version]
+ config
+
+positional arguments:
+ config YAML file with model, compile, run configuration.
+
+optional arguments:
+ -h, --help show this help message and exit
+ --artifact-root ARTIFACT_ROOT
+ Custom location to save compile/run artifacts;
+ defaults to '$DUMP_ROOT/artifact_root' (default: None)
+ --disable-override Reuse the placement from the baseline compilation
+ (default: False)
+ --compile-only Run compilation of PEFs for selected modes only
+ (default: False)
+ -m MODES [MODES ...], --modes MODES [MODES ...]
+ Select modes to execute from ['benchmark',
+ 'instrument', 'run'] (default: ['benchmark'])
+ --version version of sambatune and sambaflow.
+By default, it will run with the benchmarking mode enabled. Use the --modes flag to run +modes individually or in any combination. +Benchmark-Only:
+ +Instrument-Only:
+ +All modes:
+ +# From Bill
+python /opt/sambaflow/apps/private/anl/uno_full.py compile --weight-sharing -b 16 -mb 4 --num-spatial-batches 500 --mapping spatial --mac-human-decision /opt/sambaflow/apps/private/anl/samba_uno/human_decisions_spatial.json --pef-name=uno_16_4_500_ws --output-folder=/home/arnoldw//models_dir/1520847 --mac-v1
+
+python /opt/sambaflow/apps/private/anl/uno_full.py run --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches 500 --mapping spatial --pef=/home/arnoldw//models_dir/1520847/uno_16_4_500_ws/uno_16_4_500_ws.pef --in_dir /var/tmp/raw/ --mac-v1
+# From Bill --> Bruce
+python /opt/sambaflow/apps/private/anl/uno_full.py compile --weight-sharing -b 16 -mb 4 --num-spatial-batches 500 --mapping spatial --mac-human-decision /opt/sambaflow/apps/private/anl/samba_uno/human_decisions_spatial.json --pef-name=uno_16_4_500_ws --output-folder='.' --mac-v1
+
+export OMP_NUM_THREADS=1
+python /opt/sambaflow/apps/private/anl/uno_full.py run --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches 500 --mapping spatial --pef=./uno_16_4_500_ws/uno_16_4_500_ws.pef --in_dir /var/tmp/raw/ --mac-v1
+#TODOBRW This works. 9/19/22
+sm-01/home/wilsonb/tmp/uno_test/uno_ccle.yaml
+app: /opt/sambaflow/apps/private/anl/uno_full.py
+
+model-args: --weight-sharing -b 16 -mb 4 --num-spatial-batches 500 --mapping spatial
+
+compile-args: compile --plot --mac-human-decision /opt/sambaflow/apps/private/anl/samba_uno/human_decisions_spatial.json --mac-v1
+
+run-args: --multiprocess-pickle --use-pickle-train --measure-spatial --train-samba-spatial --mac-v1 --train_source CCLE --lr 0.001 --data-dir /software/sambanova/dataset/CCLE_16_500 --converted-pickle
+
+env:
+ OMP_NUM_THREADS: 16,
+ SF_RNT_NUMA_BIND: 2
+Run the following example:
+ +#TODOBRW
+# Stand-alone
+export UNO=.
+export NS=500
+srun python /opt/sambaflow/apps/private/anl/uno_full.py compile --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --mac-human-decision /opt/sambaflow/apps/private/anl/samba_uno/human_decisions_spatial.json --pef-name=uno_16_4_${NS}_ws --output-folder='.' --mac-v1
+
+export OMP_NUM_THREADS=1
+srun python /opt/sambaflow/apps/private/anl/uno_full.py run --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef=./uno_16_4_${NS}_ws/uno_16_4_${NS}_ws.pef --in_dir /var/tmp/raw/ --mac-v1 --train_source CCLE --data-dir /software/sambanova/dataset/CCLE_16_${NS}
+
+export UNO=.
+export NS=500
+export OMP_NUM_THREADS=1
+srun pyinstrument /opt/sambaflow/apps/private/anl/uno_full.py run --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef=./uno_16_4_${NS}_ws/uno_16_4_${NS}_ws.pef --in_dir /var/tmp/raw/ --mac-v1 --train_source CCLE --data-dir /software/sambanova/dataset/CCLE_16_${NS} > pyinstrument_1.13.log 2>&1
+
+
+
+Ricks run python ${UNO}/uno_full.py run --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef=“out/uno_16_4_${NS}/uno_16_4_${NS}.pef” --in_dir /var/tmp/raw/ --mac-v1 --train_source CCLE
+#TODOBRW
+sm-01/home/wilsonb/DL/Sambanova/apps_1.12/private/anl/uno_brw_CCLE_1_12.yaml
+export OMP_NUM_THREADS=16
+app: /home/wilsonb/DL/Sambanova/apps_1.12/private/anl/uno_full.py
+
+model-args: --weight-sharing -b 16 -mb 4 --num-spatial-batches 500 --mapping spatial
+
+compile-args: compile --plot --mac-human-decision /opt/sambaflow/apps/private/anl/samba_uno/human_decisions_spatial.json --mac-v1
+
+run-args: --measure-spatial --train-samba-spatial --mac-v1 --train_source CCLE --lr 0.001 --data-dir /software/sambanova/dataset/CCLE_16_500
+
+env:
+ OMP_NUM_THREADS: 16,
+ SF_RNT_NUMA_BIND: 2
+Run the following example:
+sambatune uno_brw_CCLE_1_12.yaml --artifact-root $(pwd)/artifact_root --modes benchmark instrument run
+
+export UNO=.
+export NS=50
+export OMP_NUM_THREADS=1
+
+srun python /opt/sambaflow/apps/private/anl/uno_full.py compile --mac-human-decision /opt/sambaflow/apps/private/anl/samba_uno/human_decisions_spatial.json --mac-v1
+
+xsrun pyinstrument /opt/sambaflow/apps/private/anl/uno_full.py run --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef=./uno_16_4_${NS}_ws/uno_16_4_${NS}_ws.pef --in_dir /var/tmp/raw/ --mac-v1 --train_source CCLE --data-dir /software/sambanova/dataset/CCLE_16_${NS} --epochs 1 > my.log 2>&1
+
+srun python /opt/sambaflow/apps/private/anl/uno_full.py run --multiprocess-pickle --measure-spatial --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef=./out/uno_full_16_47_${NS}/uno_full_16_47_${NS}.pef --in_dir /var/tmp/raw/ --mac-v1 --train_source CCLE --lr 0.001 --data-dir /software/sambanova/dataset/CCLE_16_${NS} > pyinstrument_1.13.log 2>&1
+
+cat my.log # Has pyinstrument run name.
+pyinstrument --load-prev 2022-09-21T19-21-05 -r html
+
+
+1.13
+
+source /opt/sambaflow/venv/bin/activate
+cd ~/tmp/uno_test/
+export UNO=.
+export NS=500
+export OMP_NUM_THREADS=1
+export PATH=/opt/sambaflow/bin:$PATH
+sntilestat
+
+
+
+./uno_pickl.sh compile 500
+./uno_pickl.sh run 500
+sambatune uno_brw_CCLE_1_12.yaml --artifact-root $(pwd)/artifact_root --modes benchmark instrument run
+
+export UNO=.
+export NS=50
+export OMP_NUM_THREADS=1
+
+srun python /opt/sambaflow/apps/private/anl/uno_full.py compile --mac-human-decision /opt/sambaflow/apps/private/anl/samba_uno/human_decisions_spatial.json --mac-v1
+
+xsrun pyinstrument /opt/sambaflow/apps/private/anl/uno_full.py run --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef=./uno_16_4_${NS}_ws/uno_16_4_${NS}_ws.pef --in_dir /var/tmp/raw/ --mac-v1 --train_source CCLE --data-dir /software/sambanova/dataset/CCLE_16_${NS} --epochs 1 > my.log 2>&1
+
+srun python /opt/sambaflow/apps/private/anl/uno_full.py run --multiprocess-pickle --measure-spatial --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef=./out/uno_full_16_47_${NS}/uno_full_16_47_${NS}.pef --in_dir /var/tmp/raw/ --mac-v1 --train_source CCLE --lr 0.001 --data-dir /software/sambanova/dataset/CCLE_16_${NS} > pyinstrument_1.13.log 2>&1
+
+cat my.log # Has pyinstrument run name.
+pyinstrument --load-prev 2022-09-21T19-21-05 -r html
+
+
+1.13
+
+source /opt/sambaflow/venv/bin/activate
+cd ~/tmp/uno_test/
+export UNO=.
+export NS=500
+export OMP_NUM_THREADS=1
+export PATH=/opt/sambaflow/bin:$PATH
+sntilestat
+uno_pickl.sh
+#! /bin/bash -x
+#set -e
+source /opt/sambaflow/venv/bin/activate
+SECONDS=0
+NS=${2}
+UNO=/opt/sambaflow/apps/private/anl/
+DS="ALL"
+DS="CCLE"
+
+BS=$((NS*16))
+export OMP_NUM_THREADS=16
+
+echo "Model: UNO_SPA_TRN"
+echo "Date: " $(date +%m/%d/%y)
+echo "Time: " $(date +%H:%M)
+if [ "${1}" == "convert" ] ; then
+python3 ${UNO}/uno/uno_data_loaders_converted.py --in_dir /var/tmp/raw/ --out_dir /software/sambanova/dataset/${DS}_16_${NS} --batch-size ${BS} --train_sources ${DS} --file-write-frequency 10
+
+
+elif [ "${1}" == "compile" ] ; then
+ echo "COMPILE"
+ python ${UNO}/uno_full.py compile --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --mac-human-decision ${UNO}/samba_uno/human_decisions_spatial.json --pef-name="uno_16_4_${NS}" --mac-v1
+
+
+elif [ "${1}" == "run" ] ; then
+ echo "RUN ${DS}"
+ SF_RNT_NUMA_BIND=2
+ #python ${UNO}/uno_full.py run --acc-test --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef="out/uno_16_4_${NS}/uno_16_4_${NS}.pef" --in_dir /var/tmp/raw/ --mac-v1 --train_source CCLE
+ python ${UNO}/uno_full.py run --mac-v1 --multiprocess-pickle --use-pickle-train --train-samba-spatial -b 16 -mb 4 --num-spatial-batches ${NS} --lr 0.001 --mapping spatial --data-dir /software/sambanova/dataset/${DS}_16_${NS} --converted-pickle --train_sources ${DS} --pef="out/uno_16_4_${NS}/uno_16_4_${NS}.pef" --epochs 1
+ #python ${UNO}/uno_full.py run --mac-v1 --multiprocess-pickle --use-pickle-train --train-samba-spatial -b 16 -mb 4 --num-spatial-batches ${NS} --lr 0.001 --mapping spatial --data-dir /software/sambanova/dataset/${DS}_16_${NS} --converted-pickle --train_sources ${DS} --pef="out/uno_16_4_${NS}/uno_16_4_${NS}.pef"
+
+elif [ "${1}" == "pyinstrument" ] ; then
+ echo "RUN ${DS}"
+ SF_RNT_NUMA_BIND=2
+ #python ${UNO}/uno_full.py run --acc-test --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef="out/uno_16_4_${NS}/uno_16_4_${NS}.pef" --in_dir /var/tmp/raw/ --mac-v1 --train_source CCLE
+ pyinstrument ${UNO}/uno_full.py run --mac-v1 --multiprocess-pickle --use-pickle-train --train-samba-spatial -b 16 -mb 4 --num-spatial-batches ${NS} --lr 0.001 --mapping spatial --data-dir /software/sambanova/dataset/${DS}_16_${NS} --converted-pickle --train_sources ${DS} --pef="out/uno_16_4_${NS}/uno_16_4_${NS}.pef" --epochs 1
+ #python ${UNO}/uno_full.py run --mac-v1 --multiprocess-pickle --use-pickle-train --train-samba-spatial -b 16 -mb 4 --num-spatial-batches ${NS} --lr 0.001 --mapping spatial --data-dir /software/sambanova/dataset/${DS}_16_${NS} --converted-pickle --train_sources ${DS} --pef="out/uno_16_4_${NS}/uno_16_4_${NS}.pef"
+
+elif [ "${1}" == "no_pickle" ] ; then
+ echo "no_pickle ${DS}"
+ SF_RNT_NUMA_BIND=2
+ python ${UNO}/uno_full.py run --train-samba-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef="out/uno_16_4_${NS}/uno_16_4_${NS}.pef" --in_dir /var/tmp/raw/ --mac-v1 --train_source CCLE
+
+elif [ "${1}" == "mp" ] ; then
+echo "Duration: " $SECONDS
+
+elif [ "${1}" == "mp" ] ; then
+echo "Duration: " $SECONDS
+echo "PERF"
+python uno_full.py measure-performance --measure-spatial --weight-sharing -b 16 -mb 4 --num-spatial-batches ${NS} --mapping spatial --pef="out/uno_16_4_${NS}/uno_16_4_${NS}.pef" --num-iterations 20 --mac-v1
+fi
+
+echo "Duration: " $SECONDS
+./uno_pickl.sh compile 500
+./uno_pickl.sh run 500
+./uno_pickl.sh pyinstrument 500
+pyinstrument --load-prev 2022-09-22T18-31-24 -r html
+stdout is a terminal, so saved profile output to /tmp/tmpeo5ehksn.html
+cp /tmp/tmpeo5ehksn.html .
+On dev terminal
+ +View in local browser.
+Create a directory for your work.
+ +Create small_vae.yaml with the following content using your favorite editor.
+app: /opt/sambaflow/apps/private/anl/moleculevae.py
+
+model-args: -b 128 --in-width 512 --in-height 512
+
+compile-args: compile --plot --enable-conv-tiling --compiler-configs-file /opt/sambaflow/apps/private/anl/moleculevae/compiler_configs_conv.json --mac-v2 --mac-human-decision /opt/sambaflow/apps/private/anl/moleculevae/symmetric_human_decisions_tiled_v2.json
+
+run-args: --input-path /var/tmp/dataset/moleculevae/ras1_prot-pops.h5 --out-path ${HOME}/moleculevae_out --model-id 0 --epochs 10
+
+env:
+ OMP_NUM_THREADS: 16
+ SF_RNT_FSM_POLL_BUSY_WAIT: 1
+ SF_RNT_DMA_POLL_BUSY_WAIT: 1
+ CONVFUNC_DEBUG_RUN: 0
+Run the following example:
+ +Create linear_net.yaml with the following content using your favorite editor.
+app: /opt/sambaflow/apps/micros/linear_net.py
+
+model-args: >
+ -b 1024
+ -mb 64
+ --in-features 8192
+ --out-features 4096
+ --repeat 128
+ --inference
+
+compile-args: >
+ --n-chips 2
+ --plot
+
+env:
+ SF_RNT_FSM_POLL_BUSY_WAIT: 1
+ SF_RNT_DMA_POLL_BUSY_WAIT: 1
+ CONVFUNC_DEBUG_RUN": 0
+NOTE: The following takes 45 minutes to run.
+Run the following example:
+ +#TODOBRW
+cd ~/tmp/uno_test
+screen
+sambatune uno.yaml --artifact-root $(pwd)/artifact_root --modes benchmark instrument run
+where linear_net.yaml is a user-specified configuration file you created above.
+It is recommended that you check if the port you want to use is available. You may check by:
+ +Example:
+ +Alternatively, you may check for all ports in use by sambatune_ui:
+ +If you need to free a port that you are finished with, you may use the kill command.
+If you followed the above directions, your artifact_root will be at ~/sambatune/artifact_root.
+Start the UI:
+It will tell you the username and password.
+NOTE: It is recommended to use a port other than 8576 in case someone else is using it. Select another port close to 8576.
+Next
+ +#TODOBRW
+sambatune_ui --directory ~/sambatune/artifact_root/sambatune_gen/ --port 8580
+sambatune_ui --directory /home/wilsonb/tmp/uno_test/artifact_root/sambatune_gen --port 8580
+username: "admin", password: "4f7cac2c-351e-11ed-93a3-f7ef9c6e5d46"
+username: "admin", password: "aaf1fc88-35c8-11ed-93a3-f7ef9c6e5d46"
+username: "admin", password: "bf64e4f8-3831-11ed-93a3-f7ef9c6e5d46"
+username: "admin", password: "8feca89e-384c-11ed-93a3-f7ef9c6e5d46"
+username: "admin", password: "355222d6-3a88-11ed-93a3-f7ef9c6e5d46"
+You will see something like:
+with the,
+ username: "admin", password: "05c63938-2941-11ed-93a3-f7ef9c6e5d46"
+[2022-08-31 15:24:36 +0000] [1344959] [Info] Starting gunicorn 20.1.0
+[2022-08-31 15:24:36 +0000] [1344959] [Info] Listening at: http://0.0.0.0:8576 (1344959)
+[2022-08-31 15:24:36 +0000] [1344959] [Info] Using worker: sync
+[2022-08-31 15:24:36 +0000] [1345092] [Info] Booting worker with pid: 1345092
+[2022-08-31 15:24:36 +0000] [1345093] [Info] Booting worker with pid: 1345093
+NOTE: Write down the username and password.
+NOTE: The password only works with this one instance of sambatune_ui. If you stop this instance of sambatune_ui and start another instance, it will have a new password.
+NOTE: You will need to
This describes the steps to set up port-forwarding for applications, +like SambaTune UI, which runs on the SambaNova system and binds to one or more ports. +This example uses 8576 and 18576 as port numbers. Using port numbers other than these may +avoid collisions with other users.
+This command sets up a port forward SambaNova login node to your local machine.
+Run
+ssh -N -f -L localhost:18576:localhost:18576 ALCFUserID@sambanova.alcf.anl.gov
+...
+Password: < MobilePass+ code >
+
+ssh ALCFUserID@sambanova.alcf.anl.gov
+#TODOBRW
+ssh -v -N -f -L localhost:8580:localhost:8580 wilsonb@sambanova.alcf.anl.gov
+ssh -N -f -L localhost:8580:localhost:8580 wilsonb@sambanova.alcf.anl.gov
+...
+Password: < MobilePass+ code >
+
+ssh wilsonb@sambanova.alcf.anl.gov
+replacing ALCFUserID with your ALCF User ID.
+This command sets up a port forward from a SambaNova node to the sambanova login machine.
+Below are the commands specific to sm-01. You may replace sm-01 with sm-02 when using that system.
+Run
+NOTE: The full name is sm-01.ai.alcf.anl.gov and it may also be used.
+ + +Then, navigate in your browser to, in this example, http://localhost:18576 on your local machine.
+Use the username and password from sm-01 to log in.
+Explanation of ssh command:
+-N : no remote commands
+
+-f : put ssh in the background
+
+-L <machine1>:<portA>:<machine2>:<portB> :
+
+The full command line will forward <machine1>:<portA> (local scope) to <machine2>:<portB> (remote scope)
+Adapted from: How can I run Tensorboard on a remote server?
+ + + + + + + + + + + + + + + +To create a virtual environment, one can use the --system-site-packages flag:
+ +Install packages in the normal manner such as:
+ +For more details see Use pip for installing.
+To install a different version of a package that is already installed in one's environment, one can use:
+ +Each of the samples or application examples provided by SambaNova has its own pre-built virtual environment which can be readily used. They are present in the /opt/sambaflow/apps/ directory tree within each of the applications.
++ + + + + + + + + + + + + + + +Note: Conda is not supported on the SambaNova system.
+