Skip to content

Latest commit

 

History

History
304 lines (195 loc) · 20.7 KB

README.md

File metadata and controls

304 lines (195 loc) · 20.7 KB
Raw

Proactive Python Client Examples

This repository contains a collection of example scripts that demonstrate various features of the proactive Python client. These scripts showcase how to create and manage jobs, tasks, and use advanced scheduling features in a Python environment.

Summary

Setting Up the Environment

Before running the examples, you may want to create a virtual environment to avoid conflicts with other Python projects. A virtual environment is a self-contained directory that contains a Python installation for a particular version of Python, plus a number of additional packages.

Creating a Virtual Environment (Optional)

On Mac and Linux

To create a virtual environment, run the following commands:

python3 -m venv env
source env/bin/activate

You can also use the provided Makefile:

make virtualenv

On Windows

On Windows, you can use the provided build.bat script to create and manage the virtual environment:

build.bat VIRTUAL_ENV

This command will create a new virtual environment or use the existing one based on your input.

Upgrading Required Packages

Once the virtual environment is activated, upgrade pip, setuptools, python-dotenv and humanize to the latest versions:

On Mac and Linux

python3 -m pip install --upgrade pip setuptools python-dotenv humanize

If you have used the provided Makefile, it automatically upgrades these packages when setting up the virtual environment.

On Windows

The build.bat script automatically upgrades these packages when setting up the virtual environment.

Installing the Proactive Python SDK

To install the latest pre-release or development version of the Proactive Python SDK, which includes the most recent features and fixes, use the following command:

python3 -m pip install --upgrade --pre proactive

The --pre flag is included to allow pip to find and install pre-releases.

Using Makefile (For Mac and Linux)

make install_latest

On Windows

build.bat INSTALL_LATEST

GitHub Codespaces (Linux)

To set up your environment in GitHub Codespaces for building and running the ProActive Python Client examples, follow these steps:

  1. Update the package list and install the Java 8 SDK:

    sudo apt update
sudo apt install openjdk-8-jdk

  2. Clone the jenv repository and set up jenv:

    git clone https://github.com/jenv/jenv.git ~/.jenv
echo 'export PATH="$HOME/.jenv/bin:$PATH"' >> ~/.bash_profile
echo 'eval "$(jenv init -)"' >> ~/.bash_profile
source ~/.bash_profile

  3. Add Java 8 to jenv and set it as the global version:

    jenv add /usr/lib/jvm/java-8-openjdk-amd64/jre/
jenv global 1.8
java -version

  4. Set up a virtual environment, install the Proactive Python SDK, and run the provided examples:

    # Create a virtual environment
make virtualenv

# Install the latest pre-release or development version of the Proactive Python SDK
make install_latest

# To run all the provided example scripts
make run_all

# Or activate the environment and test a specific script, e.g., 'demo_basic.py'
source env/bin/activate
python3 demo_basic.py

By following these steps, you'll have the Java 8 SDK installed and configured with jenv. Additionally, you'll be able to install the ProActive Python Client package and execute the provided examples within GitHub Codespaces.

Running the Examples

To test if the installation was successful and see the proactive client in action, you can run the provided example scripts. For instance, to run the demo_basic.py script:

python3 demo_basic.py

Using Makefile (For Mac and Linux)

For instance, to run all the provided example scripts:

make run_all

On Windows

To run all the provided example scripts:

build.bat RUN_ALL

This command will execute all .py files in the current directory.

Examples Description

  • demo_basic.py: A simple script that showcases how to connect to the ProActive Scheduler, create a job and a Python task, and execute it.

  • demo_pre_post_script.py: Illustrates adding pre-scripts and post-scripts to tasks for setup and cleanup. Enhances task flexibility by allowing pre and post execution actions. Ideal for complex setups or post-task data handling.

  • demo_job_task_var.py: Demonstrates the use of job and task-level variables within the ProActive Scheduler. This script highlights how to define and access variables that can be shared across tasks within a job, facilitating dynamic task configuration and the passing of data between tasks. It showcases setting variables at both the job and individual task levels, and accessing these variables within task implementations for flexible, dynamic workflow execution.

  • demo_global_var.py: Demonstrates the use of global variables between tasks in ProActive Scheduler. It exemplifies creating and managing jobs and tasks with a focus on inter-task communication via global variables.

  • demo_task_result.py: Highlights job submission with task dependencies and result handling, showing how to create tasks that process and utilize the results of preceding tasks.

  • demo_task_dependency.py: Illustrates how to manage task dependencies within the ProActive Scheduler. It demonstrates creating a series of tasks with specific execution orders, showcasing the scheduler's capability to handle complex dependencies. This example provides insights into structuring a job where tasks execute sequentially or in parallel, based on their dependencies, ensuring that dependent tasks wait for their predecessors to complete before starting.

  • demo_multilanguage_job.py: Showcases the creation and submission of a multi-language job (Python and Groovy tasks), emphasizing the scheduler's support for diverse programming languages within a single job.

  • demo_job_status.py: Demonstrates the process of job submission and monitoring with the ProActive Python SDK. It walks through creating a job, adding a Python task, submitting the job to the ProActive Scheduler, and monitoring its status until completion, concluding with the retrieval and display of the job's output.

  • demo_impl_file.py: Demonstrates the basic usage of the ProActive Scheduler for executing a Python task implemented in an external file.

  • demo_impl_url.py: Illustrates how to set up a machine learning task (Logistic Regression) using a script sourced from a URL, showing the ProActive Scheduler's capability to incorporate external scripts.

  • demo_exec_file.py: Demonstrates how to execute a Python task using a script sourced from an external file. This script is designed to illustrate the process of configuring and submitting a ProActive Scheduler task where the task's implementation is defined in an external Python file, showcasing how to incorporate external Python scripts into ProActive Scheduler jobs for flexible and modular task execution.

  • demo_dataspace_api.py: Demonstrates managing data transfers between local spaces and the ProActive Scheduler's data spaces (user and global), using the ProActive Python Client.

  • demo_transf_file.py: Demonstrates file handling in ProActive tasks, focusing on including input files for processing and designating output files to capture task results. This script counts the number of files in a specified directory and outputs the count both to the terminal and to a file within the same directory, showcasing data transfer and file management in ProActive workflows.

  • demo_python_module.py: Demonstrates how to execute Python tasks that depend on custom Python modules or external libraries, demonstrating the ProActive Scheduler's support for complex Python environments and dependencies.

  • demo_forkenv.py: Shows how to create and execute a Python task within a job, with emphasis on configuring the execution environment using a fork environment script.

  • demo_runtimeenv.py: Demonstrates the process of executing a Python task within the ProActive Scheduler, emphasizing the configurability of the task's execution environment. It demonstrates establishing a connection to the scheduler, creating a job, and configuring a Python task to run within a specified container environment. This example highlights the scheduler's flexibility in executing tasks in varied and complex runtime settings.

  • demo_selectionscript.py: This script explains how to create a job and a task, and ensures that the task is executed only on Linux machines using a selection script.

  • demo_loop.py: Illustrates loop control flow within a ProActive Scheduler job, demonstrating how to create loops within jobs for repeated task execution based on dynamic conditions.

  • demo_replicate.py: Demonstrates replication and merge flow with Python Tasks. This script showcases the creation and execution of a ProActive Job designed to demonstrate replication and merge control flows.

  • demo_branch.py: Demonstrates branching logic within ProActive Scheduler jobs, illustrating how to implement conditional task execution paths based on dynamic criteria.

  • demo_3controls.py: Combines replication, iteration, and branching for advanced job workflows. Demonstrates the scheduler's capability for sophisticated task management. Enables complex computational workflows with parallel execution and conditional logic.

  • demo_scheduler_api.py: This example demonstrates the basic usage of the ProActive Scheduler API within a Python environment. It covers establishing a connection to the ProActive Scheduler, creating and configuring a job with multiple tasks, and managing the submission and monitoring of the job through the scheduler's API. The example is structured to highlight key API functions such as job creation, task addition, job submission, and retrieval of job results, providing a comprehensive overview of interacting with the ProActive Scheduler programmatically.

  • demo_resource_manager_api.py: Illustrates the interaction with the ProActive Resource Manager (RM) API for managing computational resources within ProActive Scheduler jobs. The script showcases how to use the RM API to query the state of resources, manage nodes and node sources, and perform operations like adding or removing nodes from the resource manager. This example serves as a practical guide for developers looking to programmatically manage and optimize the allocation of computational resources for their tasks and jobs within the ProActive Scheduler.

  • demo_synchronization_api.py: Provides an example of utilizing the ProActive Synchronization API to manage dependencies and synchronize the execution of tasks within a ProActive Scheduler job. The script demonstrates how to establish synchronization channels, set and wait for conditions, and use synchronization locks to coordinate the execution flow of multiple tasks. This example is essential for users needing to ensure ordered execution and dependency management among tasks, highlighting the API's capabilities in facilitating complex workflow orchestration.

  • demo_ai_workflow.py: Sets up and executes a machine learning workflow for the Iris dataset using the ProActive Scheduler. It demonstrates how to create a job, add tasks, and manage their execution sequence.

  • demo_job_results.py: Demonstrates the end-to-end process of job submission, task dependency management, and result retrieval using the ProActive Python SDK. It covers creating a ProActive job, adding multiple Python tasks with dependencies, submitting the job to the ProActive Scheduler, and retrieving the job's results.

  • demo_signal_send.py: Shows how to send a signal to a specified job with or without variables using the ProActive Signal API.

  • demo_signal_wait.py: Demonstrates the usage of the ProActive Scheduler Signal API to receive and handle signals within tasks.

  • demo_signal_wait2.py: Extends the signal waiting demonstration with additional tasks to showcase more complex signal handling scenarios.

  • demo_continual_learning.py: Illustrates how to implement a continual learning workflow using the ProActive Scheduler, demonstrating data generation, incremental learning, and model updating over multiple iterations.

  • demo_virtualenv.py: Shows how to create and execute a Python task within a custom virtual environment, demonstrating the ProActive Scheduler's support for isolated Python environments.

  • demo_dockerfile.py: Demonstrates how to submit a Dockerfile task to the ProActive Scheduler, showcasing the integration of Docker-based workflows within the ProActive environment.

Additional scripts found in the demo_ai_workflows directory showcase various machine learning workflows, leveraging the ProActive Scheduler for tasks like data preprocessing, model training, evaluation, and prediction across different datasets and using various algorithms.

Please ensure the ProActive Scheduler is running and accessible, and that you have the required scripts and environments set up before executing these examples.

Leveraging Pre-built AI Tasks from Proactive AI Orchestration

This section of the repository showcases advanced examples that leverage the powerful capabilities of the Proactive AI Orchestration platform, specifically utilizing tasks from the ai-machine-learning bucket. The ai-machine-learning bucket is a comprehensive collection of generic machine learning tasks, designed to facilitate the seamless composition of workflows for the learning and testing of predictive models. These tasks are highly versatile and can be tailored to meet specific requirements, enabling users to effortlessly integrate and execute sophisticated machine learning models and workflows.

Each example demonstrates the practical application of these pre-built tasks, offering insights into how they can be customized and combined to create complex machine learning solutions. Whether you're looking to add new tasks or modify existing ones, the ai-machine-learning bucket provides a flexible framework that supports a wide range of machine learning activities, from data preprocessing and model training to evaluation and inference.

For detailed information on how to customize the ai-machine-learning bucket to suit your unique needs, and to explore the full range of tasks and capabilities it offers, please refer to the Proactive AI Orchestration User Guide.

Machine Learning Workflows

  • Diabetics Detection: Utilizes various machine learning algorithms to detect diabetic patients from clinical datasets.

    • python -m demo_ai_workflows.Diabetics_Detection_Using_AutoSklearn_Classifier: Detects diabetes using the Auto-sklearn Classifier.
    • python -m demo_ai_workflows.Diabetics_Detection_Using_Isolation_Forest: Applies Isolation Forest for anomaly detection in diabetes data.
    • python -m demo_ai_workflows.Diabetics_Detection_Using_K_Means: Employs K-Means clustering for diabetes data segmentation.
    • python -m demo_ai_workflows.Diabetics_Detection_Using_Mean_Shift: Uses Mean Shift clustering for diabetes data analysis.
    • python -m demo_ai_workflows.Diabetics_Detection_Using_One_Class_SVM: Implements One-Class SVM for outlier detection in diabetes datasets.
    • python -m demo_ai_workflows.Diabetics_Detection_Using_TPOT_Classifier: Leverages the TPOT Classifier for automated machine learning in diabetes detection.

  • House Price Prediction: Demonstrates the use of various regression models to predict house prices based on the Boston housing dataset.

    • python -m demo_ai_workflows.House_Price_Prediction_Using_AutoSklearn_Regressor: Predicts house prices using Auto-sklearn Regressor.
    • python -m demo_ai_workflows.House_Price_Prediction_Using_Bayesian_Ridge_Regression: Applies Bayesian Ridge Regression for house price prediction.
    • python -m demo_ai_workflows.House_Price_Prediction_Using_Linear_Regression: Utilizes Linear Regression for predicting house prices.
    • python -m demo_ai_workflows.House_Price_Prediction_Using_Support_Vector_Regression: Employs Support Vector Regression for house price estimation.
    • python -m demo_ai_workflows.House_Price_Prediction_Using_TPOT_Regressor: Uses the TPOT Regressor for automated machine learning in house price prediction.

  • Iris Flowers Classification: Showcases classification models for identifying Iris flower species from the Iris dataset.

    • python -m demo_ai_workflows.Iris_Flowers_Classification_Using_AdaBoost: Classifies Iris species using AdaBoost.
    • python -m demo_ai_workflows.Iris_Flowers_Classification_Using_CatBoost: Implements CatBoost for Iris species classification.
    • python -m demo_ai_workflows.Iris_Flowers_Classification_Using_Gaussian_Naive_Bayes: Utilizes Gaussian Naive Bayes for classifying Iris species.
    • python -m demo_ai_workflows.Iris_Flowers_Classification_Using_Gradient_Boosting: Applies Gradient Boosting for Iris species classification.
    • python -m demo_ai_workflows.Iris_Flowers_Classification_Using_Logistic_Regression: Uses Logistic Regression for Iris species classification.
    • python -m demo_ai_workflows.Iris_Flowers_Classification_Using_Random_Forest: Employs Random Forest for classifying Iris species.
    • python -m demo_ai_workflows.Iris_Flowers_Classification_Using_Support_Vector_Machines: Demonstrates SVM usage for Iris species classification.
    • python -m demo_ai_workflows.Iris_Flowers_Classification_Using_XGBoost: Leverages XGBoost for Iris species classification.

Advanced Examples

The repository also includes a collection of advanced examples that demonstrate more complex use cases of the ProActive Scheduler, particularly focusing on machine learning workflows. These examples illustrate how to set up, configure, and execute sophisticated machine learning models and workflows using the ProActive Python client.

CIFAR-10 Logistic Regression Classifier

The CIFAR-10 Logistic Regression Classifier example demonstrates how to train and evaluate a logistic regression model on the CIFAR-10 dataset, which is a common benchmark in machine learning for image classification tasks.

  • Repository Contents: The repository includes:

  • Scripts for training and evaluating the model (train.py and eval.py).

  • A directory structure to organize models and predictions.

  • A sample script (submit2proactive.py) to submit a job to the ProActive Scheduler, showing how to run complex machine learning workflows on distributed computing environments.

  • Getting Started: To explore this advanced example, visit the repository and follow the instructions provided in its README to set up your environment, train the model, and submit the job to ProActive.

Please ensure that your environment meets the prerequisites and that the ProActive Scheduler is running and accessible before you begin.

Additional Information

  • Ensure you have Python 3.6 or later to use the proactive client.
  • If you encounter any issues, please report them on the Issues page.
  • For more information about the proactive client, refer to the official documentation.

Contributing

We welcome contributions to this repository. If you have an improvement or a new example, please fork the repository, make your changes, and submit a pull request.