Announcing the release of TestGrid 1.0

We are pleased to announce the release of TestGrid 1.0

TestGrid provides enterprise customers with the confidence that the products and fixes they get from WSO2 are tested against all product scenarios against a wide-array of infrastructures and deployments, hence is production-ready.

We do that by running scenario tests against product deployments on numerous infrastructures. Infrastructures include public cloud providers, operating systems, databases, jdk and so on. For ease of modeling, we think that the underlying infrastructure, deployment patterns, and tests as three orthogonal concepts.

This means same test script can be used across multiple product deployments irrespective of whether the deployment is done on Linux or Windows, database is mysql or something else etc. We also think that integration test scripts are independent of the product deployment patterns. For example, a API-M single node deployment and clustered deployment should provide the same functionality, hence the functional tests does not differ between different patterns.

This is summarized here:

• Provision an infrastructure on AWS and other infrastructure providers via Infrastructure-as-Code (IAC) scripts. WSO2 internally use Testgrid with WSO2 written IaC scripts for this purpose, since it allows to validate production readiness of the scripts as well.

• Do WSO2 product deployments on top the provisioned infrastructure. We do that by running Configuration -Management scripts (Puppet/Ansible) provided for WSO2 products.

• Run JMeter/Selenium/TestNG scenarios against the deployment to certify the created deployment pattern against the given infrastructure combination.

• In the end, Testgrid generates a report that shows the outcome of the scenario tests. All is good if it is green; start worrying if it is red! :)

With this release, TestGrid has following features & improvements:

• Support to provision infrastructures for these;
  -- Cloud provider: AWS, Kubernetes (via AWS EKS)
  -- OS : CentOS 7.5, Redhat 7.5, Ubuntu 18.04, Windows 2016
  -- JDK : Oracle JDK 8, Open JDK 8
  -- DB : MySQL 5.7, 5.6, Oracle 11g, 12c, SQLServer, Postgres

• Specialized support to provision infrastructures with Cloudformation on AWS cloud. This allows you to run tests for parallel deployments using same cloudformation script content. It also provides a AWS resource manager that understands the resource limits of your AWS acct. Finally, it can look up AMI that matches the operating-system you need.

• Specialized support for JMeter and TestNG result parsing

• Build failure email alerts, and a dashboard that shows the test results with download links to relevant logs

• Real-time slack notifications for individual test plan failures

Known Issues: https://github.com/wso2/testgrid/issues

---

A Testgrid deployment is currently run internally at WSO2 to verify wum-based updates and product releases it ship. It is also available as a private beta for WSO2 customers.

We'd like to hear from you. Contact us: [email protected], [email protected]

We are pleased to announce the release of TestGrid Milestone 37

We are pleased to announce the release of TestGrid Milestone 34

TestGrid provides enterprise customers with the confidence that the products and fixes they get from WSO2 are tested against all product scenarios against a wide-array of infrastructures and deployments, hence is production-ready.

Here’s what really happens in TestGrid:

• Provision an infrastructure on AWS and other infrastructure providers via Infrastructure as Code (IAC) scripts. We use the IAC scripts provided by WSO2 for this purpose.

• Do WSO2 product deployments on top the provisioned infrastructure. We do that by running Configuration -Management scripts (Puppet/Ansible) provided for WSO2 products.

• Run JMeter/Selenium/TestNG scenarios against the deployment to certify the created deployment pattern against the given infrastructure combination.

• In the end, Testgrid generates a report that shows the outcome of the scenario tests. All is good if it is green; start worrying if it is red! :)

With this release, TestGrid has following features & improvements:

• Run WSO2-IS- integration tests via TestGrid for combinations of;
  -- OS: CentOS
  -- JDK: Oracle JDK
  -- DB: MySQL

• Improved email template by adding summarize graphs and tables.

• Test execution history graph: This graph represents the summary of previously executed build jobs.

• Last build summary graph: This graph represents the summary of the last build.

• Test failure summary table: This table represents the failed test cases against infra combinations.

• Bug fixes and improvements

• Real-time slack notifications for individual test plan failures

List of resolved features and issues can be tracked at :
https://github.com/wso2/testgrid/milestone/36

TestGrid has been tested against following infrastructures:

• Cloud Provider: AWS and K8S-on-Openstack
• Spawn instances with AMI images in any AWS region
• Any database type AWS RDS supports (MySQL, MariaDB, PostgreSQL, SQL Server and Oracle)

Known Issues: https://github.com/wso2/testgrid/issues

We are pleased to announce the release of TestGrid Milestone 34

We are pleased to announce the release of TestGrid Milestone 34

TestGrid provides enterprise customers with the confidence that the products and fixes they get from WSO2 are tested against all product scenarios against a wide-array of infrastructures and deployments, hence is production-ready.

Here’s what really happens in TestGrid:

• Provision an infrastructure on AWS and other infrastructure providers via Infrastructure as Code (IAC) scripts. We use the IAC scripts provided by WSO2 for this purpose.

• Do WSO2 product deployments on top the provisioned infrastructure. We do that by running Configuration -Management scripts (Puppet/Ansible) provided for WSO2 products.

• Run JMeter/Selenium/TestNG scenarios against the deployment to certify the created deployment pattern against the given infrastructure combination.

• In the end, Testgrid generates a report that shows the outcome of the scenario tests. All is good if it is green; start worrying if it is red! :)

With this release, TestGrid has following features & improvements:

• Introducing Testgrid debug mode.
• Parallel execution of test plans.
• Parallelized CloudFormation resource creations
• CentOS + Oracle, CentOS + MYSQL, CentOS + MSSQL support for integration tests.

List of resolved features and issues can be tracked at :
https://github.com/wso2/testgrid/milestone/34

TestGrid has been tested against following infrastructures:

• Cloud Provider: AWS and K8S-on-Openstack
• Spawn instances with AMI images in any AWS region
• Any database type AWS RDS supports (MySQL, MariaDB, PostgreSQL, SQL Server and Oracle)

Known Issues: https://github.com/wso2/testgrid/issues

TestGrid 0.9.0-m33

We are pleased to announce the release of TestGrid Milestone 33

TestGrid provides enterprise customers with the confidence that the products and fixes they get from WSO2 are tested against all product scenarios against a wide-array of infrastructures and deployments, hence is production-ready.

Here’s what really happens in TestGrid:

• Provision an infrastructure on AWS and other infrastructure providers via Infrastructure as Code (IAC) scripts. We use the IAC scripts provided by WSO2 for this purpose.

• Do WSO2 product deployments on top the provisioned infrastructure. We do that by running Configuration -Management scripts (Puppet/Ansible) provided for WSO2 products.

• Run JMeter/Selenium/TestNG scenarios against the deployment to certify the created deployment pattern against the given infrastructure combination.

• In the end, Testgrid generates a report that shows the outcome of the scenario tests. All is good if it is green; start worrying if it is red! :)

With this release, TestGrid has following features & improvements:

• Deployment tinkerer can download the product logs, and artifacts after scenario testing and integration testing.
• Added deployment tinkerer agent support for CentOS.
• Added feature to analyze surefire reports and extract status for integration tests.
• Added deployment setup scripts for APIM 2.x integration test jobs.
• Added email-able report generation for integration tests.

List of resolved features and issues can be tracked at :
https://github.com/wso2/testgrid/milestone/33

TestGrid has been tested against following infrastructures:

• Cloud Provider: AWS and K8S-on-Openstack
• Spawn instances with AMI images in any AWS region
• Any database type AWS RDS supports (MySQL, MariaDB, PostgreSQL, SQL Server and Oracle)

Known Issues: https://github.com/wso2/testgrid/issues

TestGrid 0.9.0-m32

We are pleased to announce the release of TestGrid Milestone 32

TestGrid provides enterprise customers with the confidence that the products and fixes they get from WSO2 are tested against all product scenarios against a wide-array of infrastructures and deployments, hence is production-ready.

Here’s what really happens in TestGrid:

• Provision an infrastructure on AWS and other infrastructure providers via Infrastructure as Code (IAC) scripts. We use the IAC scripts provided by WSO2 for this purpose.

• Do WSO2 product deployments on top the provisioned infrastructure. We do that by running Configuration -Management scripts (Puppet/Ansible) provided for WSO2 products.

• Run JMeter/Selenium/TestNG scenarios against the deployment to certify the created deployment pattern against the given infrastructure combination.

• In the end, Testgrid generates a report that shows the outcome of the scenario tests. All is good if it is green; start worrying if it is red! :)

With this release, TestGrid has following features & improvements:

• Provides AWS infrastructure provisioning scripts for APIM integration test jobs.
• Ability to process multiple JTL files if exist in a test scenario.
• Accommodate run-scenario.sh script for pre, post script execution.

List of resolved features and issues can be tracked at :
https://github.com/wso2/testgrid/milestone/31

TestGrid has been tested against following infrastructures:

• Cloud Provider: AWS and K8S-on-Openstack
• Spawn instances with AMI images in any AWS region
• Any database type AWS RDS supports (MySQL, MariaDB, PostgreSQL, SQL Server and Oracle)

Known Issues: https://github.com/wso2/testgrid/issues

TestGrid 0.9.0-m31

We are pleased to announce the release of TestGrid Milestone 31

TestGrid provides enterprise customers with the confidence that the products and fixes they get from WSO2 are tested against all product scenarios against a wide-array of infrastructures and deployments, hence is production-ready.

Here’s what really happens in TestGrid:

• Provision an infrastructure on AWS and other infrastructure providers via Infrastructure as Code (IAC) scripts. We use the IAC scripts provided by WSO2 for this purpose.

• Do WSO2 product deployments on top the provisioned infrastructure. We do that by running Configuration -Management scripts (Puppet/Ansible) provided for WSO2 products.

• Run JMeter/Selenium/TestNG scenarios against the deployment to certify the created deployment pattern against the given infrastructure combination.

• In the end, Testgrid generates a report that shows the outcome of the scenario tests. All is good if it is green; start worrying if it is red! :)

With this release, we provide the following,

• Implement Performance testing report generation
• Ability to monitor the last success time of the product build
• Testgrid-prod-deploy job should first delete the old wso2-testgrid before unpacking the dist
• The build jobs should use existing testgrid distribution
• Login issue in TestGrid-prod
• Persist scenario test results/artifacts in S3 in the correct build directory

List of resolved features and issues can be tracked at :
https://github.com/wso2/testgrid/milestone/30

TestGrid has been tested against following infrastructures:

• Cloud Provider: AWS and K8S-on-Openstack
• Spawn instances with AMI images in any AWS region
• Any database type AWS RDS supports (MySQL, MariaDB, PostgreSQL, SQL Server and Oracle)

Known Issues: https://github.com/wso2/testgrid/issues

TestGrid 0.9.0-m30

We are pleased to announce the release of TestGrid Milestone 30

TestGrid provides enterprise customers with the confidence that the products and fixes they get from WSO2 are tested against all product scenarios against a wide-array of infrastructures and deployments, hence is production-ready.

Here’s what really happens in TestGrid:

• Provision an infrastructure on AWS and other infrastructure providers via Infrastructure as Code (IAC) scripts. We use the IAC scripts provided by WSO2 for this purpose.

• Do WSO2 product deployments on top the provisioned infrastructure. We do that by running Configuration -Management scripts (Puppet/Ansible) provided for WSO2 products.

• Run JMeter/Selenium/TestNG scenarios against the deployment to certify the created deployment pattern against the given infrastructure combination.

• In the end, Testgrid generates a report that shows the outcome of the scenario tests. All is good if it is green; start worrying if it is red! :)

With this release, we provide the following,

• Fixed inconsistent navigation issues in the dashboard
• Improve log file storing structure on Amazon S3 bucket
• Fix log downloading issue
• Dashboard Improvements with new color palette and icons
• Allow using direct-links by non-logged-in users.

List of resolved features and issues can be tracked at :
https://github.com/wso2/testgrid/milestone/24

TestGrid has been tested against following infrastructures:

• Cloud Provider: AWS and K8S-on-Openstack
• Spawn instances with AMI images in any AWS region
• Any database type AWS RDS supports (MySQL, MariaDB, PostgreSQL, SQL Server and Oracle)

Known Issues: https://github.com/wso2/testgrid/issues

TestGrid 0.9.0-m29

We are pleased to announce the release of TestGrid Milestone 29

TestGrid provides enterprise customers with the confidence that the products and fixes they get from WSO2 are tested against all product scenarios against a wide-array of infrastructures and deployments, hence is production-ready.

Here’s what really happens in TestGrid:

• Provision an infrastructure on AWS and other infrastructure providers via Infrastructure as Code (IAC) scripts. We use the IAC scripts provided by WSO2 for this purpose.

• Do WSO2 product deployments on top the provisioned infrastructure. We do that by running Configuration -Management scripts (Puppet/Ansible) provided for WSO2 products.

• Run JMeter/Selenium/TestNG scenarios against the deployment to certify the created deployment pattern against the given infrastructure combination.

• In the end, Testgrid generates a report that shows the outcome of the scenario tests. All is good if it is green; start worrying if it is red! :)

With this release, we provide the following,

• Improved IS test success rate by fixing artifacts deployment issue.
• Automate the run-scenario script generation.
• Bug fixes and improvements.

List of resolved features and issues can be tracked at :
https://github.com/wso2/testgrid/milestone/29

TestGrid has been tested against following infrastructures:

• Cloud Provider: AWS and K8S-on-Openstack
• Spawn instances with AMI images in any AWS region
• Any database type AWS RDS supports (MySQL, MariaDB, PostgreSQL, SQL Server and Oracle)

Known Issues: https://github.com/wso2/testgrid/issues

TestGrid 0.9.0-m28

We are pleased to announce the release of TestGrid Milestone 28

TestGrid provides enterprise customers with the confidence that the products and fixes they get from WSO2 are tested against all product scenarios against a wide-array of infrastructures and deployments, hence is production-ready.

Here’s what really happens in TestGrid:

• Provision an infrastructure on AWS and other infrastructure providers via Infrastructure as Code (IAC) scripts. We use the IAC scripts provided by WSO2 for this purpose.

• Do WSO2 product deployments on top the provisioned infrastructure. We do that by running Configuration -Management scripts (Puppet/Ansible) provided for WSO2 products.

• Run JMeter/Selenium/TestNG scenarios against the deployment to certify the created deployment pattern against the given infrastructure combination.

• In the end, Testgrid generates a report that shows the outcome of the scenario tests. All is good if it is green; start worrying if it is red! :)

With this release, we provide the following,

• Per-job view: Filter the infrastructure combinations by a given infrastructure
• Generate infra combination only for product job supporting infra
• Log error if scenario run script is not found
• Support reading testgrid.yaml as both hidden and non-hidden files
• Improved report downloading feature
• Bug fixes and improvements

List of resolved features and issues can be tracked at :
https://github.com/wso2/testgrid/milestone/28

TestGrid has been tested against following infrastructures:

• Cloud Provider: AWS and K8S-on-Openstack
• Spawn instances with AMI images in any AWS region
• Any database type AWS RDS supports (MySQL, MariaDB, PostgreSQL, SQL Server and Oracle)

Known Issues: https://github.com/wso2/testgrid/issues

TestGrid 0.9.0-m27

We are pleased to announce the release of TestGrid Milestone 27

TestGrid provides enterprise customers with the confidence that the products and fixes they get from WSO2 are tested against all product scenarios against a wide-array of infrastructures and deployments, hence is production-ready.

Here’s what really happens in TestGrid:

• Provision an infrastructure on AWS and other infrastructure providers via Infrastructure as Code (IAC) scripts. We use the IAC scripts provided by WSO2 for this purpose.

• Do WSO2 product deployments on top the provisioned infrastructure. We do that by running Configuration -Management scripts (Puppet/Ansible) provided for WSO2 products.

• Run JMeter/Selenium/TestNG scenarios against the deployment to certify the created deployment pattern against the given infrastructure combination.

• In the end, Testgrid generates a report that shows the outcome of the scenario tests. All is good if it is green; start worrying if it is red! :)

With this release, we provide the following,

• Implementation of initial PoC to execute custom commands inside deployment's VMs
• Support execution of init and cleanup scripts to provision infrastructure required for samples
• Bug fixes and improvements

List of resolved features and issues can be tracked at :
https://github.com/wso2/testgrid/milestone/27

TestGrid has been tested against following infrastructures:

• Cloud Provider: AWS and K8S-on-Openstack
• Spawn instances with AMI images in any AWS region
• Any database type AWS RDS supports (MySQL, MariaDB, PostgreSQL, SQL Server and Oracle)

Known Issues: https://github.com/wso2/testgrid/issues