Skip to content

AutotestServerInstall

Jump to bottom Edit New page
Lucas Meneghel Rodrigues edited this page Jul 4, 2012 · 8 revisions

Installing an Autotest server (Ubuntu version)

Install script

We have developed a script to automate the steps described below on a Ubuntu 12.04 server. On an autotest source clone, this script is under contrib/install-autotest-server.sh:

https://github.com/autotest/autotest/blob/master/contrib/install-autotest-server.sh

If you want to download it straight from github, just

curl https://raw.github.com/autotest/autotest/master/contrib/install-autotest-server.sh > install-autotest-server.sh

Then make it executable and execute it:

chmod +x install-autotest-server.sh
./install-autotest-server.sh

Server Installation Steps

Install utility packages:

apt-get install -y unzip wget gnuplot makepasswd

Install webserver related packages (and Django):

apt-get install -y apache2-mpm-prefork libapache2-mod-wsgi python-django

Install database related packages:

apt-get install -y mysql-server python-mysqldb

Install java in order to compile the web interface, and git for cloning the autotest source code repository:

apt-get install git openjdk-7-jre-headless

Also, you'll need to install a bunch of auxiliary external packages

apt-get install python-imaging python-crypto python-paramiko python-httplib2 python-numpy python-matplotlib python-setuptools python-simplejson

You will create a special user for running autotest jobs.

sudo adduser autotest
cd /usr/local/
sudo mkdir autotest
sudo chown autotest:autotest autotest
su autotest

Make sure that the user Autotest runs as has passwordless ssh into root of any machines under test. There are many good guides here: Google search

Clone the full Autotest repository:

git clone git://github.com/autotest/autotest.git /usr/local/autotest

Set up MySQL. The following commands will set up a read-only user called nobody and a user with full permissions called autotest:

mysql -u root -p
create database autotest_web;
grant all privileges on autotest_web.* TO 'autotest'@'localhost' identified by 'some_password';
grant SELECT on autotest_web.* TO 'nobody'@'%';
grant SELECT on autotest_web.* TO 'nobody'@'localhost';

If you use safesync for migrating the databases you will want to grant access to the test database.

GRANT ALL ON test_autotest_web.* TO 'autotest'@'localhost' identified by 'some_password';

If you want mysql available to hosts other than the localhost, you'll then want to comment out the bind-address = 127.0.0.1 line in the /etc/mysql/my.cnf.

In addition, you may want to increase the set-variable = max_connections to something like 6000, if you're running on a substantial server. If you experience scalability issues, you may want to log slow queries for debugging purposes. This is done with the following lines:

log_slow_queries = /var/log/mysql/mysql-slow.log  # Log location
long_query_time = 30  # Time in seconds before we consider it slow

Note: Advanced setups may wish to use MySQL Replication

Run the build script to install necessary external packages (You will need unzip to be able to use setuptools). Currently, the packages it installs are setuptools, mysql-python, Django, numpy, matplotlib, paramiko, and GWT:

/usr/local/autotest/utils/build_externals.py

Always re-run this after a git pull if you notice it has changed, new dependencies may have been added. This is safe to rerun as many times as you want. It will only fetch and build what it doesn't already have.

NOTE: Set the HTTP_PROXY environment variable to http://proxy:3128/ before running the above if your site requires a proxy to fetch urls.

Setup Apache config files. If the only thing you want to do with Apache is run Autotest, you can use the premade Apache conf:

sudo rm /etc/apache2/sites-enabled/000-default
sudo ln -s /usr/local/autotest/apache/conf/apache-conf /etc/apache2/sites-enabled/001-autotest

Also, you'll need to enable rewrite mod rules, which you can do by

a2enmod rewrite

If you want to do other things on the Apache server as well, you'll need to insert the following line into your Apache conf, under the appropriate VirtualHost section:

Include "/usr/local/autotest/apache/conf/all-directives"

Note: You will have to enable mod_env on SuSE based distro's for the all-directives to load properly when apache is started.

Next, take a look in your /etc/apache2/apache2.conf for Debian based distros, /etc/apache2/mod_autoindex-defaults.conf for SuSE based distros or for newer distros it may also be in /etc/apache2/mods-available/autoindex.conf. If the following line exists, comment it out; it's a broken "feature" of Apache that can't be overriden:

IndexIgnore .??* *~ *# HEADER* RCS CVS *,t

In the same file, either add the following line or un-comment it out if it already exists:

AddHandler cgi-script .cgi

Enabled mod_headers for Apache

sudo a2enmod headers

Due to an incompatibility between versions of mod_wsgi between Fedora and Ubuntu, you'll have to comment the following line in /usr/local/autotest/apache/conf/django-directives:

WSGISocketPrefix run/wsgi

So, just comment this line and you'll be good. Now, edit global_config.ini to match the passwords you set earlier. Run DB migrations to set up DB schema and initial data:

database/migrate.py sync

If you have an existing database with data in it, you may want to use "safesync" instead of "sync". This will copy your data to a fresh DB and run the migration on that first, to ensure no errors occur during migration that could leave your database in an inconsistent state.

Compile the Autotest frontend web client and the new TKO client:

/usr/local/autotest/utils/compile_gwt_clients.py -a

You will need to re-compile after any changes/syncs of the frontend/client pages.

Run Django's syncdb.

/usr/local/autotest/frontend/manage.py syncdb
# you may have to run this twice, like if the debug_user does not have access to the Django Admin interface
/usr/local/autotest/frontend/manage.py syncdb

Make everything in the /usr/local/autotest directory world-readable, for Apache's sake:

chmod -R o+r /usr/local/autotest
find /usr/local/autotest/ -type d | xargs chmod o+x
chmod o+x /usr/local/autotest/tko/*.cgi

Restart apache

sudo apache2ctl restart

Check that the web frontend is working by browsing to http://localhost/afe/server/rpc_doc If you see an error or exception traceback, you need to go back and fix something.

Once that works, you should be able to access the web frontend at http://localhost/afe/.

Add client and server side tests to the frontend using utils/test_importer.py

/usr/local/autotest/utils/test_importer.py -A

To start the scheduler, you can run something like this (as user autotest):

/usr/local/autotest/scheduler/monitor_db.py /usr/local/autotest/results

To start the scheduler on reboot, you can setup init.d.

sudo cp /usr/local/autotest/utils/autotest.init /etc/init.d/
sudo update-rc.d /usr/local/autotest.init defaults

Then, you can reboot and you will autotest-scheduler-watcher and autotest-scheduler processess running.

Adding some initial frontend objects

Once you've got the frontend working, you'll want to add some objects so you can schedule some jobs.

  • Click the "Admin interface" link at the upper-right corner of the frontend.

  • Click "Labels" and then "Add label".

  • Type a name for the label, and check the "Platform" box, then click "Save".

  • Click "Home" at the upper-left corner, and click "Hosts" and then "Add host".

  • Type in a hostname, add your platform to the "chosen labels", and click "Save". If you want your job to succeed, you'll need to set things up so that the autotest user can ssh to the hostname you provide without needing a password.

  • Click "Home", and click "Tests" and then "Add test".

  • Fill in the following information:

    • Name: sleeptest
    • Test class: Kernel
    • Test type: Client
    • Path: client/tests/sleeptest/control

    Click "Save".

  • Now click "Go to Frontend" at the upper-right corner.

You should now be able to create a job on the "Create Job" tab, submit it, and see it in the "Job List" and "View Job" tabs. You can also see the status of your host on the "Hosts" tab.

Additional Tips

  • Some really old distributions have similarly ancient SSH implementations. If so, it can take hours for a SSH connection to timeout if a machine dies. You can modify your /etc/sysctl.conf to contain the line net.ipv4.tcp_keepalive_time = 60 to reduce this time to around 10 minutes. This is typically unnecessary, though.
  • You'll probably want to back up your results filesystem and mysql db on a regular basis. A cron job is a simple way to accomplish this.

See also:

Add a custom footer
Add a custom sidebar
Clone this wiki locally