Skip to content

Home

Jump to bottom
Varun Shankar S edited this page Aug 2, 2016 · 4 revisions

#GSoC project

##Current Week https://github.com/varunshankar/galaxy-godocker_tool/wiki

##Past Weeks

###Week 11[July 25,2016 - July 31,2016] ####Plan

  • Prepare for pull request.
  • Update the doc in wiki.
  • Work on godocker_tool enchancement.

####Progress ####Monday July 25,2016

  • [Submitted] Pull request.

####Tuesday July 26,2016

  • [Submitted] Fix python issues. Checked with flake8. Project builds successful.

####Wednesday July 27,2016

  • [In Progress] Update doc in wiki.

####Saturday July 30,2016

  • [In Progress] Update doc in wiki.

####Sunday July 31,2016

  • [Completed] Update doc in wiki.

###Week 8 [July 4,2016 - July 10,2016] ####Plan

  • Complete documentation/tutorial for creating a runner.[100%]
  • Clean up the code.
  • Prepare pull request to dev branch of galaxy.

####Progress ####Monday July 4,2016

  • [In Progress] Code cleanup.

####Tuesday July 5,2016

  • [Completed] Code cleanup.[Master branch]

####Saturday July 9,2016

  • [In Progress] Tutorial for creating a runner.

####Sunday July 10,2016

  • [Completed] Tutorial for creating a runner.

###Week 7 [June 27,2016 - July 3,2016] ####Plan

  • Complete documentation/tutorial for creating a runner. [>90%]
  • Enable access to galaxy packages in godocker cluster.
  • Complete the remaining tests in Test plan.
  • Wrap up with runner creation.

####Progress ####Wednesday June 30,2016

  • [Completed] Add virtualenv in destination params.
  • [In Progress] Test tool with python:2.7-wheezy image.

####Friday July 1,2016

  • [In Progress] Test tool with python:2.7-alpine image.

####Saturday July 2,2016

  • [In Progress] Debugging with python:2.7-alpine image.

####Sunday July 3,2016

  • [In Progress] Debugging with python:2.7-alpine image.

###Week 6 [June 20,2016 - June 26,2016] ####Plan

  • Work on default/Override params.[Specifically the issues opened on the issue tracker]
  • Enabling access to galaxy packages in the godocker clusters.[Find a way to do it].
  • Complete documentation/tutorial for creating a runner.[atleast 70 - 80%]

####Progress ####Monday June 20,2016

  • [Completed] Error handling for file read operations, params, etc are handled.
  • [Completed] Failure of job creation/submission is reported back to galaxy.

####Tuesday June 21,2016

  • [Completed] Override params.
  • [Completed] Test recover and various combination of container params.

####Wednesday June 22,2016

  • [In Progress] Creating documentation
  • [In Progress] Looking into galaxy code to find a way to activate galaxy env in cluster.

####Saturday June 25,2016

  • [In Progress] Creating documentation.
  • [Completed] Adding support for volumes param.
  • [In Progress] Galaxy venv.

####Sunday June 26,2016

  • [In Progress] Creating documentation.
  • [In Progress] Galaxy venv.

###Week 5 [June 13,2016 - June 19,2016] ####Plan

  • Code recover and stop job functions
  • Work on default/Override params.
  • Partially complete documentation/tutorial for creating a runner.
  • Complete an end to end execution of a job .

####Progress ####Tuesday June 14,2016

  • [Completed] error_file and output_file generated.
  • [Completed] countTool executed in godocker and returns result to galaxy.
  • [In Progress] Enhancement to check_watched_item and stop_job.
  • [In Progress] Testing runner with various tools and recording the results.

####Thursday June 16,2016

  • [Completed] Test Tool executed in godocker and returns result to galaxy.
  • [Completed] Recover and stop_job code.
  • [In Progress] Testing runner with various tools and recording the results.

####Saturday June 18,2016

  • [In Progress] Testing runner with various tools and recording the results.
  • [Completed] Re-write godocker functions in runner.

###Week 4 [June 6,2016 - June 12,2016] ####Tuesday June 7,2016

  • [Completed] A simple tool to test the GoDocker-Galaxy runner.
  • [In progress] Job submission through runner.

####Wednesday June 8,2016

  • [Completed] god_job.py and submit_job.py - test scripts created and results updated.
  • [In progress] Job submission through GoDocker runner.

####Thursday June 9,2016

  • [Completed] GoDocker runner bugs fixed.
  • [Completed] Test tool run using the GoDocker runner. Screenshots updated in /test/Result/Latest.
  • [Final Status] Runner successful starts and submits jobs to godocker now. :D

####Friday June 10,2016

  • [Completed] get_job_status from GoDocker tested.
  • [In progress] Code check_watched_item function in runner.

####Saturday June 11,2016

  • [Completed] Parsed job_wrapper and job_wrapper.tool objects.
  • [In progress] Code check_watched_item function in runner.

####Sunday June 12,2016

  • [In progress] Code stop function for godocker.
  • [In progress] Code check_watched_item function in runner.

###Week 3 [May 29,2016 - June 5,2016] ####Plan

  • Execute a job through the runner and show the results

####Progress ####Tuesday May 31,2016

  • [Partially Complete] Job submission code. Pending: Job parameters to be assigned to the job.
  • [In progress] Checking Job status code (check_watched_item func).
  • [Completed] Testing login function of godocker.

####Wednesday June 1,2016

  • [In progress] Testing job submission function of godocker.
  • [Pending] Handling exception in GoDocker API request call.
  • [In progress] A simple tool to test the GoDocker-Galaxy runner.
  • [Completed] Set of methods to interact with Godocker.
  • [In progress] Looking into galaxy handlers and job destinations.( Detailed conversation in galaxy IRC).

####Thursday June 2,2016

  • [In progress] Testing job submission function of godocker.
  • [Completed] Handling exception in GoDocker API request call.
  • [Pending] A simple tool to test the GoDocker-Galaxy runner.
  • [In progress] Looking into galaxy handlers and job destinations.( Detailed conversation in galaxy IRC).

####Sunday June 5,2016

  • [Completed] Job submission function of godocker.

###Week 2 [May 23,2016 - May28,2016] ####Plan

  • Create a simple runner with the features
  • Must get simple parameter from config.
  • Must authenicate user login in godocker.
  • Creating this skeleton will make the work progress faster and I can be sure of my understanding.

####Progress

  • Basic skeleton of the runner is created with godocker user login authentication.

####Week 1 [May 16,2016 - May21,2016] ####Plan 1: Look into existing docker integration with galaxy.
https://github.com/apetkau/galaxy-hackathon-2014
https://github.com/apetkau/galaxy-hackathon-2014/tree/master/smalt

2: Look into existing runners in galaxy.
https://github.com/varunshankar/galaxy/tree/dev/lib/galaxy/jobs/runners

OSALLOU: Look at https://github.com/varunshankar/galaxy/blob/dev/lib/galaxy/jobs/runners/drmaa.py which makes use of a library instead of executing processes, will be closer to GoDocker use case. Will use same AsynchronousJobRunner and same use cases. You should just additionally check how to grab extra configuration (Runner global config + per Job config) You will also need to document what you do on wifi on regular basis for later doc (should godocler_CLI bne installed in Galaxy virtualenv, how, .. where should new code be installed, ....)

3: Upload and run code in the remote machine provided.

4: Play around with galaxy codebase.

####Progress 1: Testing galaxy tools in local galaxy install.

2: Looked into DRMAA runner. I am preparing a report of the DRMAA runner.

https://godockergalaxy.wordpress.com/2016/05/22/my-inceptive-post/

####Progress [Until May 15,2016] 1: Local installation of Galaxy is successful. 2: Submission of job through CLI and web in VM is completed. 3: Galaxy is successfully getting started at 127.0.0.1:8080. 4: Modification of xml definition of a tool is successful.

#Development ##Galaxy runner

Galaxy/ Docker already defined some docker parameters in job configuration. Those parameters should be used instead of redefining container related images (and existing job definitions with Docker can work the same way). Existing optional params in global runner conf:

 <param id="docker_repo_override">my-docker-registry.org</param>
        <param id="docker_owner_override">superbioinfo</param>
        <param id="docker_image_override">my-tool</param>
        <param id="docker_tag_override">latest</param>
         <!-- You can also allow the destination to accept the docker container set in the tool, and only fall into
             the docker image set by this destination if the tool doesn't set a docker container, by using the
             "default" suffix instead of "override".
       <param id="docker_repo_default">my-docker-registry.org</param>
       <param id="docker_owner_default">superbioinfo</param>
       <param id="docker_image_default">my-tool</param>
       <param id="docker_tag_default">latest</param>
        <!-- REQUIRED: To play nicely with the existing galaxy setup for containers. This could be set though
             internally by the runner. -->
       <param id="docker_enabled">true</param>

To search: per tool container name, cpu/mem requirements

  • repo
  • owner
  • image
  • tag

##Virtual machine

Dev environment with Galaxy and GoDocker available at http://cloud-30.genouest.org/app/#/
GoDocker project: https://bitbucket.org/osallou/go-docker/

##Minimum requirements to execute jobs through GoDockerRunner
Galaxy instance : Code version >= galaxyproject/galaxy@a931b4b

job_conf.xml:

<job_conf>
<plugins>
    <plugin id="local" type="runner" load="galaxy.jobs.runners.local:LocalJobRunner" workers="4"/>
    <plugin id="godocker" type="runner" load="galaxy.jobs.runners.godocker:GodockerJobRunner">
        <param id="godocker_master">WWWW</param>
        <param id="user">XXXX</param>
        <param id="key">YYYYY</param>
    </plugin>
</plugins>
<handlers>
    <handler id="main"/>
</handlers>
<destinations default="god">
    <destination id="local" runner="local"/>
    <destination id="god" runner="godocker">
        <param id="docker_enabled">true</param>
        <param id="docker_default_container_id">centos:latest</param>
    </destination>
</destinations>
</job_conf>
Clone this wiki locally