Developer's Corner

This page contains documentation and links related to the maintenance and development of the Perl Data Language.

TOC

Development Ideas

This is meant to be a collection of ideas currently floating around the PDL mailing list for current and future development.

All, I've consolidated the topics and discussion from perldl and pdl-porters from the past few weeks as part of an effort to produce a PDL way forward draft. It still needs some corrections to wiki as well as some reorganization.

However, the summary list should, at least, be a single place to look for the ideas and consensus as I saw it forming. I see this as a scratch page of specific tasks to move forward with PDL.

The list is long but there are some major thoughts and key ideas that stick out:

1. General Thoughts
2. Modularize PDL
3. Better Handling of External Dependencies
4. Provide a Baseline Default 2D and 3D Graphics Capability
5. Documentation and Usability Fixes
6. Build Process Fixes
7. Computational Improvements
8. Padre Development for Strawberry Perl Pro Release
9. Coordinate PDL Plans with Perl6

Accessing the latest from Git

The first step in developing PDL is getting the source files from the Git repository on Sourceforge. To begin, you should install Git. Once you've done that, go to an appropriate directory and issue this command:

git clone git://pdl.git.sourceforge.net/gitroot/pdl/pdl

After running this command, you should have a copy of the current state of PDL in a subdirectory named pdl. You can then edit the files as you like. As you are going along, you should regularly commit your changes to your local repository by issuing a command like this:

git commit -a

at which point you will be put into your normal text editor so that you can explain what it is you've done so far. You should give a one-line summary of what this commit contains, followed optionally by more verbose paragraphs explaining some of the details. You should probably also occasionally check for any recent updates from other developers by pulling from the repository on sourceforge with a command like this:

git pull

I could go into further details, but I believe that most of this (and much more yet to be added) should be put into a PDL-Git best-practices page, so I will hold off for now.

Pushing commits

Note that if you have developer access and you want to push your most recent work back to the repository at Sourceforge, you will need to set up your local repository's remote-configuration option to use the ssh protocol. A command along these lines should work:

git config remote.origin.url ssh://[email protected]/gitroot/pdl/pdl

Modifying the Website

Daniel did a big rewrite of the website, so many things do not work like they used to. The site now makes heavy use of PHP (instead of the previous Server Side Includes), and he also installed some Javascript libraries to make the banner, the front page slideshow, and the interactive installation instructions. If you want to make an edit to the website, here are the steps:

1. pull the latest pdl-www git repository.
2. set up two convenience aliases (see below) in your ~/.bashrc or ~/.profile or wherever is appropriate on your machine.
3. make your changes, use pdl-www-sync-test to push them to /future, iterate until satisfied.
4. use pdl-www-sync to push your changes to the main site.
5. use the standard git status, diff, add, commit, and push origin master commands to upload your changes to the git repository.

Here are the recommended aliases, where USER is your Sourceforge username, and we assume that ~/Build/PDL-WWW/ is the directory where the pdl-www git repository is stored on your machine.

alias pdl-www-sync-test='cd ~/Build/PDL-WWW; rsync -avi --delete --exclude-from=rsync.excludes . USER,[email protected]:htdocs/future; cd -;'
alias pdl-www-sync='cd ~/Build/PDL-WWW; rsync -avi --delete --exclude-from=rsync.excludes . USER,[email protected]:htdocs; cd -;'

The first of these aliases pushes your changes to the server and they are available at http://pdl.perl.org/future. This is important because you want to see how your changes will look with the PHP processor working and all of the image links correct, etc. If you are satisfied, then you can run the second one, which will push those changes to the main website. Note that you may wish to add the -n option to the rsync command at first, i.e.,

rysnc -avin <etc>

which just does a dry-run and does not actually push anything. This is helpful because of the --delete option, which will delete anything in the destination directory that is not excluded by the rsync.excludes file.

Updating the Online Documentation

To do this, you need to have some documentation to put online. At present this requires building PDL, so it is best to do this from a machine on which you can install all the optional libraries. The online documentation should be as complete as possible.

1. perl Makefile.PL (make sure there are no modules that will not be built)

2. make

3. make doctest

4. cd blib/lib/PDL/HtmlDocs/PDL

5. if perl -V:perladmin gives a personal email address that you don't want in the header of every HTML file, then run:

   find . -name '*.html' |xargs perl -pi -e 's/mailto:email-adress//;'

   where email-address is the appropriately \escaped perladmin email address

6. if the links in Index.html look like ./API.html instead of just API.html, the current PHP link post-processor will mangle them, so filter those out as well:

   find . -name '*.html' |xargs perl -pi -e 's/href="\.\//href="/;'

   if you mess something up you can always remake the html documents by going up a few directories and redoing the 'make doctest' step.

7. log into the SF shell service:

   ssh -t USER,[email protected] create

8. go to the PDL project webspace, and

   rm -rf PDLdocsOld
   cp -R PDLdocs PDLdocsOld

9. then on your local machine, upload the changes:

   rsync -avi --delete . USER,[email protected]:htdocs/PDLdocs

10. if you're really nervous you can upload them to htdocs/future/PDLdocs first and check them out at http://pdl.perl.org/future

Survey Results

Read the Results from the Fall 2009 Usage and Installation Survey.

Building SciPDL for Release

This section is for Matt and other Mac OS X PDL users/developers to document the recipe for building SciPDL for a new PDL release. Ideally, following these instructions would allow another PDL developer with a Mac OS X platform to build a new SciPDL package for release.

Checking Dependencies with Devel::CheckLib

Dependency checking is a critical component of the build process for PDL where many of the submodules depend on external libraries to provide needed functionality. We would like to move to a common framework for dependency detection that this platform independent and robust.

Devel::CheckLib looks like a promising approach. The idea is that Devel::CheckLib implements checks for libraries, include files, and functionality by compiling test programs with candidate options and verifying that they build and run. To this end, the current release of Devel::CheckLib is bundled with the PDL release to support checks without additional required external dependencies.

Sorting files based on last commit date

Portions of PDL have not been modified in a long time, in some cases a decade or more. The following command (in a bash shell from the top of the PDL git repository) sorts all the files in the tree by the date of last commit. You might find something you never knew was in there! (note: this may take some time to run because git, unlike cvs, does not keep track of files, but of commits, so there is not (to my knowledge) a one-off way to do this).

git ls-tree -r --name-only HEAD | while read filename; do echo "$(git log -1 --format="%ai" -- $filename) $filename"; done | sort -r