Difference between revisions of "Managing Perl Libraries"

From WormBaseWiki
Jump to navigationJump to search
Line 15: Line 15:
 
''I maintain a local mirror of CPAN. This makes installing our web application significantly faster.''
 
''I maintain a local mirror of CPAN. This makes installing our web application significantly faster.''
  
−
Fetch [http://www.trout.me.uk/perl/mirror.sh.txt Matt Trout's minicpan mirroring script]. You can find our version at '''hg:wormbase-admin>mirror/minicpan.sh'''
+
$ sudo perl -MCPAN -e shell
−
<pre>
+
$ cpan> install CPAN::Mini
−
#!/bin/sh
 
−
# Dumbass minicpan replacement - Matt S Trout 2005/6. Perl license.
 
  
−
# Pick one from http://cpan.org/SITES.html#RSYNC
+
Create a minicpanrc file at ~/
−
REMOTEHOST='ftp.belnet.be'
 
−
REMOTEPREFIX='packages/cpan'
 
  
−
# Unless you're me, you probably want to change this too
+
<pre>
−
LOCAL='/home/matthewt/mirrors/minicpan'
+
# CPAN::Mini configuration file
 +
# to set up our local WormBase CPAN
  
−
cd $LOCAL
+
local: /usr/local/wormbase/minicpan
−
mkdir -p work authors/id modules
+
remote: http://mirror.rit.edu/CPAN/
 +
exact_mirror: 1
 +
</pre>
  
−
rsync $REMOTEHOST::"$REMOTEPREFIX/modules/03modlist.data.gz $REMOTEPREFIX/modules/02packages.details.txt.gz" modules/
+
Set up your CPAN to use the local mirror:
−
zcat modules/02packages.details.txt.gz | egrep '\.(tar\.gz|tgz|zip)$' | egrep -v '/((perl-|parrot-|ponie-)[0-9]|perl5\.004)' | awk '{print $3}' | sort -u >work/targets.dists
+
<pre>
−
cat work/targets.dists | cut -d'/' -f1-3 | sort -u | perl -pe 's!$!/CHECKSUMS!;' >work/targets.checksums
+
$ cpan
−
cat work/targets.dists | perl -pe 's!/[^/]+$!\n!;' >work/targets.dirs
+
cpan> o conf set urllist file:///usr/local/wormbase/minicpan/
 +
cpan> o conf commit
  
−
cd $LOCAL/authors/id
+
Set up the minicpan to run once a day:
−
cat $LOCAL/work/targets.dirs | xargs -n100 mkdir -p
+
* 3 * * * minicpan
−
cat $LOCAL/work/targets.checksums | xargs -n100 touch
 
−
cat $LOCAL/work/targets.dists | xargs -n100 touch
 
−
 
 
−
cd $LOCAL
 
−
rsync -vr --size-only --existing $REMOTEHOST::"$REMOTEPREFIX/authors/" authors/
 
−
find authors/id -name '*.*' | perl -pe 's!^authors/id/!!;' | sort -u >work/tree.all
 
−
sort -u work/targets.* | comm -13 - work/tree.all | perl -pe 's!^!authors/id/!;' | xargs rm -v
 
−
</pre>
 
  
 
= Installation Script =
 
= Installation Script =

Revision as of 16:30, 7 January 2011

Overview

Maintaining Perl modules across our server farm can be a time consuming and tedious task because of the heterogeneity of our setup. Further, we often need to maintain unique sets of libraries for different applications with collisions.

To solve these problems, I maintain local copies of libraries for each application. Typically, these are found at 

 /path/to/application/extlib

Installing modules in a local path is made significantly easier through the use of local::lib and Apache2/mod_perl2 "PerlSwitches" directive.

local::lib dispenses with the need to maintain multiple CPAN configurations or manual tweaks to your environment variables in order to recognize already installed modules.

Installing a local CPAN repository

I maintain a local mirror of CPAN. This makes installing our web application significantly faster. 

$ sudo perl -MCPAN -e shell
$ cpan> install CPAN::Mini

Create a minicpanrc file at ~/ 

# CPAN::Mini configuration file
# to set up our local WormBase CPAN

local: /usr/local/wormbase/minicpan
remote: http://mirror.rit.edu/CPAN/
exact_mirror: 1

Set up your CPAN to use the local mirror: 

$ cpan
cpan> o conf set urllist file:///usr/local/wormbase/minicpan/
cpan> o conf commit

Set up the minicpan to run once a day:
* 3 * * * minicpan


Installation Script

The steps described in this document are almost entirely scripted:

  wormbase-admin:maintenance/install_perl_modules.sh

  Usage:
  $ ./install_perl_modules.sh /path/to/private/libraries/extlib

  Usage (for installing Catalyst and its dependencies):
  $ ./install_perl_modules.sh /path/to/private/libraries/extlib CATALYST

Read on for additional (important) details that include appropriate settings for interactive steps in the script.


Private library locations

By convention, 3rd party Perl modules are maintained in an "extlib" directory within each project directory:

  website-classic/extlib - The "Classic" WormBase website
  website/extlib - 'The rearchitecture
  website-2.01/extlib - Version 2.01 of the rearchitecture

And so on...




Required Perl modules


WormBase "Classic"

Install the following Perl modules via CPAN. Note that you DO NOT AND SHOULD NOT be sudo.

 cd /usr/local/wormbase/extlib/classic
 perl -Mlocal::lib=./
 eval $(perl -Mlocal::lib=--self-contained,./)
 perl -MCPAN -e 'CPAN::install(MODULE)'

 // Some basics...
 YAML
 LWP
 ExtUtils::MakeMaker
 Bundle::CPAN
 
 // The rest...
 Bio::Graphics
 BioPerl   
    // When prompted, choose to download all optional modules. This will also install Ace:
 Ace
    // During configuration, choose option (3), then set the remaining variables as follows:
    Site-specific configuration files:  /usr/local/wormbase/website-classic/conf
    CGI path:  /usr/local/wormbase/website-classic/cgi-bin
    HTML path:  /usr/local/wormbase/website-classic/html

    // For Catalyst-based apps:
    Site-specific configuration files:  /usr/local/wormbase/website/conf
    CGI path:  /usr/local/wormbase/website/root/cgi
    HTML path:  /usr/local/wormbase/website/root/static

 Cache::Cache
 Cache::FileCache
 CGI
 CGI::Session  
 CGI::Cache
 CGI::Toggle
 Class::Base
 Data::Stag
 Date::Calc
 Date::Manip
 DB_File
 DBI
 DBD::mysql   (mysql devel package must be installed first)
     // To build by hand:
     perl Makefile.PL INSTALL_BASE=/usr/local/wormbase/extlib/classic --testuser=USER --testpassword=PASS
     make ; make test ; make install
 Digest::MD5
 Flickr::API
 Flickr::API::Simple
     // This is a private library
     cd /usr/local/wormbase/build
     perl ./Build.PL --install_base /usr/local/wormbase/extlib/classic
    ./Build install
 GD 
 GD::SVG
 GD::Graph
 GD::Graph::pie
 HTML::TokeParser
 IO::Scalar
 IO::String
 Image::GD::Thumbnail
 Log::Log4perl
 MIME::Lite
 mod_perl2   // Requires that apache2 already be installed; apxs is at /usr/local/apache2/bin/apxs
 Net::FTP
 Proc::Simple
 Search::Indexer
 SOAP::Lite
 Statistics::OLS
 Storable
 SVG
 SVG::Graph
 Term::ReadKey
 Test::Pod
 Text::Shellwords
 Time::Format
 WeakRef
 XML::SAX
 XML::Parser
 XML::DOM
 XML::Writer
 XML::Twig
 XML::Simple


WormBase > 2.0

WormBase2.0 and above uses the Catalyst web framework.  In addition to the modules described above we also need to install Catalyst.

These first two bundles will install the majority of the Catalyst web framework:

 Task::Catalyst
 Catalyst::Devel

  We also need:

 Catalyst::Model::DBI
 Catalyst::Model::Adaptor
 Catalyst::Plugin::ConfigLoader
 Catalyst::Plugin::Session
 Catalyst::Plugin::Session::State
 Catalyst::Plugin::Session::State::Cookie
 Catalyst::Plugin::Session::Store::FastMMap
 Catalyst::Action::RenderView
 Catalyst::Action::Rest
 Catalyst::Log::Log4perl
 Test::YAML::Valid


local::lib


Installation

local::lib is installed into the system Perl path:

  sudo perl -MCPAN -e install 'local::lib'


Setting a library path

 $ cd /usr/local/wormbase/extlib/classic 
 $ perl -Mlocal::lib=./
 export MODULEBUILDRC="/usr/local/wormbase/extlib/classic/.modulebuildrc"
 export PERL_MM_OPT="INSTALL_BASE=/usr/local/wormbase/extlib/classic"
 export PERL5LIB="/usr/local/wormbase/extlib/classic/lib/perl5:/usr/local/wormbase/extlib/classic/lib/perl5/x86_64-linux-gnu-thread-multi:$PERL5LIB"
 export PATH="/usr/local/wormbase/extlib/classic/bin:$PATH"

Take note: we will use some of these variables for, say, setting up apache.

 $ eval $(perl -Mlocal::lib=--self-contained,./)


Installing a module

 $ perl -MCPAN -Mlocal::lib=--self-contained -e 'CPAN::install(MODULE)'


Installing a module without using local::lib

You may on occasion need to install something without using local::lib.  For EUMM modules, use

  perl -I/usr/local/wormbase/[PROJECT]/extlib Makefile.PL INSTALL_BASE=/usr/local/wormbase/[PROECT]/extlib

And for Module::Build

  perl -I/usr/local/wormbase/[PROJECT]/extlib Makefile.PL --install_base /usr/local/wormbase/[PROJECT]/extlib 

Alternatively, you can manually edit ~/.cpan/CPAN/MyConfig.pm


Tips and Tricks

Some modules may require a little TLC.  After running the script you can find source at ~/.cpan/build/module-*


BioPerl

Instead of building BioPerl from CPAN, it might be necessary to build from SVN:

  $ cd /usr/local/wormbase/build
  $ svn co svn+ssh://USERNAME@dev.open-bio.org/home/svn-repositories/bioperl/bioperl-live/trunk bioperl-live
  $ mv bioperl-live bioperl-live-tharris
  $ cd bioperl-live-tharris
  $ perl ./Build.PL install_base /usr/local/wormbase/extlib
  $ ./Build test
  $ ./Build install


DBD::mysql

You will need to specify a test user and test password for tests to pass.


ModPerl2

You will need to be sudo in order to install the dso.


Updating Modules

To show all modules that are out-of-date with those on CPAN:

 $ cd /path/to/extlib
 $ perl -Mlocal::lib=./
 $ eval $(perl -Mlocal::lib=--self-contained,./)
 $ perl -MCPAN -e 'CPAN::Shell->r'

To update all of these modules at once:

 $ perl -MCPAN -e 'CPAN::Shell->install(CPAN::Shell->r)'


Managing libraries across architectures


Via CPAN bundles

 # Create a bundle
 $ cd /path/to/extlib
 $ perl -Mlocal::lib=./
 $ eval $(perl -Mlocal::lib=--self-contained,./)
 $ perl -MCPAN -e autobundle   # The bundle is (typically) in extlib

 # To reinstall:
 $ perl -MCPAN -e 'install Bundle::Snapshot_YYYY_MM_DD_00'

(Or just use the install script).
Retrieved from "https://wiki.wormbase.org/index.php?title=Managing_Perl_Libraries&oldid=9042"