Difference between revisions of "Managing Perl Libraries"

From WormBaseWiki
Jump to navigationJump to search
 
(40 intermediate revisions by 4 users not shown)
Line 1: Line 1:
 
= Overview =
 
= Overview =
  
I use local::lib to maintain distinct sets of Perl module environments for multiple applications.  It's great because you don't need to mess with multiple CPAN configurations or with tweaking your environment variables to recognize already installed local modules.  local::lib handles it all for you.
+
Maintaining Perl modules across our server farm can be a time consuming and tedious task because of the heterogeneity of our setup.  Further, we often need to maintain unique sets of libraries for different applications with collisions.
  
= Installation Script =
+
To solve these problems, I maintain local copies of libraries for each application.  Typically, these are found at
  
The following is almost entirely scripted:
+
  /path/to/application/extlib
  
  wormbase-admin:maintenance/install_perl_modules.sh
+
= Installing a local CPAN repository =
  
  Usage:
+
''I maintain a local mirror of CPAN. This makes installing our web application significantly faster.''
  $ ./install_perl_modules.sh /path/to/private/libraries/extlib
 
  
  Usage (for installing Catalyst and its dependencies):
+
$ sudo perl -MCPAN -e shell
  $ ./install_perl_modules.sh /path/to/private/libraries/extlib CATALYST
+
$ cpan> install CPAN::Mini
  
Read on for additional (important) details the includes appropriate settings for interactive steps in the script.
+
Create a ~/.minicpanrc file:
  
= local::lib Installation and Configuration =
+
<pre>
 +
# CPAN::Mini configuration file
 +
# to set up our local WormBase CPAN
  
local::lib is installed into the system Perl path:
+
local: /usr/local/wormbase/minicpan
 +
remote: http://mirror.rit.edu/CPAN/
 +
exact_mirror: 1
 +
</pre>
  
  sudo perl -MCPAN -e install 'local::lib'
+
Set up your CPAN to use the local mirror:
 +
<pre>
 +
$ cpan
 +
cpan> o conf urllist file:///usr/local/wormbase/minicpan/
 +
cpan> o conf commit
 +
</pre>
 +
 
 +
Don't forget cpanplus:
 +
<pre>
 +
$ cpanp
 +
$ CPAN Terminal> s reconfigure
 +
$ select the System configuration, then the mirror option, custom, and as above. Save and quit.
 +
</pre>
 +
 
 +
Set up the minicpan to run once a day:
 +
* 3 * * * minicpan
 +
 
 +
= Required Perl modules =
 +
 
 +
''See the [https://bitbucket.org/tharris/wormbase/src/d91a1c3571d4/Makefile.PL Makefile.PL] for a complete list of required Perl modules.''
 +
 
 +
= Installation Paths =
  
= Perl Module Locations =
+
By convention, 3rd party Perl modules are maintained in an "extlib" directory within each version of the site. This enables us to keep unique sets of libraries for each website.
  
3rd party Perl modules are maintained in an "extlib" directory within each project directory:
+
website/production/extlib - ''The current production version of the website''
 +
website/tharris/extlib - ''My private development space''
  
  website-classic/extlib - ''The "Classic" WormBase website
+
To set up your environment, copy the wormbase.env.template file to ''wormbase.env''.  Change the $APP variable to match your directory (ie ''tharris'). Then:
  website/extlib - 'The rearchitecture
+
  website-2.01/extlib - Version 2.01 of the rearchitecture
+
$ source wormbase.env
  
And so on...
+
== local::lib ==
  
= Setting your shell to install to the local environments =
+
The ''wormbase.env'' file uses local::lib to easily set up a local library path. local::lib is already installed into the system Perl path:
  
To set your current shell to use one of the local library do the following:
+
  sudo perl -MCPAN -e install 'local::lib'
  
  $ cd /usr/local/wormbase/extlib/2.0
+
local::lib dispenses with the need to maintain multiple CPAN configurations or manual tweaks to your environment variables in order to recognize already installed modules.
 +
 
 +
'''To set a local installation environment, you would:'''
 +
 
 +
  $ cd /usr/local/wormbase/website/tharris/extlib
 
  $ perl -Mlocal::lib=./
 
  $ perl -Mlocal::lib=./
 +
export MODULEBUILDRC="/usr/local/wormbase/website/tharris/extlib/.modulebuildrc"
 +
export PERL_MM_OPT="INSTALL_BASE=/usr/local/wormbase/website/tharris/extlib"
 +
export PERL5LIB="/usr/local/wormbase/website/tharris/extlib/lib/perl5:/usr/local/wormbase/website/tharris/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:$PERL5LIB"
 +
export PATH="/usr/local/wormbase/website/tharris/extlib/bin:$PATH"
 +
 
  $ eval $(perl -Mlocal::lib=--self-contained,./)
 
  $ eval $(perl -Mlocal::lib=--self-contained,./)
$ printenv    // Notice that the local path is now in your PERL5LIB
 
  
And install some modules:
+
'''To install a module'''
  
 
  $ perl -MCPAN -Mlocal::lib=--self-contained -e 'CPAN::install(MODULE)'
 
  $ perl -MCPAN -Mlocal::lib=--self-contained -e 'CPAN::install(MODULE)'
  
= Required Perl modules =
+
'''Installing a module without using local::lib'''
 +
 
 +
You may on occasion need to install something without using local::lib.  For EUMM modules, use
 +
 
 +
  perl -I/usr/local/wormbase/[PROJECT]/extlib Makefile.PL INSTALL_BASE=/usr/local/wormbase/[PROECT]/extlib
 +
 
 +
And for Module::Build
 +
 
 +
  perl -I/usr/local/wormbase/[PROJECT]/extlib Makefile.PL --install_base /usr/local/wormbase/[PROJECT]/extlib
 +
 
 +
Alternatively, you can manually edit ~/.cpan/CPAN/MyConfig.pm
  
== WormBase "Classic" ==
+
= Installing WebApp Dependencies =
  
Install the following Perl modules via CPAN. Note that you DO NOT AND SHOULD NOT be sudo.
+
Build all requirements is as easy as:
  
  cd /usr/local/wormbase/extlib/classic
+
  $ cd /usr/local/wormbase/website/$APP
  perl -Mlocal::lib=./
+
  $ source wormbase.env
  eval $(perl -Mlocal::lib=--self-contained,./)
+
  $ perl Makefile.PL
  perl -MCPAN -e 'CPAN::install(MODULE)'
+
  $ make installdeps
  
// Some basics...
+
Note: the wormbase.env should ensure local::lib use /usr/local/wormbase/extlib/ as installation path for packages.
YAML
 
LWP
 
ExtUtils::MakeMaker
 
Bundle::CPAN
 
 
// The rest...
 
Bio::Graphics
 
BioPerl 
 
      // When prompted, choose to download all optional modules. This will also install Ace:
 
Ace
 
      // During configuration, choose option (3), then set the remaining variables as follows:
 
    Site-specific configuration files:  /usr/local/wormbase/website-classic/conf
 
                                                        CGI path:  /usr/local/wormbase/website-classic/cgi-bin
 
                                                      HTML path:  /usr/local/wormbase/website-classic/html
 
                    Change these paths as appropriate for other projects.
 
Cache::Cache
 
Cache::FileCache
 
CGI
 
CGI::Session 
 
CGI::Cache
 
CGI::Toggle
 
Class::Base
 
Data::Stag
 
Date::Calc
 
Date::Manip
 
DB_File
 
DBI
 
DBD::mysql  (mysql devel package must be installed first)
 
    // To build by hand:
 
    perl Makefile.PL INSTALL_BASE=/usr/local/wormbase/extlib/classic --testuser=USER --testpassword=PASS
 
    make ; make test ; make install
 
Digest::MD5
 
Flickr::API
 
Flickr::API::Simple
 
    // This is a private library
 
    cd /usr/local/wormbase/build
 
    perl ./Build.PL --install_base /usr/local/wormbase/extlib/classic
 
    ./Build install
 
GD
 
GD::SVG
 
GD::Graph
 
GD::Graph::pie
 
HTML::TokeParser
 
IO::Scalar
 
IO::String
 
Image::GD::Thumbnail
 
Log::Log4perl
 
MIME::Lite
 
mod_perl2  // Requires that apache2 already be installed; apxs is at /usr/local/apache2/bin/apxs
 
Net::FTP
 
Proc::Simple
 
Search::Indexer
 
SOAP::Lite
 
Statistics::OLS
 
Storable
 
SVG
 
SVG::Graph
 
Term::ReadKey
 
Test::Pod
 
Text::Shellwords
 
Time::Format
 
WeakRef
 
XML::SAX
 
XML::Parser
 
XML::DOM
 
XML::Writer
 
XML::Twig
 
XML::Simple
 
  
== WormBase > 2.0 ==
+
= Tips and Tricks =
  
WormBase2.0 and above uses the Catalyst web framework.  In addition to the modules described above we also need to install Catalyst.
+
== ACePerl ==
  
These first two bundles will install the majority of the Catalyst web framework:
+
Adjust the values below as necessary for your
  
Task::Catalyst
+
[tharris@wb-dev: AcePerl-1.92-BlHove]> sudo perl Makefile.PL
Catalyst::Devel
 
  
  We also need:
+
What do you want to build?
  
Catalyst::Model::DBI
+
  1) Interface to Ace socket server and local databases (pure Perl)
Catalyst::Model::Adaptor
+
  2) The above plus XS optimizations (requires C compiler)
Catalyst::Plugin::ConfigLoader
+
  3) The above plus RPC server interface (requires C compiler)
Catalyst::Plugin::Session
 
Catalyst::Plugin::Session::State
 
Catalyst::Plugin::Session::State::Cookie
 
Catalyst::Plugin::Session::Store::FastMMap
 
Catalyst::Action::RenderView
 
Catalyst::Action::Rest
 
Catalyst::Log::Log4Perl
 
Test::YAML::Valid
 
  
= Displaying Environment variables =
+
Enter your choice:  [1] 3
  
To quickly show a list of handy environment variables, do:
+
Do you want to install Ace::Browser?  [n] y
  
$ cd /usr/local/wormbase/extlib/classic
+
You have installed Ace::Browser before, using old settings for defaults.
$ perl -Mlocal::lib=./
+
Directory for the site-specific configuration files (~username ok): [/usr/local/wormbase/website/tharris/conf/ace]
export MODULEBUILDRC="/usr/local/wormbase/extlib/classic/.modulebuildrc"
+
Directory for the acebrowser CGI scripts (~username ok): [/usr/local/wormbase/website/tharris/root/cgi-bin/ace]
export PERL_MM_OPT="INSTALL_BASE=/usr/local/wormbase/extlib/classic"
+
Directory for the acebrowser HTML files and images (~username ok): [/usr/local/wormbase/website/tharris/root/static/ace]
export PERL5LIB="/usr/local/wormbase/extlib/classic/lib/perl5:/usr/local/wormbase/extlib/classic/lib/perl5/x86_64-linux-gnu-thread-multi:$PERL5LIB"
 
export PATH="/usr/local/wormbase/extlib/classic/bin:$PATH"
 
  
We will use some of these variables for, say, setting up apache.
 
  
= Tips and Tricks =
 
  
 
''Some modules may require a little TLC.  After running the script you can find source at ~/.cpan/build/module-*''
 
''Some modules may require a little TLC.  After running the script you can find source at ~/.cpan/build/module-*''
Line 183: Line 146:
 
You will need to specify a test user and test password for tests to pass.
 
You will need to specify a test user and test password for tests to pass.
  
== Ace ==
+
= Updating Modules =
 +
 
 +
To show all modules that are out-of-date with those on CPAN:
  
You'll need to specify paths for Ace Browser.
+
$ cd /path/to/extlib
 +
$ perl -Mlocal::lib=./
 +
$ eval $(perl -Mlocal::lib=--self-contained,./)
 +
$ perl -MCPAN -e 'CPAN::Shell->r'
  
== ModPerl2 ==
+
To update all of these modules at once:
  
You will need to be sudo in order to install the dso.
+
$ perl -MCPAN -e 'CPAN::Shell->install(CPAN::Shell->r)'
  
= Installing modules without using local::lib =
+
= Managing libraries across architectures =
  
You may on occasion need to install something without using local::lib.  For EUMM modules, use
+
== Via CPAN bundles ==
  
  perl -I/usr/local/wormbase/extlib/[PROJECT] Makefile.PL INSTALL_BASE=/usr/local/wormbase/extlib/[PROJECT]
+
# Create a bundle
 +
$ cd /path/to/extlib
 +
$ perl -Mlocal::lib=./
 +
$ eval $(perl -Mlocal::lib=--self-contained,./)
 +
$ perl -MCPAN -e autobundle  # The bundle is (typically) in extlib
  
And for Module::Build
+
# To reinstall:
 +
$ perl -MCPAN -e 'install Bundle::Snapshot_YYYY_MM_DD_00'
  
  perl -I/usr/local/wormbase/extlib/[PROJECT] Makefile.PL --install_base /usr/local/wormbase/extlib/[PROJECT]
+
(Or just use the install script).
  
Alternatively, you can manually edit ~/.cpan/CPAN/MyConfig.pm
+
[[Category:Maintenance (Web Dev)]]
 +
[[Category:Developer documentation]]

Latest revision as of 18:47, 13 April 2015

Overview

Maintaining Perl modules across our server farm can be a time consuming and tedious task because of the heterogeneity of our setup. Further, we often need to maintain unique sets of libraries for different applications with collisions.

To solve these problems, I maintain local copies of libraries for each application. Typically, these are found at

 /path/to/application/extlib

Installing a local CPAN repository

I maintain a local mirror of CPAN. This makes installing our web application significantly faster.

$ sudo perl -MCPAN -e shell
$ cpan> install CPAN::Mini

Create a ~/.minicpanrc file:

# CPAN::Mini configuration file
# to set up our local WormBase CPAN

local: /usr/local/wormbase/minicpan
remote: http://mirror.rit.edu/CPAN/
exact_mirror: 1

Set up your CPAN to use the local mirror:

$ cpan
cpan> o conf urllist file:///usr/local/wormbase/minicpan/
cpan> o conf commit

Don't forget cpanplus:

$ cpanp
$ CPAN Terminal> s reconfigure
$ select the System configuration, then the mirror option, custom, and as above. Save and quit.

Set up the minicpan to run once a day:

  • 3 * * * minicpan

Required Perl modules

See the Makefile.PL for a complete list of required Perl modules.

Installation Paths

By convention, 3rd party Perl modules are maintained in an "extlib" directory within each version of the site. This enables us to keep unique sets of libraries for each website.

website/production/extlib - The current production version of the website
website/tharris/extlib - My private development space

To set up your environment, copy the wormbase.env.template file to wormbase.env. Change the $APP variable to match your directory (ie tharris'). Then:

$ source wormbase.env

local::lib

The wormbase.env file uses local::lib to easily set up a local library path. local::lib is already installed into the system Perl path:

 sudo perl -MCPAN -e install 'local::lib'

local::lib dispenses with the need to maintain multiple CPAN configurations or manual tweaks to your environment variables in order to recognize already installed modules.

To set a local installation environment, you would:

$ cd /usr/local/wormbase/website/tharris/extlib
$ perl -Mlocal::lib=./
export MODULEBUILDRC="/usr/local/wormbase/website/tharris/extlib/.modulebuildrc"
export PERL_MM_OPT="INSTALL_BASE=/usr/local/wormbase/website/tharris/extlib"
export PERL5LIB="/usr/local/wormbase/website/tharris/extlib/lib/perl5:/usr/local/wormbase/website/tharris/extlib/lib/perl5/x86_64-linux-gnu-thread-multi:$PERL5LIB"
export PATH="/usr/local/wormbase/website/tharris/extlib/bin:$PATH"
$ eval $(perl -Mlocal::lib=--self-contained,./)

To install a module

$ perl -MCPAN -Mlocal::lib=--self-contained -e 'CPAN::install(MODULE)'

Installing a module without using local::lib

You may on occasion need to install something without using local::lib. For EUMM modules, use

 perl -I/usr/local/wormbase/[PROJECT]/extlib Makefile.PL INSTALL_BASE=/usr/local/wormbase/[PROECT]/extlib

And for Module::Build

 perl -I/usr/local/wormbase/[PROJECT]/extlib Makefile.PL --install_base /usr/local/wormbase/[PROJECT]/extlib 

Alternatively, you can manually edit ~/.cpan/CPAN/MyConfig.pm

Installing WebApp Dependencies

Build all requirements is as easy as:

$ cd /usr/local/wormbase/website/$APP
$ source wormbase.env
$ perl Makefile.PL
$ make installdeps

Note: the wormbase.env should ensure local::lib use /usr/local/wormbase/extlib/ as installation path for packages.

Tips and Tricks

ACePerl

Adjust the values below as necessary for your

[tharris@wb-dev: AcePerl-1.92-BlHove]> sudo perl Makefile.PL

What do you want to build?

 1) Interface to Ace socket server and local databases (pure Perl)
 2) The above plus XS optimizations (requires C compiler)
 3) The above plus RPC server interface (requires C compiler)

Enter your choice: [1] 3

Do you want to install Ace::Browser? [n] y

You have installed Ace::Browser before, using old settings for defaults. Directory for the site-specific configuration files (~username ok): [/usr/local/wormbase/website/tharris/conf/ace] Directory for the acebrowser CGI scripts (~username ok): [/usr/local/wormbase/website/tharris/root/cgi-bin/ace] Directory for the acebrowser HTML files and images (~username ok): [/usr/local/wormbase/website/tharris/root/static/ace]


Some modules may require a little TLC. After running the script you can find source at ~/.cpan/build/module-*

BioPerl

Instead of building BioPerl from CPAN, it might be necessary to build from SVN:

 $ cd /usr/local/wormbase/build
 $ svn co svn+ssh://USERNAME@dev.open-bio.org/home/svn-repositories/bioperl/bioperl-live/trunk bioperl-live
 $ mv bioperl-live bioperl-live-tharris
 $ cd bioperl-live-tharris
 $ perl ./Build.PL install_base /usr/local/wormbase/extlib
 $ ./Build test
 $ ./Build install

DBD::mysql

You will need to specify a test user and test password for tests to pass.

Updating Modules

To show all modules that are out-of-date with those on CPAN:

$ cd /path/to/extlib
$ perl -Mlocal::lib=./
$ eval $(perl -Mlocal::lib=--self-contained,./)
$ perl -MCPAN -e 'CPAN::Shell->r'

To update all of these modules at once:

$ perl -MCPAN -e 'CPAN::Shell->install(CPAN::Shell->r)'

Managing libraries across architectures

Via CPAN bundles

# Create a bundle
$ cd /path/to/extlib
$ perl -Mlocal::lib=./
$ eval $(perl -Mlocal::lib=--self-contained,./)
$ perl -MCPAN -e autobundle   # The bundle is (typically) in extlib
# To reinstall:
$ perl -MCPAN -e 'install Bundle::Snapshot_YYYY_MM_DD_00'

(Or just use the install script).