Difference between revisions of "Website:Application overview"

From WormBaseWiki
Jump to navigationJump to search
 
(37 intermediate revisions by one other user not shown)
Line 3: Line 3:
 
= Directory Structure =
 
= Directory Structure =
  
= Configuration =
+
<pre>
 +
-rw-rw-r--    Changes
 +
-rw-rw-r--    Makefile.PL
 +
-rw-rw-r--    README
  
Catalyst offers a powerful configuration system.  We use it to provide application-wide, per-page, and per-session configuration.  In addition, local configuration files can be used to override any configuration option for production or development deployment.
+
drwxrwxr-x    conf
 +
  Application and third-party library configuration.
  
Here, $ROOT refers to the document root of your application.
+
drwxrwxr-x    design
 +
  Design elements and ideas.
  
== $ROOT/wormbase.yml ==
+
drwxrwxr-x    extlib
 +
  Directory containing builds of third party libraries.
  
This file contains the default configuration for the application.  The application defaults to using remote data sources.  If you would like to override this, see wormbase_local.yml.
+
drwxrwxr-x    lib
 +
  The core application modules.
  
== $ROOT/wormbase.yml.template ==
+
drwxrwxr-x    logs
 +
 
 +
drwxrwxr-x    private
 +
  Docs and presentations.
  
Move this template file to ''wormbase_local.yml'' and edit values to locally override the default configuration variables.  This file is maintainted in SVN as a template so that individual development preferences are not obliterated by an inadvertent svn commit.
+
drwxrwxr-x  root
 +
  /root contains static files and templates.
  
= Application structure =
+
drwxrwxr-x  script
 +
  Helper scripts and the stand-alone server.
  
The general structure of the application mirrors the organization in Acedb.  Separate Model::* packages correspond to classes in Acedb.
+
drwxrwxr-x  sql
 +
  SQL statements for various databases.
  
== Models ==
+
drwxrwxr-x  src
 +
  Third party sources.
  
===Generic Model methods===
+
drwxrwxr-x  t
 +
  Application test suite.
  
== Controllers ==
+
-rw-rw-rw-  wormbase.conf
 +
  Application-wide default configurations.
  
=== Generic Controller Actions ===
+
-rw-rw-r--  wormbase_local.conf
 +
  Template for local configuration.  Entries here will override defaults.
  
== Views ==
+
</pre>
  
= Widgets=
+
= Quick Overview =
  
Widgets are distinct presentation elements of a given model.  Widgets correspond to sections in the previous WormBase site.  For example, the Gene page has widgets for Identification, Location, and so on.
+
The code that powers the website is divided into two main components:  the code responsible for the website (henceforth "the web application") and a series of external models in the WormBase::API namespace.
  
== Specifying configuration ==
+
== External Model (aka WormBase::API) ==
  
== Required Controller actions ==
+
The WormBase::API is an external model and abstraction layer that rests on top of AcePerl/AceDB.  It's structure resembles classes in AceDB; however, models often cross class boundaries when aggregating data.
  
== Required Model methods ==
+
The external model implements a number of globally shared methods that can be used across classes (for example, collating references).
  
== Views ==
+
See the [[Website:WormBase::API|API documentation]] for additional details.
  
= Creating new Models and Controllers =
+
== The Web Application ==
  
1. Create your new Model or Controller using the wormbase_create.pl script.  This script provides stub formatting, documentation, and test files.
+
The Web Application follows the Model-View-Controller design pattern.
  
$ROOT/scripts/wormbase_create.pl model Test
+
=== Configuration ===
  
Creates...
+
Catalyst offers a powerful configuration system. We use it to provide application-wide, per-page, and per-session configuration. In addition, local configuration files can be used to override any configuration option for production or development deployment.  View-specific configuration (for example, external URLs) is maintained in Template::Toolkit configuration files.
  
  [todd@micos trunk:23]$ ./script/wormbase_create.pl model Test
+
See the [[Website:Configuration|Configuration documentation]] for details.
  exists "/Users/todd/projects/wormbase/website/trunk/script/../lib/WormBase/Model"
+
 
  exists "/Users/todd/projects/wormbase/website/trunk/script/../t"
+
=== Model ===
  created "/Users/todd/projects/wormbase/website/trunk/script/../lib/WormBase/Model/Test.pm"
+
 
created "/Users/todd/projects/wormbase/website/trunk/script/../t/model_Test.t"
+
The Web Application Model is a thin adaptor that pastes in the external model. There's not much to see and even less to do.
 +
 
 +
See the [[Website:Model|Model documentation]] for additional details.
 +
 
 +
=== View ===
 +
 
 +
The interface is a hybrid of server- and client-side processing powered by extensive Javascript, REST, and Template::Toolkit.
 +
 
 +
See the [[Website:View|View documentation]] for additional details.
 +
 
 +
=== Controller ===
 +
 
 +
Controllers handle interactions with the user. Controllers implement authorization/authentication, caching, and complete RESTful interface to the entire database.
 +
 
 +
See the [[Website:Controller|Controller documentation]] for additional details.
 +
 
 +
= General Page Structure =
 +
 
 +
Pages are composed of widgets, which are in turn composed of fields.
 +
 
 +
== Page ==
 +
 
 +
* Each page is specific in configuration (see below).
 +
* A generic template renders the page and inserts appropriate widgets by XHR.
 +
 
 +
==Widgets==
 +
 
 +
Widgets are small segments of a page comprised of a series of fields. Widgets roughly correspond to sections of the old WormBase site.
 +
 
 +
The widgets available for any page are specified in the primary configuration file. The architecture of a widget consists of:
 +
 
 +
* suitable configuration in wormbase.conf.
 +
* a (generic) REST controller action that...
 +
* ...calls methods in the external model that correspond to each component field.
 +
* and (usually) a template file.
 +
 
 +
==Fields==
 +
 
 +
Individual sections of each widget are referred to as fields. They are comprised of:
 +
 
 +
* a Controller method implementing a chained action. This calls a...
 +
* .. an external model method(s) that collates the appropriate data. The method name must match the name of the field.
 +
* field names are used as: stash keys and section names in templates.
 +
* each field is both a plain URL as well as REST target.
 +
 
 +
==Example==
 +
 
 +
  http://localhost/reports/gene/*/overview
 +
 
 +
Access a widget called overview that collects together the fields that comprise the widget.
 +
 
 +
http://localhost/reports/gene/*/ncbi
 +
 
 +
Access the ncbi field. Note that it is not necessary to know the containing widget.
 +
 
 +
= Setting up a new page =
 +
 
 +
== Add Configuration ==
 +
 
 +
Add configuration specifying the name of the page, the order widgets should be loaded, and each widget and the fields it contains to '''wormbase.conf''':
 +
 
 +
<pages>
 +
    <account_details>
 +
          widget_order profile
 +
          widget_order address
 +
          <widgets>
 +
                <profile>
 +
                      name profile
 +
                      title Account Profile
 +
                      fields id
 +
                      fields nickname
 +
                      fields email
 +
                </profile>
 +
                <address>
 +
                      name address
 +
                      title Address
 +
                      fields street
 +
                      fields city
 +
                      fields state
 +
                      fields zip
 +
                </address>
 +
        </widgets>
 +
    </account_details>
 +
  </pages>
 +
 
 +
See '''wormbase.conf''' and the documentation on [[Website:Configuration|configuration]] for full details.
 +
 
 +
==Create your WormBase::API::Object::* methods==
 +
 
 +
Remember: The names of these methods should correspond to the names of your widget fields!  If they do not, your template stash will not be populated.
 +
 
 +
== Create templates (if necessary)==
 +
 
 +
Create templates as necessary.  These should be located in:
 +
 
 +
The widget template:
 +
root/templates/classes/$class/$your_widget_name.tt2
 +
 
 +
= Caching =
 +
To ease server load and accelerate page load times, the web application implements a number of caching mechanisms.
 +
 
 +
== File cache of computationally intensive methods ==
 +
 
 +
=== Required modules ===
 +
 
 +
* Catalyst::Plugin::Cache
 +
* CHI
 +
* CHI::Driver::File
 +
 
 +
=== Usage ===
 +
 
 +
<pre>
 +
# Check the cache for the presence of your data.
 +
# my_cache_id can be any string you want (see below for details).
 +
 
 +
# $cache_id is the provided string with the current version of WormBase appended
 +
# $cached_data is data returned from the cache, if any
 +
my ($cache_id,$cached_data) = $c->check_cache("my_cache_id");
 +
 
 +
unless ($cached_data) {
 +
    my $data = some_computationally_expensive_method();
 +
    $c->set_cache($cache_id,$data);
 +
  }
 +
</pre>
 +
 
 +
=== Cache ID ===
 +
 
 +
When using the check_cache/set_cache approach, the initial cache ID that you provide will have the current version of WormBase appended to it.  This lets us automatically expire entries when a new version of the database is released.  Note that this ONLY works if you use both check_cache() and set_cache().  You should always follow this strategy as checking the cache is extremely efficient.
 +
 
 +
Here's an example from the REST controller for widgets:
 +
 
 +
<pre>
 +
sub widget_GET {
 +
    my ($self,$c,$class,$name,$widget) = @_;
 +
   
 +
    # Does the data for this widget already exist in the cache?
 +
    my ($cache_id,$cached_data) = $c->check_cache($class,$widget,$name);
 +
 
 +
    # The cache ONLY includes the field data for the widget, nothing else.
 +
    # This is because most backend caches cannot store globs.
 +
    if ($cached_data) {
 +
$c->stash->{fields} = $cached_data;
 +
    } else {
 +
 
 +
# Load the stash with the field contents for this widget.
 +
my @fields;
 +
 
 +
# Widgets accessible by name
 +
if (ref $c->config->{pages}->{$class}->{widgets}->{$widget}->{fields} ne "ARRAY") {
 +
    @fields = ($c->config->{pages}->{$class}->{widgets}->{$widget}->{fields});
 +
} else {
 +
    @fields = @{ $c->config->{pages}->{$class}->{widgets}->{$widget}->{fields} };
 +
}
 +
     
 +
foreach my $field (@fields) {
 +
    my $data = {};
 +
    $data = $object->$field if defined $object->$field;
 +
   
 +
    # Conditionally load up the stash (for now) for HTML requests.
 +
    # Alternatively, we could return JSON and have the client format it.
 +
    $c->stash->{fields}->{$field} = $data;
 +
}
 +
 +
# Cache the field data for this widget.
 +
$c->set_cache($cache_id,$c->stash->{fields});
 +
    }
 +
   
 +
    $self->status_ok($c, entity => {
 +
class  => $class,
 +
name    => $name,
 +
uri    => "$uri"
 +
    }
 +
);
 +
}
 +
</pre>
 +
 
 +
=== Types of data to store in the cache ===
 +
 
 +
In the REST widget() example, we cache all data drawn from the database necessary for the widget. We do not cache generated HTML (template processing is very efficient). Should we deem it necessary, we can exploring caching of generated HTML using Template::Plugin::Cache.
 +
 
 +
Note that it is not possible to cache data structures containing globs.
 +
 
 +
=== Cache Duration/Expiry ===
 +
 
 +
Caching is enabled for *all* installations. This enables us to test the caching mechanism in development.
 +
 
 +
Cached items in development installations are set to '''4 seconds''' to ensure that code changes are reflected in page reloads.  Cached items in production installations are set to expire after '''4 weeks'''. These values can be modified in lib/WormBase/Web.pm.
 +
 
 +
=== Cache Location ===
 +
 
 +
The cache is written to /tmp/wormbase/file_cache_chi.  This is specified during in Web.pm during application setup.
 +
 
 +
NFS?
 +
 
 +
== Component and page-level caching via reverse proxy ==
 +
 
 +
Production websites site behind a caching reverse proxy.  This proxy caches many (but not all) dynamically generated content as well.  Due to performance considerations, the front end proxy cache expires sooner than that of the back end application cache. See documentation on squid3 elsewhere on this wiki.
 +
 
 +
== Caching references ==
 +
 
 +
''The following sources contain useful information on cache options for Catalyst apps.''
 +
 
 +
*[http://wiki.catalystframework.org/wiki/adventcalendararticles/2007/11-Making_your_Catalyst_App_Cache-friendly Making your app cache friendly]
 +
*[http://wiki.catalystframework.org/wiki/wikicookbook/frontendproxy Catalyst behind a front-end proxy]
 +
*[http://www.catalystframework.org/calendar/2005/11 General caching options (old, but still somewhat useful]
 +
 
 +
 
 +
 
 +
 
 +
[[Category:Developer documentation]]
 +
[[Category:WormBase 2.0]]
 +
[[Category: Main Website (Web Dev)]]

Latest revision as of 14:03, 19 June 2014

Directory Structure

-rw-rw-r--    Changes
-rw-rw-r--    Makefile.PL
-rw-rw-r--    README

drwxrwxr-x    conf
  Application and third-party library configuration.

drwxrwxr-x    design
  Design elements and ideas.

drwxrwxr-x    extlib
  Directory containing builds of third party libraries.

drwxrwxr-x    lib
  The core application modules.

drwxrwxr-x    logs
  
drwxrwxr-x    private
  Docs and presentations.

drwxrwxr-x   root
  /root contains static files and templates.

drwxrwxr-x   script
  Helper scripts and the stand-alone server.

drwxrwxr-x   sql
  SQL statements for various databases.

drwxrwxr-x   src
  Third party sources.

drwxrwxr-x   t
  Application test suite.

-rw-rw-rw-  wormbase.conf
  Application-wide default configurations.

-rw-rw-r--   wormbase_local.conf
  Template for local configuration.  Entries here will override defaults.

Quick Overview

The code that powers the website is divided into two main components: the code responsible for the website (henceforth "the web application") and a series of external models in the WormBase::API namespace.

External Model (aka WormBase::API)

The WormBase::API is an external model and abstraction layer that rests on top of AcePerl/AceDB. It's structure resembles classes in AceDB; however, models often cross class boundaries when aggregating data.

The external model implements a number of globally shared methods that can be used across classes (for example, collating references).

See the API documentation for additional details.

The Web Application

The Web Application follows the Model-View-Controller design pattern.

Configuration

Catalyst offers a powerful configuration system. We use it to provide application-wide, per-page, and per-session configuration. In addition, local configuration files can be used to override any configuration option for production or development deployment. View-specific configuration (for example, external URLs) is maintained in Template::Toolkit configuration files.

See the Configuration documentation for details.

Model

The Web Application Model is a thin adaptor that pastes in the external model. There's not much to see and even less to do.

See the Model documentation for additional details.

View

The interface is a hybrid of server- and client-side processing powered by extensive Javascript, REST, and Template::Toolkit.

See the View documentation for additional details.

Controller

Controllers handle interactions with the user. Controllers implement authorization/authentication, caching, and complete RESTful interface to the entire database.

See the Controller documentation for additional details.

General Page Structure

Pages are composed of widgets, which are in turn composed of fields.

Page

  • Each page is specific in configuration (see below).
  • A generic template renders the page and inserts appropriate widgets by XHR.

Widgets

Widgets are small segments of a page comprised of a series of fields. Widgets roughly correspond to sections of the old WormBase site.

The widgets available for any page are specified in the primary configuration file. The architecture of a widget consists of:

  • suitable configuration in wormbase.conf.
  • a (generic) REST controller action that...
  • ...calls methods in the external model that correspond to each component field.
  • and (usually) a template file.

Fields

Individual sections of each widget are referred to as fields. They are comprised of:

  • a Controller method implementing a chained action. This calls a...
  • .. an external model method(s) that collates the appropriate data. The method name must match the name of the field.
  • field names are used as: stash keys and section names in templates.
  • each field is both a plain URL as well as REST target.

Example

http://localhost/reports/gene/*/overview

Access a widget called overview that collects together the fields that comprise the widget.

http://localhost/reports/gene/*/ncbi

Access the ncbi field. Note that it is not necessary to know the containing widget.

Setting up a new page

Add Configuration

Add configuration specifying the name of the page, the order widgets should be loaded, and each widget and the fields it contains to wormbase.conf:

<pages>
    <account_details>
         widget_order profile
         widget_order address
         <widgets>
               <profile>
                     name profile
                     title Account Profile
                     fields id
                     fields nickname
                     fields email
                </profile>
                <address>
                      name address
                      title Address
                      fields street
                      fields city
                      fields state
                      fields zip
                </address>
        </widgets>
    </account_details>
</pages>

See wormbase.conf and the documentation on configuration for full details.

Create your WormBase::API::Object::* methods

Remember: The names of these methods should correspond to the names of your widget fields! If they do not, your template stash will not be populated.

Create templates (if necessary)

Create templates as necessary. These should be located in:

The widget template: root/templates/classes/$class/$your_widget_name.tt2

Caching

To ease server load and accelerate page load times, the web application implements a number of caching mechanisms.

File cache of computationally intensive methods

Required modules

  • Catalyst::Plugin::Cache
  • CHI
  • CHI::Driver::File

Usage

 # Check the cache for the presence of your data.
 # my_cache_id can be any string you want (see below for details).

 # $cache_id is the provided string with the current version of WormBase appended
 # $cached_data is data returned from the cache, if any
 my ($cache_id,$cached_data) = $c->check_cache("my_cache_id");

 unless ($cached_data) {
     my $data = some_computationally_expensive_method();
     $c->set_cache($cache_id,$data);
  }

Cache ID

When using the check_cache/set_cache approach, the initial cache ID that you provide will have the current version of WormBase appended to it. This lets us automatically expire entries when a new version of the database is released. Note that this ONLY works if you use both check_cache() and set_cache(). You should always follow this strategy as checking the cache is extremely efficient.

Here's an example from the REST controller for widgets:

sub widget_GET {
    my ($self,$c,$class,$name,$widget) = @_; 
			     
    # Does the data for this widget already exist in the cache?
    my ($cache_id,$cached_data) = $c->check_cache($class,$widget,$name);

    # The cache ONLY includes the field data for the widget, nothing else.
    # This is because most backend caches cannot store globs.
    if ($cached_data) {
	$c->stash->{fields} = $cached_data;
    } else {
  
	# Load the stash with the field contents for this widget.
	my @fields;

	# Widgets accessible by name
	if (ref $c->config->{pages}->{$class}->{widgets}->{$widget}->{fields} ne "ARRAY") {
	    @fields = ($c->config->{pages}->{$class}->{widgets}->{$widget}->{fields});
	} else {
	    @fields = @{ $c->config->{pages}->{$class}->{widgets}->{$widget}->{fields} };
	}
       		
	foreach my $field (@fields) {
	    my $data = {};
	    $data = $object->$field if defined $object->$field;
	    
	    # Conditionally load up the stash (for now) for HTML requests.
	    # Alternatively, we could return JSON and have the client format it.
	    $c->stash->{fields}->{$field} = $data; 
	}
		
	# Cache the field data for this widget.
	$c->set_cache($cache_id,$c->stash->{fields});
    }
    
    $self->status_ok($c, entity => {
	class   => $class,
	name    => $name,
	uri     => "$uri"
		     }
	);
}

Types of data to store in the cache

In the REST widget() example, we cache all data drawn from the database necessary for the widget. We do not cache generated HTML (template processing is very efficient). Should we deem it necessary, we can exploring caching of generated HTML using Template::Plugin::Cache.

Note that it is not possible to cache data structures containing globs.

Cache Duration/Expiry

Caching is enabled for *all* installations. This enables us to test the caching mechanism in development.

Cached items in development installations are set to 4 seconds to ensure that code changes are reflected in page reloads. Cached items in production installations are set to expire after 4 weeks. These values can be modified in lib/WormBase/Web.pm.

Cache Location

The cache is written to /tmp/wormbase/file_cache_chi. This is specified during in Web.pm during application setup.

NFS?

Component and page-level caching via reverse proxy

Production websites site behind a caching reverse proxy. This proxy caches many (but not all) dynamically generated content as well. Due to performance considerations, the front end proxy cache expires sooner than that of the back end application cache. See documentation on squid3 elsewhere on this wiki.

Caching references

The following sources contain useful information on cache options for Catalyst apps.