Difference between revisions of "Log file management and analysis"

From WormBaseWiki
Jump to navigationJump to search
Line 44: Line 44:
[[Category:Maintenance (Web Dev)]]
[[Category:Developer documentation]]
[[Category:Developer documentation]]

Latest revision as of 17:31, 19 June 2014


I've used just about every logfile analysis tool known to man. Here are a few I use regularly.


Analog (coupled with ReportMagic) is the old stalwart. The graphs are ugly but the statistics are solid. Analog offers a huge number of configuration options and the ability to tweak just about every aspect of stat collection and report display.

For mapping non-resolved numerical IPs, I use jdresolve. It's fast and works very well.



Website: http://hping.org/visitors

License: Open-source


wget http://www.hping.org/visitors/visitors-0.7.tar.gz
tar xzf visitors-0.7.tar.gz
cd visitors_0.7


Note: Logs here already contain IP addresses resolved by hostname bruteforce or jdresolve.

todd> zcat /home/todd/projects/wormbase/log_archive/with_hosts/access_log.2008.gz | \
            visitors -A -T --prefix http://www.wormbase.org --prefix http://wormbase.org - \
            > log_analysis.html

To generate a dot representation of trails, try

todd> zcat /home/todd/projects/wormbase/log_archive/with_hosts/access_log.2008.gz | \
            visitors -A --prefix http://www.wormbase.org --prefix http://wormbase.org --graphviz - \
            > log_analysis.html
todd> dot graph.dot -Tpng > graph.png


Mint is a cool (and commercial) analytics tool. I use it for my personal sites but have never used to for analyzing the WormBase logs.