Products

Hammurapi

Hammurapi rules

jIncarnate

Supporting libraries

Common library

Enterprise extensions

Mesopotamia

Contact us

Discussion board

Newsletter

Abstract

This article describes a metrics framework. The conceptual architecture of the framework is similar to Log4j, but instead of consuming logging messages and routing them to appenders it collects measurements and routes them to measurement consumer, which typically aggregate them and either store to a database for later analysis and/or produce reports on JVM shutdown. Metrics framework is part of the common library. Due to its simple basic concepts it can be used to collect virtually any type of metric. For example it can serve as BAM (Business Activity Monitoring e.g. monitoring number, average order amount and volume of sales) and BPM (Business Process Monitoring, e.g. number of milliseconds to process order) tool at the same time.
The framework has pluggable architecture which allows to keep application code monitoring vendor neutral and "hook" the code to a particular monitoring tool at deployment time using configuration files. The framework also provides out-of-the-box measurement consumers and HTML report generator.

Introduction

It started about two years ago - one of my colleagues approached me with a flummoxing problem. There was a sporadic performance degradation in a web application he was a lead developer of.  The problem had place only on one server in production environment and could not be reproduced on any other server. The organization where we worked had a commercial performance monitoring tool in place as well as remote debugging and profiling tool, but because of organizational processes and policies both the tool were useless. The monitoring tool had some licensing limitations and could not be set up for the needed level of details. Profiling/debugging tool was useless because a) it is not the right tool to catch sporadic events b) setting JVM to profiling would kill performance, and c) nobody wanted to take risk of changing JVM parameters of production server. So the only means the project team had in possession was embedding monitoring into application code. I helped them by writing several classes and JSP to monitor performance of their JSP's and servlets. That time I believed that such a situation was caused by improper IT proper and such a solution would will not be needed as soon as the organization's IT stuff gains sufficient experience with the commercial monitoring and profiling tools. So I put the code I written aside and almost forgot about its existence.
Later when I started working on Hammurapi I needed some very simple mechanism to collect metrics such as method length, classes per package, ... I decided to use old metric classes with minor refactoring.
When Hammurapi had performance problems caused by non-optimal SQL I tried to use Eclipse Hyades to profile the code, but it was quite difficult because with profiling turned on performance went down sharply and reporting wasn't exactly what I needed. So I decided to revive the original purpose of metrics classes - performance monitoring and quickly found SQL statements which needed improvement.
I still believed that the only user of my metrics framework would be myself because there are industry standard tools tools to collect metrics, which all developers except me use. But several months ago my former boss told me that the situation which I described in the beginning of the introduction have not change much - there were more monitoring tools than two years ago, but developer didn't have access to those tools (because, to cut costs, licenses were purchased only for production environments) and operations people weren't familiar with application code and programming at all. So a) metrics collection couldn't be set properly b) collected metrics couldn't be interpreted.
That lead me to a conclusion that metrics collection shall be coded into application by developers, because they are the people who know critical classes and methods in their code where metrics collection shall be performed. I asked several developers working in other organizations how they incorporate metrics collection and monitoring into their applications and found that those who do it use custom-built solutions and nobody could point me to a de-facto standard for metrics collection similar to Log4j for logging. I hope the metrics framework presented here can fill the gap,

Samples

Code samples are located in samples folder of Common library in biz.hammurapi.metrics.sample package. Use Boot sample with metrics-config.xml file.

Concepts

• Metric - a name to associate measurements with. Metrics are organized into namespaces using categories.
• Measurement has the following characteristics:
• Metric name is is associated with.
• Value (double).
• Time when measurement was collected.
• Measurement consumer - an interface consuming measurements.
• Slice - an aggregation of measurements over a period of time. Slices has the following properties:
• Min value
• Max value
• Average
• Number of measurement in slice (intensity)
• Sum of all values (total)
• Deviation. Do not mistake it with normal deviation. It serves the same purpose, but is calculated differently because, generally, individual measurements are not stored - only aggregated values. This deviation is calculated as: deviaton=sum(abs(valuei-avgi))/measurements where xxx_i is a value and i-th measurement. The following table demonstrates the idea:

  #	Value	Avg	deviation
  1	1	1	(1-1)/1=0
  2	0	0.5	(abs(0-0.5)+0)/2=0.25
  3	-1	0	(abs(-1-0)/3+0.25)/3=0.19(4)

  As you can easily see, this deviation is not commutative - if we swap measurements #2 and #3 then deviation will be different, but this is the best what can be achieved without keeping all measurements in memory. You can use biz.hammurapi.metrics.sample.Deviation class to play with deviation calculation.
• Tick - length of slice in milliseconds.
• Slice consumer - an interface consuming slices. Slices are typically produced by SlicingMeasurementConsumer.
• Measurement category - a way to organize metrics. Following Log4j practice category name is typically a class name where particular metrics are collected.
• Time interval category - type of measurement category optimized for collecting time interval measurements, e.g. method execution time.
• Category factory - class which client classes use to obtain categories by name.
• Period - an aggregation of metrics belonging to some category and its subcategories over a period of time. Period is divided in configurable number of slices to show how metric/category behaved over time. Period's and its slices' total, average and number of measurements is calculated by aggregation and extrapolaiton of 'physical' slices stored in the database which overlap period and its 'logical' slices.
• Period visualizer - class generating charts.

What an application developer shall know

Since the primary goal of the framework is simplicity for an application developer I'll start with a description of "client" API. Classes and interfaces of concern of a developer are:

1. MeasurementCategoryFactory - this class has static methods to obtain MeasurementCategory and TimeIntervalCategory
2. MeasurementCategory - this interface primary purpose is to collect non-time measurements, e.g. amount of financial transactions, or size of incoming messages. It has two methods:
1. addMeasurement() to record a measurement
2. isActive() to determine if there are measurement consumers behind the category. This method is useful if measurement requires expencive calculations.
3. TimeIntervalCategory - Special case of  Measurement category to measure time intervals. It has the following methods:
1. getTime() - returns time in milliseconds or 0 if the category is inactive.
2. addInterval() - records time interval measurement.

This sample shows how to use MeasurementCategory:

...

import biz.hammurapi.metrics.MeasurementCategoryFactory;
import biz.hammurapi.metrics.MeasurementCategory;

public class MeasuerementCategorySample {
	private static final MeasurementCategory measurementCategory=MeasurementCategoryFactory.getCategory(Boot.class);

	public void transferFunds(Account debit, Account credit, BigDecimal amount) throws InsufficientFundsException { 
		...
		measurementCategory.addMeasurement("transfer-amount", amount.doubleValue(), 0);
	}
}

It collects measurements of transfer amount. The next sample collects measurements of time taken to process a transfer:

...

import biz.hammurapi.metrics.MeasurementCategoryFactory;
import biz.hammurapi.metrics.TimeIntervalCategory;

public class MeasuerementCategorySample {
	private static final TimeIntervalCategory tic=MeasurementCategoryFactory.getTimeIntervalCategory(Boot.class);

	public void transferFunds(Account debit, Account credit, BigDecimal amount) throws InsufficientFundsException { 
		long start=tic.getTime();
		...
		tic.addInterval("transfer-time", start);
	}
}

Now we'll combine the two samples:

...

import biz.hammurapi.metrics.MeasurementCategoryFactory;
import biz.hammurapi.metrics.TimeIntervalCategory;
import biz.hammurapi.metrics.MeasurementCategory;

public class MeasuerementCategorySample {
	private static final MeasurementCategory measurementCategory=MeasurementCategoryFactory.getCategory(Boot.class);
	private static final TimeIntervalCategory tic=MeasurementCategoryFactory.getTimeIntervalCategory(Boot.class);

	public void transferFunds(Account debit, Account credit, BigDecimal amount) throws InsufficientFundsException { 
		long start=tic.getTime();
		...
		measurementCategory.addMeasurement("transfer-amount", amount.doubleValue(), start);
		tic.addInterval("transfer-time", start);
	}
}

Please note that start is passed to addMeasurement() instead of 0. This saves one invocation of System.currentTimeMillis() when there are consumers for the category.

Architecture

This section describes the architecture is more detail. The cornerstone class of the whole architecture is MeasurementCategoryFactory. It decouples measurement producers (application code) from measurement consumers.
The diagram below depicts a simple scenario. 

1. MeasurementConsumer registers itself with MeasurementCategoryFactory by invoking static metod register().
2. MeasurementCategoryFactory creates a proxy class and registers the consumer with the proxy
3. Application class invokes static method getCategory() of MeasurementCategoryFactory. The factory returns a proxy instance for a category.
4. Application class invokes addMeasurement() of the proxy
5. The proxy invokes addMeasurement() for all consumers registered with the category.

Please note that a proxy is created only once for a category and is stored in an internal map.
There is a more advanced approach, though it looks the same for the application.

1. A subclass of MeasurementCategoryFactory, which implements abstract method getMeasurementConsumer(), registers itself with MeasurementCategoryFactory.
2. Application class invokes getCategory()
3. MeasurementCategoryFactory create a proxy (only once per category)
4. All registered factories are iterated over and getMeasurementConsumer() is invoked. 
5. Factories interested in a given category return MeasurementConsumer implementations, others return null.
6. Application class invokes addMeasurment() of the proxy.
7. The proxy iterates over registered consumers and invokes their addMeasurement() method.

Additional notes:

• Both the approaches (measurement consumer and factory) can be used at the same time. 
• One category can have zero or more consumers and one consumer can be registered with more than one category. 
• register() and getCategory() can be invoked in any order. Therefore consumers and factories can be added dynamically at runtime.
• There are unregister() methods as well and as such consumers and factories can be also removed dynamically at runtime.
• Registration of consumers and factories can be done"
• Explicitly - by calling register() method.
• Implicitly - by setting a system property and providing configuration file. 

Adding measurement consumers and factories implicitly

By default the framework tries to load configuration from hg-metrics-config.xml classpath resource. To change it you can set system property biz.hammurapi.metrics.MeasurementCategoryFactory:config to the location of list of consumers and factories. The list can be loaded either from

• File - prefix file:, 
• URL - prefix url:, 
• Classpath resource - prefix resource:. 

For example to load configuration from metrics.xml file add the following to Java command line: -Dbiz.hammurapi.metrics.MeasurementCategoryFactory:config=file:metrics-config.xml After that a configuration file shall be created. It is an XML file with the following structure:

	consumer or factory class name">
	... configuration parameters ...
	

	... more definitions ...

Configuration parameters for each of out-of-the box consumers and factories are listed below.
Example:

	
		
			metrics.html
		
	
	
	
		Metrics
		100
		1000
		50
		500
		300
		
		biz.hammurapi
	

Read JavaDoc of biz.hammurapi.config.DomConfigFactory class if you are going to add definitions of your own consumers/factories.

Slicing

biz.hammurapi.metrics.SlicingMeasurementConsumer is a class, which aggregates measurements into slices. It then invokes consumeSlice() of a SliceConsumer registered with. This method is invoked in a separate thread and as such doesn't affect directly application threads performance.

Persisting metrics and measurements

Several of out-of-the-box consumers and factories store metrics slices and optionally measurements to a relational database. The database schema is shown below. DDL scripts can be found in hgcommons.jar/biz/hammurapi/metricspersistent/HypersonicPersistingSliceConsumer.sql and hgcommons.jar/biz/hammurapi/metricspersistent/HypersonicPersistingSliceConsumerWithIdentity.sql

See the following sections for more details.

Visualization

The framework provides class biz.hammurapi.metrics.persistent.PeriodVisualizer to generate different charts:

1. Main chart is a candlestick chart which shows average, min, max, deviation and total (volume)
   
2. Average chart shows only average
   
3. Intensity chart shows number of measurements per slice in period
   
4. Total chart shows sum of measurement values per slice in period
   

Several measurement factories use this class to generate metrics documentation on JVM shutdown.
I'm planning to create metrics visualization servlets in XMenu in the future.

Measurement category factories

There are the following measurement category factories available out of the box:

SlicingMeasurementCategoryFactory

This factory produces slicing consumers, which aggregate measurements for each metric over a defined period of time (tick), thus forming a slice, which is then passed to SliceConsumer.consumeSlice() method. Configuration of this shown below:

	1000
	1000
	
		
	biz.hammurapi
		
	

where:

• tick - aggregation period in milliseconds.
• max-queue - maximum slices queue size. If slices are produces at very high rate (tick value is too low) or slice consumer is very slow then there can be a situation when slice queue size grows over time. Once it reaches max-queue value additional slices will be discarded.
• keep-measurements - optional element. Defaults to false. If true then slice will contain not only aggregated values but also values of individual measurements.
• category - optional element. If no categories are defined then factory will produce consumers for all categories, otherwise it will do so only for listed categories and their subcategories. In the sample above measurements will be collected only for biz.hammurapi category and its subcategories, e.g. biz.hammurapi.sql, biz.hammurapi.metrics.sample.
• slice-consumer - defines slice consumer. The sample above shows ConsoleSliceConsumer. Consumers available as part of common library are listed below.

HypersonicTmpPersistingSlicingMeasurementCategoryFactory

This factory uses temporary Hypersonic database to store metrics and generates HTML documentation on JVM shutdown. It can be used to analyze behavior of long-running batch processes, e.g. Hammurapi. Sample configuration:

	Metrics>
	100
	1000
	20
	500
	300
		
	biz.hammurapi

where

• output-dir - directory where documentation shall be generated
• tick - "physical" slice length in milliseconds
• max-queue - maximum size of slice queue
• slices - number of "logical" slices on charts
• width - chart width
• height - chart height
• category - categories to collect metrics from. There can be zero or more elements. Zero means that metrics will be collected from all categories.

Sample output

RemoteSlicingMeasurementCategoryFactory

Using this factory you can send slices to RemoteSliceConsumer through RMI. Here is how it should be configured:

	1000
	
	1000		
	biz.hammurapi	
	MyMachine.MyApp		
	//localhost/sliceConsumer>

where:

• root-category - category prefix for all categories.
• remote-slice-consumer - bind name of remote slice consumer.

semantics of other elements is identical to those in SlicingMeasurementCategoryFactory. To run a sample with remote consumer you should start biz.hammurapi.metrics.sample.RemoteConsumer first and then execute biz.hammurapi.metrics.sample.Boot.

HtmlMeasurementCategoryFactory

This factory collects metrics in-memory and generates a single HTML file on JVM shutdown. Take a look at a sample output. Below is an example of factory coniguration:

	
		metrics.html
	

XmlMeasurementCategoryFactory

Similar to HtmlMeasurementCategoryFactory, but output is in XML format. Sample configuration:

	
		metrics.xml
	

Sample output:

	from="Wed May 25 23:33:32 EDT 2005"
	to="Wed May 25 23:33:37 EDT 2005">
			category="biz.hammurapi.metrics.sample.Boot">
					name="Metric 2"
			avg="0.4820268203745048"
			min="-0.0882087897958237"
			max="1.0060246740964969"
			total="409.24077049795454"
			number="849"/>
					name="SimpleMetric"
			avg="-0.1151906159094869"
			min="-1.0619790579653057"
			max="1.0605061372713105"
			total="-115.1906159094869"
			number="1000"/>
	

Slice consumers

This section lists readily available slice consumers.

ConsoleSliceConsumer

This consumer outputs slices to System.out. Here is the sample configuration:

	1000
	1000
	
		
	biz.hammurapi
		
	

Sample output:

SLICE null: Name=SimpleMetric; Total=0.0; Avg=0.0; Min=0.0; Max=0.0; Deviation=0.0; From=1117045649424 (Wed May 25 14:27:29 EDT 2005); To=1117045650426 (Wed May 25 14:27:30 EDT 2005); Measurements=145
SLICE null: Name=Metric 2; Total=72.67828516556597; Avg=0.5012295528659721; Min=0.043215209117970256; Max=1.0358423898135782; Deviation=0.2287470263660576; From=1117045649424 (Wed May 25 14:27:29 EDT 2005); To=1117045650426 (Wed May 25 14:27:30 EDT 2005); Measurements=145
SLICE null: Name=SimpleMetric; Total=-46.523866824679956; Avg=-0.3040775609456206; Min=-1.0665476399573137; Max=0.0; Deviation=0.2507453454584542; From=1117045650436 (Wed May 25 14:27:30 EDT 2005); To=1117045651437 (Wed May 25 14:27:31 EDT 2005); Measurements=153
SLICE null: Name=Metric 2; Total=87.84143253452514; Avg=0.5741270100295761; Min=0.16426162792955448; Max=0.9739061731337673; Deviation=0.16737707208499528; From=1117045650436 (Wed May 25 14:27:30 EDT 2005); To=1117045651437 (Wed May 25 14:27:31 EDT 2005); Measurements=153
SLICE null: Name=SimpleMetric; Total=-106.1433923531543; Avg=-0.639418026223821; Min=-1.0480945086347246; Max=-0.06063115232461657; Deviation=0.23508446978608458; From=1117045651447 (Wed May 25 14:27:31 EDT 2005); To=1117045652449 (Wed May 25 14:27:32 EDT 2005); Measurements=166
SLICE null: Name=Metric 2; Total=87.92065438744022; Avg=0.5296424963098808; Min=0.3401971876692154; Max=0.7902520506037256; Deviation=0.07269198525507503; From=1117045651447 (Wed May 25 14:27:31 EDT 2005); To=1117045652449 (Wed May 25 14:27:32 EDT 2005); Measurements=166
SLICE null: Name=SimpleMetric; Total=93.60784739799429; Avg=0.5538925881538124; Min=-0.06023095135942347; Max=1.0101447346038979; Deviation=0.2891448716587698; From=1117045652449 (Wed May 25 14:27:32 EDT 2005); To=1117045653450 (Wed May 25 14:27:33 EDT 2005); Measurements=169
SLICE null: Name=Metric 2; Total=44.0140720764613; Avg=0.4445865866309222; Min=0.19691851114767056; Max=0.5373599832165616; Deviation=0.03612431607802528; From=1117045652449 (Wed May 25 14:27:32 EDT 2005); To=1117045653971 (Wed May 25 14:27:33 EDT 2005); Measurements=99
SLICE null: Name=SimpleMetric; Total=106.03821014342533; Avg=0.6711279123001603; Min=-0.04821581216180051; Max=1.056910167052953; Deviation=0.2465854061723151; From=1117045653460 (Wed May 25 14:27:33 EDT 2005); To=1117045654462 (Wed May 25 14:27:34 EDT 2005); Measurements=158
SLICE null: Name=Metric 2; Total=59.992718119796635; Avg=0.36140191638431707; Min=-0.09129462775918917; Max=0.8547822670397655; Deviation=0.21302598652130092; From=1117045654011 (Wed May 25 14:27:34 EDT 2005); To=1117045655013 (Wed May 25 14:27:35 EDT 2005); Measurements=166
SLICE null: Name=SimpleMetric; Total=-121.70101645176823; Avg=-0.7420793686083429; Min=-1.0593072326460664; Max=-0.05674434512945988; Deviation=0.26771134821925974; From=1117045654462 (Wed May 25 14:27:34 EDT 2005); To=1117045655463 (Wed May 25 14:27:35 EDT 2005); Measurements=164

JndiSliceConsumer

This consumer routes slices to RemoteSliceConsumer looked up via JNDI, e.g. EJB, which implements RemoteSliceConsumer interface.

	1000
	1000
	
	
	biz.hammurapi
		
	
		org.jnp.interfaces.NamingContextFactory
		org.jboss.naming:org.jnp.interfaces
		MySliceConsumerSessionBean
		MyComputer.MyApp			
	

where:

• environment-property - property to pass to InitialContext constructor.
• name - jndi name
• root-category - category prefix for all categories.

Log4jInfoSliceConsumer

This consumer sends slices to corresponding Log4j category at INFO level. Configuration:

	1000
	1000
		
	biz.hammurapi
		
	

Sample console output:

0 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=SimpleMetric; Total=0.0; Avg=0.0; Min=0.0; Max=0.0; Deviation=0.0; From=1117051991392 (Wed May 25 16:13:11 EDT 2005); To=1117051992394 (Wed May 25 16:13:12 EDT 2005); Measurements=152
30 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=Metric 2; Total=81.77792766627225; Avg=0.538012682014949; Min=0.030154851115090284; Max=1.0392398253555095; Deviation=0.2283103907766461; From=1117051991392 (Wed May 25 16:13:11 EDT 2005); To=1117051992394 (Wed May 25 16:13:12 EDT 2005); Measurements=152
801 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=SimpleMetric; Total=-43.704603247069926; Avg=-0.30562659613335613; Min=-1.0672021155040734; Max=0.0; Deviation=0.25218242010619807; From=1117051992454 (Wed May 25 16:13:12 EDT 2005); To=1117051993455 (Wed May 25 16:13:13 EDT 2005); Measurements=143
801 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=Metric 2; Total=80.48816780482925; Avg=0.5628543203134913; Min=0.20230932555646366; Max=0.9944371380757888; Deviation=0.15659933177000057; From=1117051992454 (Wed May 25 16:13:12 EDT 2005); To=1117051993455 (Wed May 25 16:13:13 EDT 2005); Measurements=143
1803 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=SimpleMetric; Total=-105.69001745520147; Avg=-0.7189797105796019; Min=-1.0527991295802928; Max=-0.20495797297247204; Deviation=0.18919226631123243; From=1117051993455 (Wed May 25 16:13:13 EDT 2005); To=1117051994457 (Wed May 25 16:13:14 EDT 2005); Measurements=147
1803 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=Metric 2; Total=77.81564364333222; Avg=0.5293581200226681; Min=0.3052855606964068; Max=0.8128183478108597; Deviation=0.08772221617151542; From=1117051993455 (Wed May 25 16:13:13 EDT 2005); To=1117051994457 (Wed May 25 16:13:14 EDT 2005); Measurements=147
2814 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=SimpleMetric; Total=53.59036921433336; Avg=0.3549031073796911; Min=-0.26152550510773526; Max=0.9098857747935313; Deviation=0.2882461056194501; From=1117051994467 (Wed May 25 16:13:14 EDT 2005); To=1117051995468 (Wed May 25 16:13:15 EDT 2005); Measurements=151
3605 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=Metric 2; Total=54.88494139646643; Avg=0.46121799492828935; Min=0.2494830534667637; Max=0.6602491143848346; Deviation=0.033916245452648416; From=1117051994467 (Wed May 25 16:13:14 EDT 2005); To=1117051996259 (Wed May 25 16:13:16 EDT 2005); Measurements=119
3826 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=SimpleMetric; Total=134.36998539315076; Avg=0.8725323726827972; Min=0.42252355376136147; Max=1.0651836458723283; Deviation=0.1144095105406261; From=1117051995478 (Wed May 25 16:13:15 EDT 2005); To=1117051996480 (Wed May 25 16:13:16 EDT 2005); Measurements=154
4627 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=Metric 2; Total=51.95153954092873; Avg=0.3684506350420477; Min=-0.08455418981878893; Max=0.8024922820730672; Deviation=0.21290245772610084; From=1117051996279 (Wed May 25 16:13:16 EDT 2005); To=1117051997281 (Wed May 25 16:13:17 EDT 2005); Measurements=141
4827 [Slice queue processor] INFO biz.hammurapi.metrics.sample.Boot  - biz.hammurapi.metrics.sample.Boot: Name=SimpleMetric; Total=-51.18263952605262; Avg=-0.3481812212656641; Min=-1.0268736989072162; Max=0.5075787219252352; Deviation=0.3839846709492152; From=1117051996480 (Wed May 25 16:13:16 EDT 2005); To=1117051997481 (Wed May 25 16:13:17 EDT 2005); Measurements=147

PersistingSliceConsumer

This consumer stores slices to a relational database. Database schema is shown in "Persisting metrics and measurements" section.
This consumer cannot be directly configured through XML. There is an adapter class - PersistingSliceConsumerDomConfigurableAdapter to configure this class.

PersistingSliceConsumerDomConfigurableAdapter

Adapter class to store slices to a relational database using PersistingSliceConsumer. Optionally generates HTML documentation on JVM shutdown.
Configuration:

	100		
	
	1000
		
	biz.hammurapi		
		
	
		org.hsqldb.jdbcDriver
		jdbc:hsqldb:file:C:\_Metrics\metrics
		sa
		
	
				
			
		biz.hammurapi.sql.hypersonic.HypersonicIdentityRetriever
	
		MyMachine
		
		1 day
		10 min		
			
		Metrics
		50
		500
		300		
			

where

• driver-class - JDBC driver class
• database-url - Database URL
• user - Database user
• password - Database password.
• name-map - Name map elements allow to map "logical" table names, e.g. SLICE be mapped to "physical" table names e.g. MY_SCHEMA.SLICE
• identity-manager - Class which implements biz.hammurapi.sql.IdentityGenerator or biz.hammurapi.sql.IdentityRetriever interface. E.g. HypersonicIdentiryRetriever uses "CALL IDENTITY()" to retrieve last autogenerated  primary key. For Oracle IdentityGenerator, which would read next sequence value is more appropriate. This entry is optional. If it is missing then SQLProcess.nextPk() method will be used to generate primary key value, but this method might not works well with all databases if  there are multiple processes generating primary keys at the same time.
• root-category - Root metrics category, e.g. machine name or IP address.
• history - indicates how long slices shall be kept in the database. Format: sec|min|hour|day|week|month|year.
• maintenance-interval - indicates how often database maintenance shall be performed. Format is the same as for history.
• output-dir - directory where documentation shall be generated. Optional.
• slices - number of "logical" slices on charts. Optional.
• width - chart width. Optional.
• height - chart height. Optional.

This picture depicts possible use of slice persistence.

DestinationSliceConsumer

Sends slices to JMS destination. Configuration:

	1000
	1000
	
		
	biz.hammurapi
		
	
		com.ibm.mq.jms.context.WMQInitialContextFactory
		localhost:1414/SRV_CHANNEL
		MyConnectionFactory
		SliceTopic
		sliceproducer
		secret
		MyComputer.MyApp			
		xml
	

where

• environment-property - property to pass to InitialContext constructor.
• connection-factory - Connection factory JNDI name.
• destination - Destination JNDI name.
• user - JMS user (optional).
• password - JMS password (optional)
• root-category - Root category.
• format - Message format. Optional. xml value forces creation of TextMessage with XML content. Any other value or absence of the tag causes creation of ObjectMessage.
  

QueueSliceConsumer

Almost identical to DestinationSliceConsumer, but uses JMS Queue API instead of more general Destination API. Use this consumer with old JMS providers, which do not support new JMS features.

	1000
	1000
	
		
	biz.hammurapi
		
	QueueSliceConsumer">
		com.ibm.mq.jms.context.WMQInitialContextFactory
		localhost:1414/SRV_CHANNEL
		MyConnectionFactory
		SliceQueue
		sliceproducer
		secret
		MyComputer.MyApp			
		xml
	

TopicSliceConsumer

Almost identical to DestinationSliceConsumer, but uses JMS Topic API instead of more general Destination API. Use this consumer with old JMS providers, which do not support new JMS features.

	1000
	1000
	
		
	biz.hammurapi
		
	TopicSliceConsumer">
		com.ibm.mq.jms.context.WMQInitialContextFactory
		localhost:1414/SRV_CHANNEL
		MyConnectionFactory
		SliceTopic
		sliceproducer
		secret
		MyComputer.MyApp			
		xml
	

MetricsServiceMBean

This MBean is part of XMenu. It wraps SlicingMeasurementSink and PersistingSliceConsumer. This is the content of metrics-service.xml:

  
  
  

    
    jboss.jca:name=MetricsDS,service=DataSourceBinding

    java:/MetricsDS    
    
    
    60000    
    
    
    false    
    
    
    1000    
    
    
    MyComputer    
    
    
    1440    
    
    
    10    
    
    

   

    
    

    

	biz.hammurapi.sql.hypersonic.HypersonicIdentityRetriever
  

Classes Instrumented out-of-the-box

biz.hammurapi.sql.SQLProcessor

This class reports metrics of SQL statement execution. Sample

biz.hammurapi.sql.MeasuringDatabaseObject

This class reports metrics for instantiation, insert, update, and delete operation. This class is typically used indirectly by subclasses generated by SQLC when "smartBase" attribute is set to the class name. Here is sample

biz.hammurapi.cache.MemoryCache

This class reports intensity of get and produce operations and cache size. Sample

Auto-instrumentation

It is also possible to automatically weave metrics collection into application code using:

• AOP
• Interceptors - in JBoss you can define an interceptor to collect metrics about EJB method execution.
• Dynamic proxies - one more way to collect method execution metrics.

Conclusion

This is a first cut of the article. Your feedback is greatly appreciated!

Acknowledgments

Metrics framework uses:

• JFreeChart for chart generation, 
• SQLC-generated classes in persistence layer,
• Several consumers and factories use Hypersonic database.