BeyondJ metrics with Camel, Jetty and Coda Hale
Apache Camel is a popular framework that enables message routing, transformation and mediation. BeyondJ uses Camel for message routing. This happens when a message is initially received by BeyondJ and the container has to determine a message handler. At this time BeyondJ accepts only HTTP/S messages, but there is no reason why BeyondJ could not accept AMQP, Web Socket, FTP, JDBC, JMS or any type of message, as Camel is well able to handle over 80 types of message protocols. Using Camel gives us access to a rich set of functionality and feature set which have been debugged and proven by the industry and community. Coda Hale metrics is a metrics library that gives you insight into what your application is doing. It is also used in the web framework Drop Wizard.
Jetty is a web application container that is widely used and is easy to embed in any Java application. The combination of Camel, Jetty and Coda Hale metrics gives us a versatile launching pad from which we can inspect the activities of all our application instances in BeyondJ with great granularity.
Coda Hale metrics are also used in the Drop Wizard project by Coda Hale and Yammer. They consist of Gauges, Timers, Counters, Histograms, Meters and Metric Reporters. Gauges consist of JMX Gauges, Ratio Gauges, Cached Gauges, Derivative Gauges. Histograms consist of Uniform Reservoirs, Exponentially Decaying Reservoirs, Sliding Window Reservoirs, and Sliding Time Window Reservoirs.
Reporters consist of JMX, Console, CSV, SLF4J, Servlet, Ganglia and Graphite reporters.
The core of the library is the MetricsRegistry, which is a repository all metrics within the instance of BeyondJ. If you were running your application in the classical JVM without BeyondJ, you would need a MetricsRegistry per application. Since you will be running multiple instances of your application within BeyondJ, each will have its own MetricsRegistry and additionally there is be a global MetricsRegistry within the JVM and that is owned by the core BeyondJ application. If you are running your application within the Jetty servlet container, the per application metrics int the local MetricsRegistry will also be added to the global MetricsRegistry. So not only do you get a panoramic view of all your application instances, you also get detailed insight into each of your application instances in terms of performance, memory usage, usage statistics and so on.
The metrics collector within the BeyondJ will periodically update other instances of BeyondJ with its local metrics via the Hazelcast cluster. Thus you also get metrics registration across the whole cluster. Each instance of BeyondJ will download the metrics of all other instances and add them to its metrics own global metrics registry. The end result is that when you connect to a single instance of BeyondJ, you get information about all other instances across the cluster. That is really, really powerful!
Each request in BeyondJ is routed via a Camel endpoint. Hence its is easy to immediately measure the request instance and add it to the metrics statistics. We are able to do this for any kind of request be it HTTP, FTP, AMPQ, Web Socket, or whatever, without writing much code.
Each measure is accumulated into a Coda Hale metric and sent to the metrics registry and the cluster publishing endpoint.
The Jetty servlet container actor also keeps per application instance metric registry. This registry is fed metrics from the Jetty ConnectorStatistics object on the Jetty server (for the application instance).
This local registry will periodically forward its statistics to the main BeyondJ metrics registry, which will then periodically publish all metrics to the cluster.
The nice thing is that you can view these metrics via the web console, both at the application instance level, container level and cluster level.
We are also able to get container metrics from the hawtio plugin: