An Architect's View

CFML, Clojure, Software Design, Frameworks and more...

An Architect's View

Instrumenting Clojure for New Relic Monitoring

May 1, 2013 ·

We've recently started evaluating the New Relic monitoring service at World Singles and when you use their Java agent with your web application container, you can get a lot of information about what's going on inside your application (JVM activity, database activity, external HTTP calls, web transaction traces). For a CFML application tho', all you tend to get in the web transaction traces is the Servlet entry point, some JDBC SQL reports, and some of the low-level Java libraries (if you're lucky!).

However, we have a mixture of CFML and Clojure, running on the free open source Railo server so I thought it might be possible to somehow instrument the Clojure code to enable more visibility into our application traces.

This New Relic blog post talks about custom instrumentation and shows how you can use method annotations in Java to make specific function calls show up in web transaction traces. (The New Relic documentation, accessible from their monitoring console, provides more detail)

In Java (approximately):

import com.newrelic.api.agent.Trace;

public class Thing {

    @Trace
    public void someMethod() {
        ...
    }

}

So how do you do the same thing in Clojure? Annotations have been supported for quite a while in Clojure but they're very poorly documented and they require some specialized code.

Let's suppose you have this Clojure code:

(ns my-project.stuff)

(defn some-func [x y z] ...)

(defn another-fn [a b] ...)

We want to annotate both of these functions so they show up in New Relic web transaction traces. You can add annotations in a deftype so we have to transform our code quite a bit. First off, you'll need the New Relic JAR as a dependency in project.clj:

  :dependencies
    [...
     [com.newrelic.agent.java/newrelic-api "2.17.2"]
     ...]

Then you need to import the Trace annotation:

(ns my-project.stuff
  (:import com.newrelic.api.agent.Trace))

Since we need to use deftype we need to define an interface type for the functions we want to instrument. We also need to use Java-compatible names. Here's a first cut:

(definterface INR
  (some_func [x y z])
  (another_fn [a b]))

Now we can define the implementation type with the annotations. Then we'll need to expose a Clojure API based on that. Let's start by renaming our existing implementations and making them private, so we can call them from the deftype with minimal code changes:

(defn- some-func* [x y z] ...)

(defn- another-fn* [a b] ...)

Now we can write our deftype with annotations:

(deftype NR []
  INR
  ;; @Trace maps to Trace {} metadata:
  (^{Trace {}} some_func  [_ x y z] (some-func* x y z))
  (^{Trace {}} another_fn [_ a b]   (another-fn* a b)))

Here we have an implementation - NR - of our interface - INR - which provides the two methods. Note that they ignore their first argument (this) because the object type is just an artifact of deftype for us to add the annotations. Finally, we can reimplement our original API in terms of the new type. In order to do that, we need an instance of our type, but since we don't need it for anything particular, we can create a single private instance to use:

(def ^:private nr (NR.))

And finally here is original API reimplemented in a traceable way:

(defn some-func  [x y z] (.some_func nr x y z))

(defn another-fn [a b]   (.another_fn nr a b))

Now you just need to start your web application container with the JVM option -javaagent:/path/to/newrelic.jar (which is your licensed JAR file downloaded from your monitoring console, not the one pulled in by Leiningen based on the dependency we added above!).

When you drill into web transaction traces, you should see my_project.stuff.NR.some_func and my_project.stuff.NR.another_fn entries in it!

Tags: clojure

4 responses

  • 1 tzach // May 2, 2013 at 2:05 AM

    Thanks
    I'm actually going to try this right now!

    BTW, minor typo in the dependencies part:
    "agenet"
  • 2 Sean Corfield // May 2, 2013 at 4:49 PM

    Typo fixed - thanx for spotting that!

    I saw on Twitter you had this working on Heroku - nice!
  • 3 tzach // May 3, 2013 at 12:24 AM

    Thanks you sir :)
    I'm thinking of wrapping this in a more func way, maybe like
    (def new-fun (relic-wrap old-fun))

    Two point on I learned on Heroku and relic:
    lein read JVM_OPTS from the env, but not JAVA_OPTS.
    Newrelic doc include the second - but till I used the first it didn't work (or is it vise versa? I just included both)

    my .gitignore made sure I will not upload the newrelic.jar, which you need, till I force it to.

    Cheers
    Tzach
  • 4 Ben Poweski // Jun 15, 2013 at 3:39 PM

    We ran through a similar process trying to get instrumentation to work using the custom annotations. Ultimately at getaroom.com we decided it was better to use AOT compilation along with an uberjar which allowed us to use custom XML extensions.

    I wrote up a post on how we did this http://www.bensblog.com/blog/2013/06/15/using-new-relic-with-clojure/.