New Relic Offers Lucene and Solr Performance Monitoring

Two technologies that many sites and services rely on these days are web applications and search. Those relying heavily on web apps plus Apache Lucene (news, site) and Apache Solr (news, site) now have a new tool available to help them track down and identify performance problems within their sites, thanks to a company called New Relic.

Who's New Relic?

New Relic offers SaaS web application profiling and management. Their product consists of a collection of user agents that can be installed into web applications. Once installed, they collect performance information while transactions work their way through the applications. This data is sent once a minute to New Relic's servers, which aggregate the data for the customer, who can then view the incoming metrics on a variety of devices.

New Relic's SaaS product showing web application troubleshooting information.

In aggregating statistics across all of their customers, about six months ago the company noticed something interesting happening with both Ruby on Rails and Java deployments. The number of Solr instances to handle search for applications was rising and displacing Google Search Appliance, often for e-commerce and social networking.

Deconstructing Solr Problems

Search is a critical site feature, especially in the world of Web Content Management. If your search is slow or poor, people will quickly give up and move on to your competitors. Since such a large portion of their customer base is using Apache Solr, New Relic offers specialized dashboards that appear as a user drills down to view the search application's performance.
Rather than displaying generic performance information, these dashboards show Solr cache wait times, how much memory and CPU is used for each search, what the longest searches are, which the best and worst performing are, and more. The ability to drill down into individual searches lets developers determine what's causing the problems, such as inefficiently-built queries.
New Relic's SaaS product changes automatically to show Solr troubleshooting information in a different format.

Giving Back

New Relic's product is not open source, though Mike Malloy, their VP of Marketing, stated that they're looking at going in this direction in the future. Perhaps that's because they've noticed that their customers are almost entirely using completely open source stacks, with the occasional proprietary components here and there such as IBM's WebSphere or Oracle's WebLogic.

The State of the Stack

One thing that New Relic's customers have found is that their issues aren't always with their own code. Sometimes the problems stem from one of the open source components they're using. In many cases, the projects behind web application components don't hear back from the users. According to Malloy, receiving real performance data from production servers has brought enthusiastic response from the projects.