Choosing a Central Logging Tool: 5 Important Features, 6 Optional Tools

This post is brought to you by Ravello R&D based on our own internal best practices and lessons learned. We need to continually monitor our servers, production and dev environments. As our environments grow and scale out it becomes increasingly difficult to debug failures and crisis analysis requires multi SSH-ing to different servers. Therefore we wanted to be able  to view all the logs for all our servers from one single entry point. We also wanted to be notified of abnormal activity in our logs, because we can’t sit and watch them all day long.

Centralized use of logging tools, with their many features, allows Ops teams to analyze the root cause of a crisis in infrastructure and helps DevOps teams to easily analyze and troubleshoot production/development issues.  To this end, we surveyed the capabilities of the available tools before choosing a tool for us.  Here is what we found.

5 Common Features of Central Logging Tools

  1. Log collection: From any static/on-demand resource to a single secured sign-in application, accessible from anywhere (not only from our own VPN).
  2. Alerts: The stakeholders of any specific exceptions in logs receive notifications based on configurable criteria, helping detect issues in production even before a user complains about them.
  3. Aggregation: Scaled-out servers behind load balancers each produce their own log files, making it impossible to debug a single action flow that distributed between servers, unless the logs converge into a single article.
  4. History: Keeping old logs can be very helpful when trying to understand why and when a specific product behavior began.
  5. Visual indicators: Abnormal behaviors can be detected faster when we see them in a visual instrument such as a graph, where peak points are easily noticed.

6 Popular Log Tools

Most of the tools available are based on configuring the syslog on the required server to send data to the remote applications that handle them. Here are examples of some of the popular tools:

  • Splunk Storm – Provides cloud-based operation analysis and troubleshooting for your application. Splunk storm supports multiple integration,  with applications such as AWS and Heroku. It also allows searching, visualizing, and sharing logs and monitoring data. www.splunkstorm.com
  • Graylog – An open source self-hosted application allowing to search logs, create charts and reports, add alerts for incidents. graylog2.org
  • Sumlogic – Collects, centralizes, alerts, and visualizes logs. A cloud-based SaaS app, it  requires agent on designated servers. Sumlogic includes  “Prediction” features to detect issues before they arise. www.sumologic.com
  • Logentries – A cloud-based SaaS application , a simple and powerful tool to search, tag, alert, and track log data from a single location. Logentries supports integration to aws cloudWatch and Heroku. www.logentries.com
  • Papertrail – A clean and simple cloud-based SaaS application that collects and aggregates  logs from multiple sources. It provides powerful search capabilities, alerting, and visual indicators, as well as alert integrations into HipChat, PagerDuty and more. It  features simple and secured configuration and setup, an intuitive UI, and API. www.papertrailapp.com

Bonus: another very strong player in the field is Elasticsearch + Logstash, providing fast log searching from a central source. Also Kibana can be used to visualize the data.

Final Notes

We sought  a simple tool that provides just what we need, that is centralizing all our servers (production and development environments) in a single place, where we can just log in and start debugging. A tool that would alert us for special events in the server logs, and allow secure  but simple implementation for each server.  We chose Papertrail.

The pricing was fairly reasonable, for about 50G of log data saved for 2 weeks, and archived for 1 year, for both our dev/prod accounts that are separated but easily switched between.

At Ravello Systems, because we deploy many on-demand cloud VMs it is important for us to monitor and access logs. We were able to easily set up a simple configuration on each server as it goes live – so we can immediately track its behavior, detect issues, and receive alerts.


Watch the video below and learn more about Ravello Systems

All systems

About Ravello Systems

Ravello is the industry’s leading nested virtualization and software-defined networking SaaS. It enables enterprises to create cloud-based development, test, UAT, integration and staging environments by automatically cloning their VMware-based applications in AWS. Ravello is built by the same team that developed the KVM hypervisor in Linux.

Check our product demo video
2 minutes product demo

Choosing a Central Logging Tool: 5 Important Features, 6 Optional Tools
David Goldberger

By David Goldberger

David Goldberger is a software engineer on the Ravello Systems DevOps team. He specializes in implementing and integrating processes and delivering fully automatic E2E CI/CD/Test/Monitoring systems.
More posts from David Goldberger
  • Richard Pijnenburg

    Bit disappointed that Logstash isn’t included in this list.

    • http://www.ravellosystems.com/ D Gold

      Richard, you should not be disappointed :), we included only tools/services we POC-ed before selecting the one that best fits *US*, apparently we were not introduced with logstash neither recommended of, back than, that does not mean it’s not “good enough” or any other assumption. JIC – we added a bonus bullet, so that readers will have links for this strong service as well.
      thanks for your comment.

  • bzhtux

    hi, link to sumo logic is not ok : http://www.sumologic.comb remove the trailing ‘b’ in the tld.

    • http://www.ravellosystems.com/ Ravello Systems

      Fixed now. Thanks @bzhtux:disqus!

  • splunkninja

    Wouldnt Splunk be more appropriate than Splunk Storm?

    • http://www.ravellosystems.com/ D Gold

      ninja, thanks for your comment, as we saw it while perusing our best fitted solution, Splunk (and i guess you meant Enterprise), was a bit of an overkill for what we really needed – just to centralize and easily navigate in our distributed machines logs. Storm was much more suited solution for our basic requirement, and was in the same competitive level with papertrail and the others…

  • http://matthewskelton.net/ Matthew Skelton

    Echoing Richard P, why is LogStash (or ELK, ElasticSearch + LogStash + Kibana) not included in this list?

    • http://www.ravellosystems.com/ D Gold

      Echoing my reply ;).
      thanks.

  • Yaniv Wainer

    Thanks @disqus_9NHFRV4uM0:disqus for the info!

  • OohLaLog

    Great article, David. We also just released OohLaLog to beta. It’s a cloud-based solution that uses ElasticSearch.

  • Pingback: Choosing a Central Logging Tool: 5 Important Fe...()