Is your logging a complete waste of everyone's time?

in Software Engineering, DevOps

Most people are first introduced to logging in the course of debugging errors. Usually because whatever they're debugging does not have access to standard out. The logging is kept for production because it is useful. From there the logging of errors and warnings grows because it is a good thing. In reality the usefulness of such logs is marginal and at worst they are an outright waste of time. The logs are likely scattered across servers rather than centralized, giving a patchy overview. Worse, each log is a wall of text of generic messages and stack traces.

"If debugging is the process of removing software bugs, then programming must be the process of putting them in." - Edsger Dijkstra

Another essential log people encounter is the cryptic output of IIS. A confusing jumble of bits of URL, browser agent strings and random numbers. For a busy site an individual server's log for a day can be hundreds of thousands of lines long. Consulting this log is the last leg of a journey that started with, "We have no idea what's happening."

OMG someone kill me now!

So, is logging a wholly pointless activity? No. In the previous examples the core principle is still valid. We want to record errors and IIS requests. Both are rich structured events that give us important insights and valuable information. When they are gutted to fit the limitations of text file formats we lose that. This is the crux of the problem; text files are just not a good medium for storing such events.

Many alternatives to text files exist. Document databases are perfect for storing events in a structured form such as JSON or XML. Their ability to query and analyze data improves at a steady pace. Elasticsearch is a document database built around flexible full text search. It offers many benefits such as scalability, ease of set up, a strong community and it ships with many different text analyzers.

"It has been said that XML is like violence; if a little doesn’t solve the problem, use more." - Anon.

Storing data in a structured form is only half the story. We also need to be able to analyze and visualize it. In the world of numbers, Excel is the go to tool for visualizing data. In the world of structured logging Kibana is its closest analogue. It can work with data from a variety of sources but is particularly suited to Elasticsearch. The famous ELK Stack consists of Elasticsearch, Kibana and Logstash. Logstash is an excellent and powerful tool beyond the scope of this post but you should go and read about it.

By subtle application of the right tool for the right job principle, logs are now useful again. We should be analyzing them earlier and more often without fear. We should also strive to unlearn our last-resort-of-the-desperate mentality.

"Fear is the path to the Dark Side. Fear leads to anger, anger leads to hate, hate leads to suffering." - Yoda

Analysis and visualization extends the usefulness of structured logging beyond handling errors and incidents. Imagine logging any time someone searches, buys a product or shares a link to an article on a social network. You are now into the realm of business intelligence. Stake holders and business analysts can gather the metrics they need to form hypotheses. These hypotheses drive design decisions and work priorities for your site. Measuring the impact of changes made validates the hypotheses.

This data is available in real time. Kibana provides functionality to produce real time dashboards. It is also simple to poll the data from your preferred dashboard framework eg dashing. Being able to see the impact on key performance indicators (KPIs) as soon as you make changes is awesome. Display these dashboards next to your developers. Make them available to stake holders.

In summary, structured logging:

  • makes developers' lives easier
  • provides rapid feedback to stakeholders they didn't even know was possible
  • validates that you are building the right thing