Wednesday 6 January 2016

Logging Abstraction Layer - Take 2

As I did on the previous post, I'll begin by saying I won't go into another "My, how long it has been" post. I do, however, hope I'm not too late to express my Best Wishes for 2016.

Now, then...

I'm having another iteration on my idea for an abstraction layer for logging libraries, which is basically fancy-speak for “a collection of (hopefully, organized) macros” that will allow me to replace logging libraries with minimal effort.

I've been going back and forth between Boost Log and Poco Logger. I like both, even though I tend to prefer Poco Logger, because I haven't yet found a satisfactory solution for the issue with Boost Log's preference for truncating an existing log file when the application is restarted.

I've hit some maintenance problems with my “collection of macros”, so I decided to give Boost Preprocessor another go. Back when I began working on this idea, I looked into Boost PP, but got caught in a blind spot, probably a faulty reaction to all the metaprogramming examples. I had the feeling I could use it, but I wasn't able to make it work.

So, I rolled my own and have since been banging my head against several walls, thus gaining the required experience to make a number of questions I didn't even know were there. Which gave me a different perspective when I approached Boost Preprocessor again, and things went much better this time.

I quickly set up a PoC, and then I hit the issue I'm looking at now – How to format the log record. This is only somewhat important for readability, but it's crucial for automation. Without a structured format, we'll have a hard time creating tools to operate on the logs.

My initial design, which I haven't changed, treats the logging statement as a sequence of values to output. E.g.,

SOMEPREFIX_DEBUG(object_id, discount_id, “Getting object price”);

But a structured format requires other artifacts – at a minimum, we need delimiters. And an obvious requirement for some sort of intelligent automatism is that the nature of the values must be identified, i.e., we need to know if an element logged as an integer is, say, a total or an ID; so, we need to decorate values with names.

Boost Log gives us attributes, values which are added to the log record without the need to output them explicitly on the logging statements. Furthermore, attributes can be used in filtering and formatting, and can be scoped. I suppose there may be even performance benefits in the use of attributes, but that's something I'd have to measure.

At first, I thought this could be the way to go. Define attributes, create formatters or override the output operator, and control the way the record is written to file.

However…

Poco Logger does not have this concept. And while I'm not considering any other logging library at the moment, that may change in the future. And, as I said above, my main goal is to “replace logging libraries with minimal effort”. So, if this is my main goal, it is important enough to warrant an under-utilization of the library.

Naturally, if I don't use the library's capabilities to output this data, I'll need to output it on the logging statements. This means that formatting will also have to happen there. And since I'm already using the preprocessor as an abstraction mechanism, I'll just… use it some more.

So, I already have something like this:

SOMEPREFIX_DEBUG(object_id, discount_id, “Getting object price”);

What I need is something along these lines (simplified):

SOMEPREFIX_DEBUG
(
    SOMEPREFIX_FORMAT(“ObjectID”, object_id),
    SOMEPREFIX_FORMAT(“DiscountID”, discount_id), 
    “Getting object price”
);

Which could, then, output something like this (line breaks for readability):

2016-01-01 00:00:00 <DEBUG> 
    { "ObjectID": "42", “DiscountID”: “24” } Getting object price

Then, one day, if we decided life had become too easy, we could switch formats to something like this, without changing the logging statements:

2016-01-01 00:00:00 [DEBUG]
    <LogRecord><ObjectID>42</ObjectID>
    <DiscountID>24</DiscountID></LogRecord>
    Getting object price

So, the next step will be identifying the required formatting elements, and how to incorporate them in the logging statements. The goal here is to keep the logging statements as simple as possible.

On this iteration, I will leave out any sort of container/compound data type. Not only will these make the design a lot more complex, but I'm prepared to do without them – in my experience, I have found very few scenarios requiring the logging of these types, and it has always been possible to find a workaround somewhere between acceptable and undesirable.