Anixe.IO.NLog.Extensions 4.0.1

Anixe.IO.NLog.Extensions

The motivation to create this assembly is a need for logging large streams. The NLog itself is dependent on layout renderers system which always produces a string. In consequences, it makes the whole system prone to memory leaks. This assembly intruduces layour formatters system which depends only on stream provided by NLog target.

Layout Formatters

• ILayoutFormatter - the main interface which allows to build custom formatter
• LayoutFormatterBase - the abstract which provides only initialization and disposal routines. Useful to reduce some code.
• ExceptionJsonFormatter - the formatter serializes exception taken from LogEventInfo.Exception into target stream in JSON format.
• StreamFormatter - the formatter copies stream from provided property into the target stream
• TransactionLayoutFormatter - the formatter serializes log message into structured, delimited, one liner message (like CSV) and copies it into stream, it has configuration:
  • Autoflush - FLush after each log entry, default: true
  • LogAllFields - Log into the stream all fields from 'Fields' property regardless if field's value in present in logEvent, default: true
  • Delimiter - Delimit fields with this sign, default: '|'
  • BufferSize - Used in copy data from log into the target stream, default: 4096
  • Encoding - Text encoding used in stream writer initialization, default: utf-8
  • Field - Define which fields must be logged from LogEventInfo.Properties, comma delimited values.

StreamFormatter and TransactionLayoutFormatter formatters expect log message to have everything put in LogEventInfo.Properties and configuration can be provided via the formatter properties

Targets

The assembly contains also predefined targets which works with layout formatters, but you can write your own.

• AsyncFormattedLogFileTarget - writes formatted message into the file stream. Is gains behaviour of https://nlog-project.org/documentation/v4.7.0/html/T_NLog_Targets_AsyncTaskTarget.htm
  • FileName - path to the file (can be layout renderer)
  • FileBufferSize - buffer size passed to file open method (default: 2 << 18)
• FormattedLogFileTarget - a synchronous version of AsyncFormattedLogFileTarget
• AsyncConsoleTargetEx - writes formatted message into the console stream. Is gains behaviour of https://nlog-project.org/documentation/v4.7.0/html/T_NLog_Targets_AsyncTaskTarget.htm
  • Error - if tree then log entries with go to stderr.
• ConsoleTargetEx - a synchronous version of AsyncConsoleTargetEx

TargetWrappers

The wrappers allows you to modify log stream on-the-fly during copy into the target's stream. It is handled by target wrapper for sake of separation of concerns and SPR.

• StreamModifierWrapper - a wrapping target, modifies the message stream according to the set up modifier_attribute collection
  • FieldName - defines the key in LogEventInfo.Properties under which the stream will be present.
  • OneLiner - removed newline char from the stream. Useful when one log entry must fit into single line.
  • Base64 - If true then stream will be encoded into base64 ASCII (default: false).
  • Compress - If true then stream will be compressed (default: false). It has no side effect if the stream is already compressed.
  • CompressionThreshold - all logs below that threshold won't be compressed (default: 1024).
  • Decompress - If true then stream will be decompressed (default: false). It has no side effect if the stream is already decompressed.

It works in sequence oneliner->compress/decompress->encodebase64

Logger Metrics

Internal logger activity is exposed to the application via LoggingMetrics class which provides the following metrics:

• LogsReceived - Incremented by one if AsyncFormattedLogFileTarget or AsyncConsoleTargetEx received any log to write
• LogsWritten - Incremented by one if AsyncFormattedLogFileTarget or AsyncConsoleTargetEx managed to write any bytes
• BytesWritten - Incremented by number of bytes written if AsyncFormattedLogFileTarget or AsyncConsoleTargetEx managed to write any bytes

Configuration

Everything can be configured in XML config.

1. The example configures target to file with TransactionLayoutFormatter configured to format rid,mid,tid,Content-Type,Content-Encoding,data fields with LogTimestamp at first column delimited by | into the target's stream (file in this case). The StreamModifierWrapper will provide the data field (which is expected to be stream) as compressed, base64 encoded message.

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" throwExceptions="true" autoReload="false">

  <extensions>
    <add assembly="Anixe.IO.NLog.Extensions"/>
  </extensions>

  <targets>
    <target type="StreamModifierWrapper" name="formatted_console">
      <modifier_attribute FieldName="data" Base64="true" Compress="true" />
      <target xsi:type="AsyncFormattedLogFileTarget" FileName="/var/log/transaction_logs.dat" FormatterType="Anixe.IO.NLog.Extensions.Formatters.TransactionLayoutFormatter">
        <formatter_attribute name="Fields" values="rid,mid,tid,Content-Type,Content-Encoding,data"/>
        <formatter_attribute name="Encoding" value="utf-8" />
        <formatter_attribute name="BufferSize" value="32000" />
        <formatter_attribute name="Delimiter" value="|" />
        <formatter_attribute name="LogTimestamp" value="true" />
      </target>
    </target>
  </targets>

  <rules>
    <logger name="System.Net.Http.HttpRequestMessage" minlevel="Debug" writeTo="formatted_console" />
    <logger name="System.Net.Http.HttpResponseMessage" minlevel="Debug" writeTo="formatted_console" />
  </rules>
</nlog>

2. The example prints the stream into the console. Is also decompress compressed messages.

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" throwExceptions="true" autoReload="false">

  <extensions>
    <add assembly="Anixe.IO.NLog.Extensions"/>
  </extensions>

  <targets>
    <target type="StreamModifierWrapper" name="console">
      <modifier_attribute FieldName="data" Decompress="true" />
      <target xsi:type="AsyncConsoleTargetEx" name="console" FormatterType="Anixe.IO.NLog.Extensions.Formatters.StreamFormatter">
        <formatter_attribute name="Field" value="data"/>
        <formatter_attribute name="BufferSize" value="32000"/>
      </target>
    </target>
  </targets>

  <rules>
    <logger name="*" minlevel="Trace" writeTo="console" final="true" />
  </rules>
</nlog>

3. Write HTTP messages into the transaction_logs.dat and exceptions into the exception_logs.dat

<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd" 
  xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" throwExceptions="true" autoReload="false">

  <extensions>
    <add assembly="Anixe.IO.NLog.Extensions"/>
  </extensions>

  <targets>
    <target type="StreamModifierWrapper" name="transactions_target">
      <modifier_attribute FieldName="data" Base64="true" Compress="true" />
      <target xsi:type="AsyncFormattedLogFileTarget" FileName="/var/log/transaction_logs.dat" FormatterType="Anixe.IO.NLog.Extensions.Formatters.TransactionLayoutFormatter">
        <formatter_attribute name="Fields" values="rid,mid,tid,Content-Type,Content-Encoding,data"/>
        <formatter_attribute name="Encoding" value="utf-8" />
        <formatter_attribute name="BufferSize" value="32000" />
        <formatter_attribute name="Delimiter" value="|" />
        <formatter_attribute name="LogTimestamp" value="true" />
      </target>
    </target>


    <target xsi:type="AsyncFormattedLogFileTarget" name="exception_target" FileName="/var/log/exception_logs.dat" FormatterType="Anixe.IO.NLog.Extensions.Formatters.ExceptionJsonFormatter" />
  </targets>

  <rules>
    <logger name="System.Net.Http.HttpRequestMessage" minlevel="Debug" writeTo="transactions_target" />
    <logger name="System.Net.Http.HttpResponseMessage" minlevel="Debug" writeTo="transactions_target" />
    <logger name="*Exception*" minlevel="Debug" writeTo="exception_target" />
  </rules>
</nlog>

FormattersFactory

Its worth to mention about the factory pattern which is responsible for creating layout formatters for configured targets because these entities don't belong to NLog's world. Check the FormattersFactory for details. Please set FormattersFactory.CreateInstanceFunc custom function if you need to resolve layout formatters from own DI container.