Skip to main content
Kinetic Community

Structured Logging

This article covers the basics on structured logging within Kinetic Request CE

Overview

Starting with Kinetic Core v2.0.2 a new feature was implemented referred to as structured logging. Structured logging essentially is ensuring that a log file is a preset, consistent, and machine readable format. Structured logs are useful for enterprise log aggregation tools like Splunk, Graylog, or Elastic.


We here at Kinetic Data leveraged this feature for our kinops.io  SaaS offering. Take a look at this brief blog article on the benefits we found by using structured logging with Elasticsearch. If you're interested in integrating with Elasticsearch as well for log aggregation, see our community article Elasticsearch and Filebeat integration with Kinetic Structured Logs. That article will go over how to use Filebeat to read our log files and send them over to Elasticsearch.
 

The structured logs consist of the following files in the %DATA_DIR%/logs directory -

  • structured.access.log
    • Logs an entry for every time a resource is accessed through the kinetic application. Who, when, how long, what, etc. This can be used for troubleshooting, auditing, or analytics.
  • structured.application.log
    • Contains entries for application warnings, errors, debug, or trace level messages. Used for troubleshooting.
  • structured.authentication.log
    • Authentication attempts get logged to this file. Who, when, and authentication type. Used for troubleshooting and auditing.
  • structured.system.log
    • Heartbeat checks, application startups and shutdowns, and other non-frequent events like that are the purpose for this log file.

 

Enabling Structured Logging  

Brand new install

Good news! There is nothing to do for a brand new install. The log4j-default.xml file that comes with Kinetic Request CE located in the WEB-INF/config directory is already configured to start writing to the structured log files.
 

Upgrading Kinetic Request CE

If you have customized the log4j.xml or log4j-default.xml file, you'll need to append the following text in your log4j.xml file if you wish to enable the new structured logs. These lines can be added anywhere inside the <log4j:configuration ...> XML element in your customized log4j xml file, but we suggest adding it before the <!-- LOGGERS --> line. For example of an example of where to place the additional lines, take a look at the new WEB-INF/config/log4j-default.xml file that comes with the 2.0.2 Kinetic Request CE war file.

    <!-- Structured Access File Appender -->
    <appender name="structured-access-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.access.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} - %X{access.requestMethod} %X{access.requestPath}%X{access.requestQuery} %X{access.responseStatus} %X{access.responseTime}ms%n"/> 
        </layout> 
    </appender>
    
    <!-- Structured Authentication File Appender -->
    <appender name="structured-authentication-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.authentication.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} - %X{auth.event} %X{auth.strategy} PRINCIPAL:%X{auth.principal} %m%n"/>
        </layout> 
    </appender>
    
    <!-- Structured Application File Appender -->
    <appender name="structured-application-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.application.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} %5p [%t] - %m%n"/> 
        </layout>
        <filter class="com.kineticdata.commons.logging.ContextFilter">
            <param name="KeyToMatch" value="app.requestId" />
            <param name="PatternToMatch" value=".+" />
        </filter>
    </appender>
    
    <!-- Structured System File Appender -->
    <appender name="structured-system-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.system.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %5p [%t] - %m%n"/> 
        </layout>
        <filter class="com.kineticdata.commons.logging.ContextFilter">
            <param name="KeyToMatch" value="app.requestId" />
            <param name="PatternToMatch" value="^$" />
        </filter>
    </appender>

Finally, restart Tomcat or the Kinetic Request CE web application.
 

CE Structured Logging Variables

Overview

The structured logging variables are variables that are available to use to format a log message in a format you want. The sub-section 'Upgrading Kinetic Request CE' above shows a sample of how these variables, which are highlighted, are used in the log4j-default.xml and log4j.xml files to configure the output format in the structured.access.log, structured.application.log, structured.auth.log, and structured.system.log files.
 

Variable Definitions

access.addressChain Either the IP addresses listed in the X-Forwarded-For HTTP header or the IP address establishing the request to Kinetic Core if no X-Forwarded-For HTTP header is present. Examples for the IP address establishing the request include a load balancer's IP address, or the client's IP address if no load balancer or reverse proxy is in front of Kinetic Request.
access.requestMethod The HTTP request method used for the access request
access.requestPath Portion of the request uri after the host and before the query string, ie /kinetic/app/api/v1/version
access.requestProtocol Protocol used for the request - Example: HTTP/1.1
access.requestQuery Provides the request query string, starting with “?”. The request query string is the part of a URL proceeding the request path and the start of the request query is indicated by the question mark character.
access.responseStatus The HTTP response code returned from Kinetic Request.
access.responseTime The amount of time taken to process the request on the server side in milliseconds. This does not include any network latency.
app.correlationId This is an id that is generated for each end user HTTP request, passed between kinetic components to associate internal calls
app.instanceId A consistent key based upon the server’s network interfaces and webapp location, can be used when aggregating logging to determine the source of the log message
app.requestId A unique ID generated for each received HTTP request
app.sessionId A user's session id or NO_SESSION for sessionless interaction; such as HTTP Basic API calls
app.space The space slug for which the logged event occurred within
app.user The username of the user associated to the logged event
auth.event This is the type of the authentication event that occurred. Valid options are LOGIN, LOGIN_FAILURE, LOGOUT, CHECK, EXPIRE.

Because the application always allows local authentication (default CE form based authentication) as a fallback, there will only be LOGIN_FAILURE messages when all of the configured security strategies fail AND local authentication fails (this means pre-authentication strategies like x509 will never have LOGIN_FAILURE messages). CHECK represents an unsuccessful authentication check for a security strategy which indicates the next security strategy will be tried
auth.principal The authentication principal is the unique identifier from an authentication strategy for a user, before they're mapped to a local CE username.
auth.strategy The authentication strategy used for this logged event. Some example strategies are: ldap, ldap/basic, local, local/basic, x509, saml