How to Enable Structured Logging

Anne Ramey

Good news! There is nothing to do for a brand new install. The log4j-default.xml file that comes with Kinetic Request CE located in the WEB-INF/config directory is already configured to start writing to the structured log files.

Upgrading Kinetic Request CE

If you have customized the log4j.xml or log4j-default.xml file, you'll need to append the following text in your log4j.xml file if you wish to enable the new structured logs. These lines can be added anywhere inside the >log4j:configuration ...< XML element in your customized log4j xml file, but we suggest adding it before the >!-- LOGGERS --< line. For example of an example of where to place the additional lines, take a look at the new WEB-INF/config/log4j-default.xml file that comes with the 2.0.2 Kinetic Request CE war file.

    <!-- Structured Access File Appender -->
    <appender name="structured-access-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.access.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} - %X{access.requestMethod} %X{access.requestPath}%X{access.requestQuery} %X{access.responseStatus} %X{access.responseTime}ms%n"/> 
        </layout> 
    </appender>
    
    <!-- Structured Authentication File Appender -->
    <appender name="structured-authentication-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.authentication.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} - %X{auth.event} %X{auth.strategy} PRINCIPAL:%X{auth.principal} %m%n"/>
        </layout> 
    </appender>
    
    <!-- Structured Application File Appender -->
    <appender name="structured-application-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.application.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.space} %X{app.user} %X{access.addressChain} %5p [%t] - %m%n"/> 
        </layout>
        <filter class="com.kineticdata.commons.logging.ContextFilter">
            <param name="KeyToMatch" value="app.requestId" />
            <param name="PatternToMatch" value=".+" />
        </filter>
    </appender>
    
    <!-- Structured System File Appender -->
    <appender name="structured-system-file-appender" class="org.apache.log4j.RollingFileAppender"> 
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.core.dataDirectory}/logs/structured.system.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %5p [%t] - %m%n"/> 
        </layout>
        <filter class="com.kineticdata.commons.logging.ContextFilter">
            <param name="KeyToMatch" value="app.requestId" />
            <param name="PatternToMatch" value="^$" />
        </filter>
    </appender>

Finally, restart Tomcat or the Kinetic Request CE web application.

CE Structured Logging Variables

Overview

The structured logging variables are variables that are available to use to format a log message in a format you want. The sub-section 'Upgrading Kinetic Request CE' above shows a sample of how these variables, which are highlighted, are used in the log4j-default.xml and log4j.xml files to configure the output format in the structured.access.log, structured.application.log, structured.auth.log, and structured.system.log files.

Variable Definitions

|---|---| | access.addressChain | Either the IP addresses listed in the X-Forwarded-For HTTP header or the IP address establishing the request to Kinetic Core if no X-Forwarded-For HTTP header is present. Examples for the IP address establishing the request include a load balancer's IP address, or the client's IP address if no load balancer or reverse proxy is in front of Kinetic Request. | | access.requestMethod | The HTTP request method used for the access request | access.requestPath | Portion of the request uri after the host and before the query string, ie /kinetic/app/api/v1/version | | | access.request | Protocol Protocol used for the request - Example: HTTP/1.1 | | access.request | Query Provides the request query string, starting with “?”. The request query string is the part of a URL proceeding the request path and the start of the request query is indicated by the question mark character. | | access.response | Status The HTTP response code returned from Kinetic Request. | | access.response | Time The amount of time taken to process the request on the server side in milliseconds. This does not include any network latency. | | app.correlationId | This is an id that is generated for each end user HTTP request, passed between kinetic components to associate internal calls | | app.instanceId | A consistent key based upon the server’s network interfaces and webapp location, can be used when aggregating logging to determine the source of the log message | | app.requestId | A unique ID generated for each received HTTP request | | app.sessionId | A user's session id or NOSESSION for sessionless interaction; such as HTTP Basic API calls | | app.space | The space slug for which the logged event occurred within | | app.user | The username of the user associated to the logged event | | auth.event | This is the type of the authentication event that occurred. Valid options are LOGIN, LOGINFAILURE, LOGOUT, CHECK, EXPIRE. |

Because the application always allows local authentication (default CE form based authentication) as a fallback, there will only be LOGINFAILURE messages when all of the configured security strategies fail AND local authentication fails (this means pre-authentication strategies like x509 will never have LOGINFAILURE messages). CHECK represents an unsuccessful authentication check for a security strategy which indicates the next security strategy will be tried auth.principal The authentication principal is the unique identifier from an authentication strategy for a user, before they're mapped to a local CE username. auth.strategy The authentication strategy used for this logged event. Some example strategies are: ldap, ldap/basic, local, local/basic, x509, saml