Skip to main content
Kinetic Community

Structured Logging with Kinetic Bridgehub and Filehub

Overview

Starting with Kinetic Bridgehub v1.2.0 and Kinetic Filehub v1.2.0 a new feature was implemented referred to as structured logging. Structured logging essentially is ensuring that a log file is a preset, consistent, and machine readable format. Structured logs are useful for enterprise log aggregation tools like Splunk, Graylog, or Elastic.


We here at Kinetic Data leveraged this feature for our kinops.io  SaaS offering. Take a look at this brief blog article on the benefits we found by using structured logging with Elasticsearch. If you're interested in integrating with Elasticsearch as well for log aggregation, see our community article Elasticsearch and Filebeat integration with Kinetic Structured Logs. That article will go over how to use Filebeat to read our log files and send them over to Elasticsearch.
 

The structured logs consist of the following files in the %DATA_DIR%/logs directory -

  • structured.access.log
    • Logs an entry for every time a resource is accessed through the kinetic application. Who, when, how long, what, etc. This can be used for troubleshooting, auditing, or analytics.
  • structured.application.log
    • Contains entries for application warnings, errors, debug, or trace level messages. Used for troubleshooting.
  • structured.authentication.log
    • Authentication attempts get logged to this file. Who, when, and authentication type. Used for troubleshooting and auditing.
  • structured.system.log
    • Application startups and shutdowns, and other non-frequent events like that are the purpose for this log file.
  • heartbeat.log
    • Logs an entry once a minute for the duration of the application running. Useful for knowing if the application is still running as expected while only needing to check the logs.

 

Enabling Structured Logging  

Brand new install

Good news! There is nothing to do for a brand new install. The log4j-default.xml file that comes with Kinetic Bridgehub and Filehub located in the WEB-INF/config directory is already configured to start writing to the structured log files.
 

Upgrading Kinetic Bridgehub and Filehub

If you have customized the log4j.xml or log4j-default.xml file, you'll need to append the following text in your log4j.xml file if you wish to enable the new structured logs. These lines can be added anywhere inside the <log4j:configuration ...> XML element in your customized log4j xml file, but we suggest adding it before the <!-- LOGGERS --> line. For example of an example of where to place the additional lines, take a look at the new WEB-INF/config/log4j-default.xml file that comes with the 1.2.0 Kinetic Bridgehub and Filehub war files.

Structured Log Appenders

    <!-- Heartbeat File Appender -->
    <appender name="heartbeat-file-appender" class="org.apache.log4j.RollingFileAppender">
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.filehub.dataDirectory}/logs/heartbeat.log"/>
        <param name="MaxFileSize" value="1MB"/>
        <param name="MaxBackupIndex" value="2"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d{ISO8601} %m%n"/>
        </layout>
    </appender>
    
    <!-- Structured Access File Appender -->
    <appender name="structured-access-file-appender" class="org.apache.log4j.RollingFileAppender">
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.bridgehub.dataDirectory}/logs/structured.access.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.user} %X{access.addressChain} - %X{access.requestMethod} %X{access.requestPath}%X{access.requestQuery} %X{access.responseStatus} %X{access.responseTime}ms%n"/>
        </layout>
    </appender>
    
    <!-- Structured Authentication File Appender -->
    <appender name="structured-authentication-file-appender" class="org.apache.log4j.RollingFileAppender">
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.bridgehub.dataDirectory}/logs/structured.authentication.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.user} %X{access.addressChain} - %X{auth.event} %X{auth.strategy} PRINCIPAL:%X{auth.principal} %m%n"/>
        </layout> 
    </appender>
    
    <!-- Structured Application File Appender -->
    <appender name="structured-application-file-appender" class="org.apache.log4j.RollingFileAppender">
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.bridgehub.dataDirectory}/logs/structured.application.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %X{app.sessionId} %X{app.correlationId} %X{app.requestId} %X{app.user} %X{access.addressChain} %5p [%t] - %m%n"/> 
        </layout>
        <filter class="com.kineticdata.commons.logging.ContextFilter">
            <param name="KeyToMatch" value="app.requestId" />
            <param name="PatternToMatch" value=".+" />
        </filter>
    </appender>
    
    <!-- Structured System File Appender -->
    <appender name="structured-system-file-appender" class="org.apache.log4j.RollingFileAppender">
        <param name="Append" value="true"/>
        <param name="File" value="${com.kineticdata.bridgehub.dataDirectory}/logs/structured.system.log"/>
        <param name="MaxFileSize" value="20MB"/>
        <param name="MaxBackupIndex" value="10"/>
        <layout class="org.apache.log4j.PatternLayout"> 
            <param name="ConversionPattern" value="%d{ISO8601} %X{app.instanceId} %5p [%t] - %m%n"/> 
        </layout>
        <filter class="com.kineticdata.commons.logging.ContextFilter">
            <param name="KeyToMatch" value="app.requestId" />
            <param name="PatternToMatch" value="^$" />
        </filter>
    </appender>

Using the Appenders

<!-- Kinetic Filehub Application Logger -->
<logger name="com.kineticdata.filehub" additivity="false">
    <!-- Level is reset once the application is loaded based on configuration properties. -->
    <level value="info"/>
    <appender-ref ref="application-appender"/>
    <appender-ref ref="structured-application-file-appender"/>
    <appender-ref ref="structured-system-file-appender"/>
</logger>
    
<!-- Kinetic Clustering Logger -->
<logger name="com.kineticdata.clustering" additivity="false">
    <!-- Level is reset once the application is loaded based on configuration properties. -->
    <level value="info"/>
    <appender-ref ref="application-appender"/>
    <appender-ref ref="structured-system-file-appender"/>
</logger>
    
<!-- Disable specialized loggers unless they are explicitly opted in to. -->
<logger name="com.kineticdata.filehub.loggers" additivity="false">
    <level value="off" />
</logger>
    
<!-- Kinetic Filehub Access Logger -->
<logger name="com.kineticdata.filehub.loggers.access" additivity="false">
    <level value="info"/>
    <appender-ref ref="structured-access-file-appender"/>
</logger>

<!-- Kinetic Filehub Authentication Logger -->
<logger name="com.kineticdata.filehub.loggers.authentication" additivity="false">
    <level value="info"/>
    <appender-ref ref="structured-authentication-file-appender"/>
</logger>
    
<!-- Kinetic Filehub Heartbeat Logger -->
<logger name="com.kineticdata.filehub.loggers.heartbeat" additivity="false">
    <level value="info"/>
    <appender-ref ref="heartbeat-file-appender"/>
</logger>

Finally, restart Tomcat or the Kinetic Bridgehub or Filehub web application.
 

Bridgehub/Filehub Structured Logging Variables

Overview

The structured logging variables are variables that are available to use to format a log message in a format you want. The sub-section 'Upgrading Kinetic Bridgehub and Filehub' above shows a sample of how these variables, which are highlighted, are used in the log4j-default.xml and log4j.xml files to configure the output format in the structured.access.log, structured.application.log, structured.auth.log, and structured.system.log files.
 

Variable Definitions

access.addressChain Either the IP addresses listed in the X-Forwarded-For HTTP header or the IP address establishing the request to Kinetic Bridgehub/Filehub if no X-Forwarded-For HTTP header is present. Examples for the IP address establishing the request include a load balancer's IP address, or the client's IP address if no load balancer or reverse proxy is in front of the application.
access.requestMethod The HTTP request method used for the access request
access.requestPath Portion of the request uri after the host and before the query string, ie /kinetic-bridgehub/app/api/v1/bridges/bridgeSlug/search
access.requestProtocol Protocol used for the request - Example: HTTP/1.1
access.requestQuery Provides the request query string, starting with “?”. The request query string is the part of a URL proceeding the request path and the start of the request query is indicated by the question mark character.
access.responseStatus The HTTP response code returned from Kinetic Bridgehub/Filehub.
access.responseTime The amount of time taken to process the request on the server side in milliseconds. This does not include any network latency.
app.correlationId This is an id that is generated for each end user HTTP request, passed between kinetic components to associate internal calls
app.instanceId A consistent key based upon the server’s network interfaces and webapp location, can be used when aggregating logging to determine the source of the log message
app.requestId A unique ID generated for each received HTTP request
app.sessionId A user's session id or NO_SESSION for sessionless interaction; such as HTTP Basic API calls
app.user The username of the user associated to the logged event
auth.event This is the type of the authentication event that occurred. Valid options are LOGIN, LOGIN_FAILURE, LOGOUT, EXPIRE
auth.principal The authentication principal is the unique identifier from an authentication strategy for a user (for Bridgehub/Filehub, this is always the local user's username)
auth.strategy The authentication strategy used for this logged event (for Bridgehub/Filehub, this is always 'local')