Skip to main content
Kinetic Community

Elasticsearch Adapter

Overview

The Elasticsearch Adapter for Bridgehub is an adapter that allows retrieving indexed documents from Elasticsearch to be pulled back using bridging.

The examples on this page for field or qualification mapping use the example data sets from Elasticsearch.

Installation

Note: Check to ensure that this adapter isn't already installed into your bridgehub as part of the base installation before undertaking the installation tasks.

  • Download the kinetic-bridges-adapter-elasticsearch.zip file here
  • Put the jar file into the WEB-INF/lib folder for you kinetic-bridgehub installation
  • Restart the tomcat that your kinetic-bridgehub installation is located in

Setup

Configuration Values

Name Description
Elastic URL URL to the Elasticsearch instance.
Username Optional username to specify for basic authentication if your Elasticsearch implementation requires it.
Password Optional password to specify for basic authentication if your Elasticsearch implementation requires it.

Example Configuration

Name Value
Elastic URL https://elasticsearch.yourdomain.com:9200/
Username  
Password  

Structures, Fields, and Queries

Structures

  • Any valid Elasticsearch URI search options for searching across indices or types or both.
  • Examples
    • _all/tweets
    • index1,index2/tweets
    • twitter/users,tweets
    • twitter

Fields

  • JsonPath is used for mapping field values. The default JsonPath root path is set to $.hits.hits. Both bracket, e.g. $['hits']['hits'] , and dot notation, e.g. $.hits.hits , are supported. A different JsonPath root path can be specified in the qualification mapping, please see the Queries section below for details. Changing the root path is not necessary unless you're using the Elasticsearch DSL query type and your Elasticsearch JSON query is complicated / uses aggregation.

    If the root path evaluates to an array (such as the default root path $.hits.hits) then this as seen as matching multiple records if the array size is greater than one. If the root path evaluates to an object instead of an array then this as seen as matching only one result for the bridge.
     
  • If you'd like to practice using JsonPath, you can use the jsonpath online evaluator webpage

Shakespeare data set examples:

  • ${fields('_source.line_id')}
  • ${fields('_.source.play_name')}

Accounts data set examples:

  • ${fields('_source.account_number')}
  • ${fields('_source.balance')}

Qualification Mapping

The following qualification mapping syntax options are supported:

  1. Lucene query syntax.
  2. A JSON Object.

 

  1. Lucene query syntax
    1. This is the most straight forward qualification mapping option.
    2. The query syntax directly maps to the Elasticsearch Query String Syntax
    3. All parameter values have the Elasticsearch reserved characters escaped. This is to protect against any input parameters possibly changing the structure or the intentions of a bridge query.
    4. Example accounts data set qualification mappings:
      1. Find all accounts where the clients age is in a specified range:
        1. (age:>${parameters('lower age limit')} AND age:<${parameters('upper age limit')})
      2. Find all accounts where the client is in a selected state and has a street address with the word Court in it.
        1. state:${parameters('favorite state')} AND address:Court
           
  2. JSON Object
    • Example Shakespeare data set qualification mappings:

      • mapping qualification: {"type": "Kinetic DSL", "query": "${parameters('kinetic dsl json')}"}
        ${parameters('kinetic dsl json')} : {"speaker": {"value": "HENRY", "matcher": "like"}, "text_entry": {"value": "thou", "matcher": "exact"}}
        dynamic Lucene query produced: speaker:*HENRY* AND text_entry:thou
         

      • mapping qualification: {"type": "Kinetic DSL", "concateOperator": "OR", "query": "${parameters('kinetic dsl json')}"}
        ${parameters('kinetic dsl json')} : {"speaker": {"value": "HENRY", "matcher": "like"}, "text_entry": {"value": "thou", "matcher": "exact"}}
        dynamic Lucene query produced: speaker:*HENRY* OR text_entry:thou
         

      • mapping qualification: {"type": "Kinetic DSL", "whitelistFields": ["speaker", "text_entry"], "query": "${parameters('kinetic dsl json')}"}
        ${parameters('kinetic dsl json')} : {"speaker": {"value": "HENRY", "matcher": "like"}, "text_entry": {"value": "thou", "matcher": "exact"}, "play_name": {"value": "Hamlet", "matcher": "exact"}}
        dynamic Lucene query produced: speaker:*HENRY* AND text_entry:thou
         

      • mapping qualification: {"type": "Kinetic DSL", "queryPrefix": "play_name:Henry\ IV", "query": "${parameters('kinetic dsl json')}"}
        ${parameters('kinetic dsl json')} : {"speaker": {"value": "HENRY", "matcher": "like"}, "text_entry": {"value": "thou", "matcher": "exact"}}
        dynamic Lucene query produced: play_name:Henry\ IV AND (speaker:*HENRY* AND text_entry:thou)
         
      • mapping qualification: {"type": "Elasticsearch DSL", "query": "${parameters('elasticsearch dsl json')}", "jsonRootPath": "$['aggregations']['lines per character']['buckets']"}
        ${parameters('elasticsearch dsl json')} : {"size":0,"query":{"bool":{"must":[{"term":{"play_name":"Henry IV"}}]}},"aggs":{"lines per character":{"field":"speaker"}}}
        Elasticsearch query produced: {"size":0,"query":{"bool":{"must":[{"term":{"play_name":"Henry IV"}}]}},"aggs":{"lines per character":{"terms": {"field":"speaker"}}}}
         
    • The expected JSON Object expected structure looks like the following:

      {
        "type": "...",
        "concateOperator": "...",
        "jsonRootPath": "...",
        "queryPrefix": "...",
        "whitelistFields": ["..."],
        "query": "{...}"
      }

    • type (required)

      • Either Elasticsearch DSL or Kinetic DSL. Changes the expected syntax of the query key value

    • concateOperator (optional)

      • Only used for the Kinetic DSL type

      • Controls the logical operator used between defined field comparisons.

      • Valid values are AND or OR. AND is the default value if the concateOperator is not defined.

    • jsonRootPath (optional)

      • Only potentially necessary for the Elasticsearch DSL query type option. See the last mapping qualification example above for an example.

      • Specifies the starting JsonPath position for the Elasticsearch results returned.

      • Defaults to $.hits.hits

    • queryPrefix (optional)

      • Only used for the Kinetic DSL type

      • Any lucence query you want to prefix to the dynamically built query. The AND operator will automatically be added to the end of the queryPrefix value and before the dynamically generated query from the 'query' key.

      • Useful as a security / scoping measure to 'jail' any query  an end-user may define on their own if the query is left wide open by setting it to ${parameters('json parameter')}

      • Parameters cannot be used here.

    • whitelistFields (optional)

      • Only used for the KInetic DSL type
      • An array of elasticsearch document fields that are allowed to query against.
      • If not specified, all fields are valid to query against in the query key
      • Useful as a security / scoping feature to 'jail' any query a user may define.
      • Parameters cannot be used here.
    • query (required)

      • Expected value is a JSON object. Note that while the expected value is in the form of a JSON object, the JSON object must be inside a string.

      • Object keys and values are dependent on the key type (Kinetic DSL or Elasticsearch DSL) value.

      • Elasticsearch DSL JSON format maps directly to the Elasticsearch Request Body Searching / Query DSL

      • Kinetic DSL JSON follows this format:
        {
          "field_one": {
            "isPhrase": true|false,
            "matcher": "...",
            "value": "..."
          },
          "field_two": {
            ...
          }

        • Kinetic DSL is parsed and then converted to the Lucene query syntax in a structured way.

        • isPhraseOptional. Valid options are either true or false. Controls whether quotes are placed around the field value match or not. Default value is false, no quotes.

        • matcherOptional. Controls how astrisk wild cards are placed around the field value. Valid options are startsWith, endsWith, exact, and like. startsWith places an asterisk after the field value. endsWith places an asterisk before the field value. exact does not append any asterisks. like spaces asterisks before and after the field value in the query. Default value is exact.

        • valueRequiredThe value that the field should be matched against. 

Changelog

v1.0.0 (2017-08-15)

  • Initial version