Skip to main content
Kinetic Community

Processing XML using JSON and Eval

In this article I will discuss using Eval to manage a relatively common situation - processing and extracting data from XML. XML data is used extensively in the Task Engine, but frequently the format is not exactly what one wants or only certain values in the XML string are needed. Creating a specialized handler is not an ideal solution because of the overhead of testing and installing coupled with the potential differences to the XML format you want to process. This creates a perfect scenario for using Eval which creates essentially a handler "in place" and is easily customizable and changeable. .

Usage

Introduction

The Eval System Utility is provided by the Task Engine and allows one to write Ruby code that processes directly in the Task Tree without needing an external Handler.

In general, consider using Eval when you:

  • Do not need to access external systems or require an integration.
  • You have more complex coding requirements or logic that doesn't fit easily into breaking down the requirements into individual Task Tree Nodes.
  • Need to manipulate data  - for example, changing the case of a string - that will be consumed by other Task Tree Nodes.

Example Case - Software Selection

Consider this case: your Service Item allows a Requester to add one or more rows to a table to represent individual selections. These selections could be Software Packages to install, a list of Attendees or Approvers, items on a Purchase Order. Anything, really, where you populate an HTML table with individual selections.

Like this Software Selection Table, for example:

cap1.jpg

 

For every option the Requester makes, you want to process each row in your Task Tree and do something - for example, send an Approval or Notification, create a Service Request, or make a Calendar entry. Or perhaps you want to consolidate selections into a single value that can be used within a Message Template or the Notes section of a Request.

The steps to do this are fairly straightforward - we take the rows and columns of data from the Table and serialize them into a readable format. Then, add a hidden Question to your Template to hold that serialized data. Then our Task Tree can pick up the Question's value and process the data. We represent the HTML Table data using a technology called JSON.

Before we step into the Task Tree, it may be useful to know a little about how to get that data from the HTML Table.

A little bit about JSON

JSON is an acronym for Javascript Object Notation. It is a way of taking a Javascript object (or any object, really), and representing it as a string. The format is very simple -

  • [] - Square brackets enclose Arrays.
  • {} - Curly braces enclose objects.
  • :   - Colons separate name/value pairs.
  • ,  - Commas separate elements
  • " - Quotes enclose names and values.

So, a simple JSON string to represent an array of simple objects may look like this:

[{"name" : "object 1", "type" : "object"}, {"name" : "object 2", "type" : "object"}]

Kinetic Request uses a very robust and flexible set of Javascript classes based on the YUI library. For an HTML Table created using a Data Source, we can access the rows of the Table by using a Recordset (something you may be familiar with as the idea is used in several other technologies). The YUI Datatable also provides access to the Columnset which is useful for accessing the "fields" represented by your HTML Table.

Working with Data Tables can be a large subject so I don't want to delve too deeply in this article, but below is some boilerplate Javascript you can use as a starting point. This function gets a reference to the Table (myTable), accesses the Recordset for the Table and gets a list of Records from the Recordset. It then builds a JSON formatted string from the Rows contained in the Records and Columns defined in the Table. Finally, it uses the KD.utils.Action.setQuestionValue function to put that JSON string into a Question.

function saveTable() {
    var records = myTable.getRecordSet().getRecords();
    var str="[";
    for (var i=0; i < records.length; i++) {
        str= str+"{\"";
        var keys = myTable.getColumnSet().keys;
        for (var j=0; j < keys.length; j++) {
            str= str+keys[j].getKey();
            str=str+"\":\"";
            str=str+records[i].getData(keys[j].getKey());
            if(j== (keys.length)-1) {
                if(i== (records.length)-1) {
                    str= str+"\"}";
                } else {
                    str= str+"\"},";
                }
            } else {
                str= str+"\",\"";
            }
        }
    }  
    str=str+"]";
    KD.utils.Action.setQuestionValue("myTableData", str);

}

The above function will produce output with a format similar to this:

[
 {"title" : "Visio", "manufacturer" : "Microsoft", "requires_approval" : "No"},
 {"title" : "DOOM". "manufacturer" : "ID Software", "requires_approval" : "Yes"},
 {"title" : "Firefox", "manufacturer" : "Mozilla", "requires_approval" : "No"}
]

We use setQuestionValue to insert the JSON string into a Question field. If you're not familiar with the KD.utils Javascript Library, documentation is available here.

So, now we have a JSON string in a Question and we're ready to extract and process this data in our Task Tree.

The JSON to XML Handler

The next step is to use the JSON to XML utility handler (J2X) to convert the JSON string to XML. It is attached to this article for easy retrieval. The J2X handler takes the JSON string and converts it into an XML string using a fairly simple convention. Arrays are converted to XML elements of type "Array", Objects are converted to elements of type "Object" and Properties are converted to elements of type "String".

The README in the handler file has more details if you wish to investigate further.

Since we're working with tabular data, we will have an Array of Rows. Each Row is represented by an Object element and each Field in the Row is represented by a String element. The Field element's Name attribute is the name of the Field, and the Field element's value is the value of that field.

<node type='Array'>
   <node type='Object'>
     <node type='String' name='Column Name'>Value</node>
   </node>
</node>

When the J2X handler returns a value, it will be in standard handler return format.

<results>
   <result name="XML"/>
 </results>

Also note that the XML returned from the handler will be URL encoded. Meaning, the XML processing characters (<, >, ' and ") are replaced by their URL encoded character equivalents (&lt;, &gt;, &apos;, and &quot;).

So, the combined result of the J2X handler as viewed in the Management Console will be:

<results>
   <result name='XML'>&lt;node type='Array'&gt;&lt;node type='Object'&gt;
   &lt;node type='String' name='Column Name'&gt;Value&lt;/node&gt;&lt;/node&gt;&lt;/node&gt;
   </result>
</results>

The reason the XML is encoded is to prevent incorrect values being extracted from the results string. When a result it used by other handlers in the Task Tree, the "<results><result/></results>" tags are stripped, leaving just the result value. If the result value is valid XML, the Task Engine interprets that XML as part of the extraction path and not the value itself and this causes problems with returning the correct values.

Using our example data above, the result will look like this:

<node type='Array'>
  <node type='Object'>
   <node type='String' name="title">Visio</node>
   <node type='String' name="manufacturer">Microsoft</node>
   <node type='String' name="requires_approval">No</node>
  </node>
  <node type='Object'>
   <node type='String' name="title">DOOM<node>
   <node type='String' name="manufacturer">ID Software</node>
   <node type='String' name="requires_approval">Yes</node>
  </node>
  <node type='Object'>
   <node type='String' name="title">Firefox</node>
   <node type='String' name="manufacturer">Mozilla</node>
   <node type='String' name="requires_approval">No</node>
  </node>
 </node>

Getting our data.

Now it's time to extract our data with an Eval node. For Eval, we use Ruby in place to process data. The Eval node takes a single parameter, Code, which is the actual Ruby source.

There are a few caveats to using Eval which should make the example code much easier to understand.

  • The Eval handler will produce a single result: "output"
  • The value of output is the last processed value of the script.
  • Double quotes are escaped using the "\" (backslash) character.
  • To embed a Newline in a string, use the sequence "\n".

The result we want is a list of Software Titles separated with a line break which, for example, we could insert into a section of a Request or an Email Message.

First, I'll present the Code in its entirety, then explain each step. For further information on the operators and syntax being used, a good reference is ruby-lang.org.

<%=
require 'rexml/document'

str_xml = @results['JSON to XML']['XML']

doc = REXML::Document.new(KineticTask::Utils.decode(str_xml))

if doc.nil?
 raise("Document is NIL!")
end

xpath_str = "/node/node/node[@name='title']"

ret_str = ""

doc.elements.each(xpath_str) {|e|
ret_str << "\"#{e.text}\n\""
}



%>


The Code starts and ends with the ERB (Embedded RuBy) tags "<%=" and "%>. The ERB tags tell the Task Engine's Ruby interpreter to process the lines between them as code. The "=" at the beginning tells the Ruby interpreter to output the result. If you leave the "=" out, the Eval statement will process, but no data will be returned.

The require statement tells the Ruby interpreter to load the Rexml gem. Rexml is a library used to process XML documents. For more information on using Rexml, this site offers excellent information.

require 'rexml/document'

We set a variable "str_xml" to the result of the J2X Task Node. If you're familiar with the Task Engine already, you'll note that the ERB tags ("<%=...%>") are not a part of the reference to the Task Result. Since we've already included them at the beginning of the Code section, they aren't necessary again.

str_xml = @results['JSON to XML']['XML']

Now we use the Rexml library to set a variable called "doc" to a newly created XML document.

We use a special method to process the XML string before loading it into a Document using Rexml - KineticTask::Utils.decode.

Recall from above the discussion about replacing special characters. This method removes the URL encoded values for those characters and replaces them with the characters themselves.

doc = REXML::Document.new(KineticTask::Utils.decode(str_xml))

If there was a problem loading the XML string into a document object, we check to see if the doc is "nil". If it is nil, we raise an error.

if doc.nil?
 raise("Document is NIL!")
end

We create an XPath search string. XPath is a way of searching through an XML document to return elements that match the pattern. In this case, we're looking for all elements nested under the 3rd level of node elements that contain the attribute (@) that equals "title".

xpath_str = "/node/node/node[@name='title']"

The returned value would essentially be this set of elements:

<node type='String' name="title">Visio</node>
<node type='String' name="title">DOOM<node>
<node type='String' name="title">Firefox</node>

We create an empty string variable to hold our results.

ret_str = ""

Now we build the string.

doc.elements.each(xpath_str) {|e|
  ret_str << "\"#{e.text}\n\""
}

Note that there are several things going on with this statement.

We're calling the elements.each method on the XML document and passing in the XPath expression we built above. What this does is gets a reference to the elements collection of the XML document, returns a set of elements from the XPath expression, then iterates over them using the each method.

The code to execute for each Element is enclosed in curly braces. Inside the braces we create a parameter called "e" that represents the object being operated upon for that iteration. This makes the object visible to the code block and is identified by being enclosed in the pipe character.

For each of the Elements returned, we use this statement to concatenate the string, adding a new line (\n) to the string after each loop:

  ret_str << "\"#{e.text}\n\""

Note that there are escaped quotes inside the return string.

  ret_str << "\"#{e.text}\n\""

This is needed because there are two operations going on - the first resolves the text method of the e object. This "comsumes" the outer quotes. The second is the concatenation operation using the "<<" method of the built-in String class.

Without the escaped quotes, the statement would be interpreted by Ruby as if we literally wrote:

  ret_str << Visio

If we just used two sets of quotes, Ruby would see that as an empty string and would fail processing.

  ret_str << ""#{e.text}\n""

So, to make sure we have Ruby treat the return value of the e.text method as a string, we add a second, escaped, set of quotes.

Eval will return the last value processed unless explicitly directed using a return statement. Since we're working with a code block to built the string, we want to make sure we don't return just the last value of the iteration. That is, we don't want just the last value for e.text but the value of ret_str.

It is probably a better convention to use return explicitly as it makes it much easier to identify what is being output, but either approach will work

The value of ret_str wil be:

Visio
DOOM
Firefox

In order to use the results of your Eval, you can reference the "output" parameter. So, for an Eval node called "Extract Software List", a Node on your Task Tree can reference <%= @results['Extract Software List']['output']%>

Conclusion

Using the Eval handler to extract XML data is just the beginning. With this utility, you now have an option that provides the power of a fully developed handler, but with the flexibility to specialize code to your particular needs and make changes without the overhead of handler registration.

As always, let your business requirements lead the way - if you find yourself considering creating a Handler to manage a particular task, but have varying needs in terms of what you need that Handler to do, this might indicate a place where the Eval utility provides an optimum solution.

 

Example