Kibana - Next level

Several use-cases can be envisioned, we will only relate here the data conversion steps to go through given the widespread necessity of such a basic task.

Data type conversion

Let’s consider a metadata processed by Fast2 as a String instead of a float. One frequent use-case could be reporting the sum of all content size processed during a campaign. Adding up String values never ended up well so far, Kibana will have to parse these values beforehand, to have the user access the newly created value with the correct type.

We will base our example on the following punnet structure:

<?xml version='1.0' encoding='UTF-8'?>
<ns:punnet xmlns:ns="http://www.arondor.com/xml/document" punnetId="FileNetSource#page_0#pageIndex_0">
	<ns:documentset>
		<ns:document documentId="{1B62F7C4-8E75-4D99-B84C-0AAD14B13A4E}">
			<ns:contentset>
				<ns:content mimeType="application/pdf">
					<ns:url>path/to/file/content</ns:url>
				</ns:content>
			</ns:contentset>
			<ns:dataset>
				<ns:data name="MimeType" type="String">
					<ns:value>application/pdf</ns:value>
				</ns:data>
				<ns:data name="ContentSize" type="String">
					<ns:value>43315.0</ns:value>
				</ns:data>
				<ns:data name="name" type="String">
					<ns:value>file_name</ns:value>
				</ns:data>
				...
			</ns:dataset>
			<ns:folderset />
			<ns:annotationset />
		</ns:document>
	</ns:documentset>
	<folderSet />
</ns:punnet>

Here, the data type of the ContentSize property is String, as the type attribute states. Our job will be to parse this String value to Float, since we have decimal.

This operation happens in 2 steps:

1. Making the original field (with the wrong type) accessible from a script,
2. Writing the correct parsing script.

Step 1: Making the field accessible

Open the Kibana retrieving data from the Elasticsearch database populated by the campaigns executed by Fast2.

Access the Dev Tools of Kibana to execute the next query :

PUT f2_*/_mapping
{
  "properties": {
    "punnet": {
      "properties": {
        "documents": {
          "properties": {
            "data": {
              "properties": {
                "ContentSize": {
                  "type": "text",
                  "fielddata": true
                }
              }
            }
          }
        }
      }
    }
  }
}

This query has to be built a specific way in order to change the mapping of the field to convert. The index prefix of Fast2 data is f2_, which is why the request has to specify the index to apply the PUT operation on. Next you have to write in the request body the whole way to the property to convert, first by accessing the punnet, then the properties of the punnet where the documents are stored, then the properties of the documents object where the metadata are stored, and so on.

Finally specify the current type (text in our case), with fielddata:true.

The query is successfully executed once the acknowledged:true message is returned.

Step 2: The parsing script

Head now to the Kibana Management section, choose ‘Index patterns’, and select the one related to Fast2 (f2_*).

Take some time to refresh the f2_* index, to make sure all the properties are fetched from Elasticsearch.

Click on the ‘Scripted field’ tab to create a new one using the data property you just made accessible.

Enter a relevant name (ex/ ContentSize-float), select the new type of this field (number, for you example), and write the following script:

if (doc.containsKey('punnet.documents.data.ContentSize'))
  return Float.parseFloat(doc['punnet.documents.data.ContentSize'].value);
else
 return -1;

Save this new field, and create a new visualization displaying a sum of this data per campaign (as this was our initial challenge).

And that’s it ! Data conversion from Elasticsearch for Kibana. More use cases can be addressed by scripts with higher level of complexity, to create new data, digest or sort data, you name it.

This new dimension of data analysis via Kibana opens up way more possibilities, while increasing the precision of data aggregation to bring the best answers to the project management team.

In case of even more specific use of Kibana, please refer to the official documentation at Elasticsearch.co.