Develop Kibana Plugins

As of Elasticsearch 5, site plugins are no longer supported, https://github.com/elastic/elasticsearch/pull/16038, only JVM plugins are supported. Site plugins are supposed to be reworked to be installed on Kibana. 

(Well, that goes my beloved kOpf plugin https://github.com/lmenezes/elasticsearch-kopf, before it is reworked into a Kibana plugin, I will have to deal with exploring Elasticsearch in other ways. )

For my log analysis project, I need to develop a Kibana plugin, which will create or update Kibana  Discovery, Visualize and Dashboard automatically based on user inputs:

My log index is named logstash-(customer)-(node)-day, my plugin will search indices matching the given pattern, and create Discovery, Visualize and Dashboard automatically. The goal is to have a separate dashboard for each customer, with one panel for one node’s logs.

This customized Kibana is used in this system An ELK Docker cloud.

Generate boiler plate Kibana plugin code

https://github.com/elastic/generator-kibana-plugin is a tool that creates boiler plate plugin code, which will serve as a good starting point for writing Kibana Plugins. Use the following steps to generate a plugin:

npm install -g yo

npm install -g generator-kibana-plugin

//place the plugin into kibana\installedPlugins folder

cd c:\devops\kibana-5.0.0-alpha2\installedPlugins

mkdir autosetup

cd autosetup

yo kibana-plugin

Now start Kibana, you will see the plugin you created in the above step appears in the left panel. 

Kibana plugin structure

kibana plugin has UI part and Server part:

$ ls  devops/kibana-5.0.0-alpha2/installedPlugins/autosetup

README.md  index.js  package.json  public  server

public folder contains UI code, server folder contains server code. The boiler code contains server side route and UI side route and UI AngularJS skeleton, it is not difficult to work on top of boiler code to implement your own login. 

Use Kibana goodies

The boiler plate code contains just the very basic function, a real plugin needs to do more. The function of my plugin is to create Discovery, Visualize and Dashboard based on the pattern of indices. To do this, I need to access Elasticsearch to do search and update, and on the UI side, I need to save user’s input. Kibana has all the goodies, though you need to do some debugging to know how to tap into them. 

Elasticsearch API

With my plugin, I need to access Elasticsearch on the server side, you can get elasticsearch client from server:

const client = server.plugins.elasticsearch.client;

I didn’t find any good online documentation of elsasticsearch client, I just refer to   node_modules\elasticsearch\src\lib\apis\master.js to understand its API usage. Here are a few examples:

search API

esclient.search({
  index: '.kibana',
  type: 'search',
  q: '_id:'+customer+"*",
  filter_path: "hits.hits._id"
  })

This searches the search type of .kibana index for any doc whose _id starts with the given customer string, returning only the _id field.  

create or update doc API

esclient.index({

  index: '.kibana',

  type: type,

  body: docString,

  id: id,

})

This will create or update the doc if it doesn’t exist, there is another API esclient.create, which uses "op=create" parameter, and will throw out an error if the doc already exists.  

By the way, Kibana itself accesses Elasticsearch from the UI side: the UI side uses elasticsearch API to construct http requests, on the server side, plugins\elasticsearch\lib\map_uri.js will translate the http url to access Elasticsearch, for example, url from UI /elasticsearch/_msearch gets mapped to http://16.165.216.61:9200/es/_msearch (http://16.165.216.61:9200/es is the Elasticsearch url).

Session storage

On the UI side, I need to store something on the session, so it can be used as a global variable. To access it, you need to:

import 'ui/storage';

Now you can inject sessionStorage into AngularJS components (controller, service etc). 

Develop into the Kibana plugin

You know what, Kibana itself is a plugin of Kibana – sounds mouthful, right? Kibana has 4 tabs, Setting, Discovery, Visualize, and Dashboard. These 4 parts make up the Kibana plugin, which has the same structure: 

$ ls devops/kibana-5.0.0-alpha2/src/plugins/kibana/

index.js  package.json  public  server

The plugin created so far is a completely independent one, in AngularJS terms, it is a separate ng-app, just like the Kibana plugin. 

This creates a problem for me, my plugin should respond to browser refresh: whenever the browser is refreshed, my plugin should check if there is new indices and create or update Discovery, Visualize and Dashboard directly, users don’t have to click on my plugin to invoke its function. Since my plugin is an independent ng-app equaling to the Kibana plugin, refreshing Kibana components (Discovery, Visualize and Dashboard) will not be able to invoked my plugin’s function. In order to achieve this function, my plugin needs to be part of the Kibana plugin. 

To do so, move my plugin’s UI code to Kibana’s public folder and the server code to Kibana’s server folder:

 

Now to hook things together, we need to change kibana-5.0.0-alpha2\src\plugins\kibana\index.js: 

import autosetup from './server/routes/api/autosetup';

…

//add tab

links:[

…

{
  title: 'Autosetup',
  order: -1004,
  url: '/app/kibana#/autosetup',
  description: 'define index patterns, change config, and more',        
}

//init server

init: function (server, options) {
  ingest(server);
  search(server);
 
  autosetup(server);
}

UI side routing is easy, just need to have this in kibana-5.0.0-alpha2\src\plugins\kibana\public\autosetup\index.js:

import uiRoutes from 'ui/routes';
import template from 'plugins/kibana/autosetup/public/templates/index.html';

uiRoutes
  .when('/autosetup', {
    template,
  });

Now add this index.js to kibana-5.0.0-alpha2\src\plugins\kibana\public\kibana.js:

import 'plugins/kibana/autosetup/index';

With these changes, my plugin ceased to be independent, and becomes part of the Kibana plugin. The rest is simple.