Kusabana

---

About

Kusabana is a proxy server between Kibana and Elasticsearch.
It was developed to cache result each query.

Design

Kusabana is coded by Ruby 2.0, using em-proxy gem, and depending on memcached.

Why

Elasticsearch + Kibana are becoming typical solution for data mining.
However, it is said that they have some performance problems. Why?

I think, there are some problems about behavior of Kibana. For example, Kibana make request to ES on almost clicking.
Futhermore, Elasticsearch doesn’t have a mechanism of ‘query cache’ but for ‘filter cache’.

These problems are seemed to be resolve by proxy that have simple caching system. However, The query produced by kibana is variable by time, path or dashboard’s environment.
This make us have to create more functional caching.

Kusabana is a solution implemented for this problem. Needless to say caching, Kusabana can store log of itself to Elasticsearch.
This will make you able to make configration easier.

Installation

git clone https://github.com/voyagegroup/kusabana
cd kusabana
bundle install

Because of memcached gem, it require libsasl2-dev or any other similer package.

Config

proxy:
  host: '0.0.0.0'
  port: 9292
  daemonize: false
  timeout: 15
# output: 'log/kusabana.log' 
# pid: 'log/kusabana.pid'
es:
  remote:
    host: 'localhost'
    port: 9200
# output:
#   index: 'kusabana-log-1'
#   hosts:
#   - host: 'localhost'
#     port: 9200
cache:
  url: 'localhost:11211'

• proxy
  • host, port: Host and port used by Kusabana
  • daemonize: When true, Kusabana will work as daemon
  • timeout: The seconds for harakiri each connection
  • output: The file for output log. If you make it comment out, the log will output to STDOUT
  • pid: Used by daemon mode (default value is ./kusabana.pid)
• es
  • remote: The Elasticsearch used for proxying access
  • output: The Elasticsearch used to store and to aggregate Kusabana’s log
• cache
  • url: Memcached url

If you want to store Kusabana’s log and set output Elasticsearch, You should PUT index template to ES.
Run

bundle rake template:create

Additionally, You can use Kibana Dashboard for montoring Kusabana’s log.

bundle rake dashboard:create

Then, the Dashboard is going to be seen at /dashboard/elasticsearch/kusabana.

Usage

The configration of cache is available in ./bin/kusabana.

# Default settings
rules = []
search_caching = Kusabana::Rule.new('POST', /^\/.+\/_search(\?.+)?$/, 300)
timestamp_modifier = Kusabana::QueryModifier.new(/@timestamp/) do |query|
  if query.key?('from') && query.key?('to')
    query['from'] = query['from'] / 100000 * 100000
    query['to'] = query['to'] / 100000 * 100000
  end
  query
end
search_caching.add_modifier(timestamp_modifier)
rules << search_caching

rules << Kusabana::Rule.new('GET', /^\/_nodes$/, 300)
rules << Kusabana::Rule.new('GET', /^\/\S+\/_mapping$/, 300)
rules << Kusabana::Rule.new('GET', /^\/\S+\/_aliases(\?.+)?$/, 300)

Kusabana::Rule

Kusabana::Rule.new(method, path_pattern, expire)

Kusabana::QueryModifier

Kusabana::QueryModifier.new(key_pattern, &block)

When Kusabana serve request, it will be checked whether there are any Rule that is matched by method and path_pattern.
If match, Kusabana will try parsing query, scanning JSON’s key and executing $block of matched QueryModifier by key_pattern.
The query will be modified by returning of &block.

This behavior is made for ignoring tiny differents between queries(e.g. The filter by range of @timestamp).

Each Rule and QueryModifier is apply only first matched one.

Log Format

Request

• match[boolean]
• method[string]
• orig_query[string]
  • The query before modified
• mod_query[string]
  • The query afetr modified
• mod_query[string]
• path[string]
• session[string]
  • The UUID created for each Request-Response

Response

• cache[string]
  • no, store, use or error
• expire[integer]
  • The time cache living
• key[string]
  • The memcached key used for cache
  • A format is METHOD::PATH::HASH
• method[string]
• path[stirng]
• session[string]
• status[integer]
• took[float]
  • Seconds spent betwenn Request and Response

Status

• key[string]
• count[intger]
• from[date]
• to[date]
• efficiency[float]
  • [took when store]*count/expire SO SUITABLE

• expire[integer]

• avg[float]
• max[float]
• min[float]
• sum[float]
  • Stats of took time to get cache from Memcached

Benchmark

Not yet

Contribute

Today, Kusabana is experimental product yet.
Don’t mind to send patch!

Todo

1. Fork repo
2. Make it change
3. Run test (bundle exec rake test)
4. Add a test for change
5. Submit pull request