Log Stash Support

Testpool provides a structured log of pool status that includes the number of available VMs for each pool. This information can be pushed to logstash and visualized with Kibana or Graphana.

The following instructions explain how to enable structured logging and push them to Logstash using Filebeat.

ELK Installation

ELK stack 5.5 is required which natively supports JSON FileBeat output. There are numerous sites to explain ELK installation e.g. http://www.itzgeek.com/how-tos/linux/ubuntu-how-tos/setup-elk-stack-ubuntu-16-04.html was used to test the following content.

Testpool Configuration

Configure testpool to save pool status. Edit the YAML file:

/etc/testpool/testpool.yml

Validate changes:

tplcfgcheck /etc/testpool/testpool.yml

Uncomment tpldaemon.pool.log. The default value is /var/log/testpool/pool.log and restart testpool daemon:

sudo systemctl restart tpl-daemon

Configure Logstash to receive JSON structured content. An example configuration file at /etc/testpool/etc/logstash/conf.d/02-testpool-beat-input.conf.:

sudo cp /etc/testpool/etc/logstash/conf.d/02-testpool-beat-input.conf /etc/logstash/conf.d/02-testpool-beat-input.conf
sudo systemctl restart logstash

Make sure elastic search and logstash start on boot:

sudo systemctl enable logstash sudo systemctl enable elasticsearch

The Logstash 02-testpool-beat-input.conf content.

##
# testpool stores profile status in a structured log located at
# /var/log/testpool/profile.log. As of elastic stack 5.0,
# structured log is supported natively.
input {
  beats {
    port => 5045
    codec => "json"
#    ssl => true
#    ssl_certificate => "/etc/ssl/logstash-forwarder.crt"
#    ssl_key => "/etc/ssl/logstash-forwarder.key"
  }
}

output {
 elasticsearch {
  hosts => "127.0.0.1"
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
  }
  stdout {
    codec => rubydebug
  }
}

Configure Filebeat to push JSON content. An example is available at /etc/testpool/filebeat/filebeat.yml and can be copied verbatim.:

sudo cp /etc/testpool/filebeat/filebeat.yml /etc/filebeat/filebeat.yml
sudo systemctl restart filebeat

The content below shows a sample Filebeat configuration.

############## Testpool Filebeat Configuration Example ##############
#
# This file is an example configuration file highlighting only the
# most common options. The filebeat.full.yml file from the same
# directory contains all the supported options with more comments.
# You can use it as a reference.
#
# You can find the full configuration reference here:
# https://www.elastic.co/guide/en/beats/filebeat/index.html
#
#============= Testpool Filebeat prospectors ========================

filebeat.prospectors:

- input_type: log

  ##
  # By default testpool stores profiel status at the path below.
  # Change this if you change the location in
  # /etc/testpool/testpool.yml
  paths:
    - /var/log/testpool/profile.log
  json:
    message_key: event
    keys_under_root: true
  ##
  # Change server to anything you want
  fields:
    server: localhost
  tags: [
    "testpool"
  ]

#========================== General =================================
#
# The name of the shipper that publishes the network data. It can be
# used to group all the transactions sent by a single shipper in the
# web interface.
#name: testpool

# The tags of the shipper are included in their own field with each
# transaction published.
#tags: ["service-X", "web-tier"]

#========================== Outputs =================================
#
# Configure what outputs to use when sending the data collected by
# the beat.  Multiple outputs may be used. The configuration below
#  assumes logstash

#---------------------- Elasticsearch output ------------------------
#output.elasticsearch:
  # Array of hosts to connect to.
  #hosts: ["localhost:9200"]
  #template.name: filebeat
  #template.path: filebeat.template.json

  # Optional protocol and basic auth credentials.
  #protocol: "https"
  #username: "elastic"
  #password: "changeme"

#----------------------- Logstash output ----------------------------
##
# uncomment the appropriate authentication
output.logstash:
  # The Logstash hosts
  hosts: ["127.0.0.1:5045"]
  template.name: filebeat
  template.path: filebeat.template.json

  # Optional SSL. By default is off.
  # List of root certificates for HTTPS server verifications
  #ssl.certificate_authorities: ["/etc/pki/root/ca.pem"]

  # Certificate for SSL client authentication
  #ssl.certificate: "/etc/pki/client/cert.pem"

  # Client Certificate Key
  #ssl.key: "/etc/pki/client/cert.key"
  #ssl.key: "/etc/ssl/logstash-forwarder.key"
  #tls:
  #  certificate_authorities ["/etc/ssl/logstash-forwarder.crt"]


#============================ Logging ===============================

# Sets log level. The default log level is info.
# Available log levels are: critical, error, warning, info, debug
#logging.level: debug

# At debug level, you can selectively enable logging only for some
# components. To enable all selectors use ["*"]. Examples of other
#  selectors are "beat", "publish", "service".
#logging.selectors: ["*"]

Kibana 5.3

A sample Kibana dashboard with supporting visualization can be imported into Kibana. Content is available at: /etc/testpool/kibana/testpool.json. /etc/testpool/kibana/testpool-dashboard.json.