Sunday, February 28, 2016

Logstash Multiline codec issue

Be careful with the multiline codec in logstash. From my tests, it finally works in logstash 2.2.2.
The problem with the multiline codec is that it will mix events from multiple files. Below is the illustration. 

There are two log files test1.log and test2.log:

The logstash configuration file test.conf is:

input {
  file { 
      path => "/devops/tmp/test*.log"    
      start_position => "beginning"
      sincedb_path => "/devops/tmp/.sincedb"   
      codec => multiline {    
      negate => true
      what => previous

output { 
   stdout {    
      codec => rubydebug     

The multiline codec configuration means: if a line doesn’t (negate=>true) start with the pattern, i.e. doesn’t start with a status level, this line belongs to the previous (what=>previous) line(s). negate controls whether to match or not match the pattern, what controls whether this current line belongs to the previous or next.

Running logstash 2.0.0 produces:

Notice, logstash mixes the line STATUS CAT1 in file test2.log.The last line in test2.log is not processed.  

Now if we add a line into test1.log, it will prompt logstash to process the last line in test2.log, but logstash will credit this line to test1.log.

Needless to say, this is very confusing. Thankfully, this issue is solved in 2.2.2. Here is the result in 2.2.2:

You will notice that the last line in each log file is missing, which is understandable, because multiline codec needs to read the next line to determine if the multiline event is completed.

Friday, February 12, 2016

One thing to rule them all – Ansible on Vagrant

Vagrant allows to play with multiple VMs easily, but with multiple VMs, it is dizzy to switch VMs back and forth, it is easy to do something on VM B when the intended VM is A.

Ansible comes to the rescue. With Ansible, you sit comfortably at one VM and control other VMs. In this blog, I will show you how to setup Ansible on Vagrant. 

The eventual setup will be like this:

My physical machine is win7, on it, I start 4 VMs. VM mgmt is my management VM, which I will install ansible and rule over other VMs. VM infra is where I install infrastructure tools, such consul and other monitoring tools, the rest VMs app1 and app2 is where I install applications. 

I do development work on win7, after testing out, I deploy them into VMs. A shared folder is setup between win7 and mgmt, so it is easy to move things around between win7 and VMs.



Vagrant.configure("2") do |config|
   if Vagrant.has_plugin?("vagrant-proxyconf")
#replace with your proxies
        config.proxy.http= " https://<user>:<password>@<proxy-host>:<proxy-port>"
        config.proxy.https= " https://<user>:<password>@<proxy-host>:<proxy-port>"
        config.proxy.ftp= " https://<user>:<password>@<proxy-host>:<proxy-port>"
         config.proxy.no_proxy  = "localhost,"  
    if Vagrant.has_plugin?("vagrant-timezone")
    end = "ubuntu14.04-amd64"
   config.vm.box_url = ""
   config.ssh.forward_agent = true
   config.vm.provider "virtualbox" do |vb|      
             vb.gui = true 
            vb.customize ["modifyvm", :id, "--natdnshostresolver1", "on"]
              vb.customize ["modifyvm", :id, "--natdnsproxy1"       , "on"]            
             vb.customize ["modifyvm", :id, "--memory", 8192]
             vb.customize ['modifyvm', :id, '--nicpromisc1', 'allow-all']

             vb.customize ['modifyvm', :id, '--nicpromisc2', 'allow-all']
             vb.customize ["modifyvm", :id, "--ioapic"  , "on"]
             vb.customize ["modifyvm", :id, "--cpus"    , 2]
             vb.customize ["modifyvm", :id, "--pae"    , "on"]
             vb.customize ["modifyvm", :id, "--nictype1", "virtio"]
             vb.customize ["modifyvm", :id, "--nictype2", "virtio"]   
             vb.customize ["modifyvm", :id, "--chipset" , "ich9"]            

  # create mgmt node
  config.vm.define :mgmt do |mgmt_config|
      mgmt_config.vm.hostname = "mgmt" :private_network, ip: "" 
      mgmt_config.vm.provision :shell, path: ""        
       mgmt_config.vm.synced_folder "../../devops", "/devops" 
   # create infra node
  config.vm.define :infra do |infra_config|
      infra_config.vm.hostname = "infra" :private_network, ip: ""         
  # create app nodes
  (1..2).each do |i|
    config.vm.define "app#{i}" do |node|       
        node.vm.hostname = "app#{i}" :private_network, ip: "{i}"     
             if Vagrant.has_plugin?("vagrant-proxyconf")               
                    node.proxy.no_proxy  = "localhost,,{i},app#{i}"


I’ve shared the Vagrantfile techniques in my previous blogs, there are only a couple of differences:
  • This single Vagrant file creates 4 VMs: mgmt, infra, app1, app2 
  • This Vagrant installs ansible on VM mgmt using

And here is the content of
#!/usr/bin/env bash

# install ansible (
apt-get update
apt-get -y install software-properties-common
apt-add-repository -y ppa:ansible/ansible
apt-get update
apt-get -y install ansible

cat >> /etc/hosts <<EOL
# vagrant environment nodes  mgmt  infra  app1  app2

Note, when you run this Vagrantfile using vagrant up, Vagrant will fail for each VM complaining shared folder can’t be setup, refer to my previous blogs for the solution.

vagrant up will create and start all 4 VMs, you can also use vagrant up hostname to start one VM, and use vagrant ssh hostname to log on one VM.



Now vagrant ssh mgmt, and verify ansible is installed correctly:

vagrant@mgmt:~$ ansible --version
  config file = /etc/ansible/ansible.cfg
  configured module search path = Default w/o overrides

Now configure Ansible to work on infra and app1, app2 nodes. To do so, we need to setup inventory.ini and ansible.cfg, here is my file structure:

inventory.ini configures the nodes that are under the control of Ansible:

ansible.cfg configures Ansible properties:
inventory = /devops/SaaS-Example/ansible/inventory.ini
callback_plugins = /devops/SaaS-Example/ansible/callback_plugins
(this callback will be explained later)

Let us say hello from mgmt to other VMs. The command to use is ansible all -m ping. Let us break this command down:
  • all: the target nodes, all means all nodes in inventory.ini, we could also use a single node or node groups, such as app1 or app.
  •  -m: module
  •  ping: the module name. Ansible has a lot of modules, using Ansible is to use these modules to accomplish certain things. If you fail to find a certain module, you can always use the old good shell module.
Our first hello attempt fails. This is because the other nodes do not know who mgmt is, we need to setup the ssh trusty relationship between mgmt and other nodes. 

Although this attempt fails, it puts infra, app1, app2 into the known_hosts of mgmt, another way to accomplish this is to ssh to these boxes directly, and enter yes upon prompting.  

To establish ssh trust, first create public and private keys using ssh-keygen -t rsa -b 2048. At prompting, simply hit enter. This will create keys in /home/vagrant/.ssh/ and /home/vagrant/.ssh/id_rsa.

For the next step we can use Ansible to setup the trusty relationship. Create file ssh-addkey.yml:
- hosts: all
  become: yes
  become_method: sudo 
  gather_facts: no

  - name: install ssh key
    authorized_key: user=vagrant
                    key="{{ lookup('file', '/home/vagrant/.ssh/') }}"

This yml file is an ansible playbook, despite its format, it follows similar structure as the above ansible command line:
  •  hosts: all: target at all nodes
  •  authorized_key: a module that sets up keys for user (here vagrant) for remote nodes, user, key, state are parameters for this module
Now run this playbook with command ansible-playbook ssh-addkey.yml --ask-pass:

With --ask-pass, ansible will prompt you for password, enter vagrant.

Now the trust relationship is setup. We can say hello again:
This time, it succeeds.

By the way, ansible will cache ssh connections for 10 seconds (configurable in ansible.cfg), you can check out the connections:
So if you say hello again very soon, ansible will reuse the existing ssh connections.

Yml format is very strict, in the beginning this always gets on me (still does):

Indention must be lined up, and must be whitespaces. If you use notepad++, you can convert tab to whitespaces in settings/preferences: