{"id":1474,"date":"2014-04-04T13:00:28","date_gmt":"2014-04-04T12:00:28","guid":{"rendered":"http:\/\/www.unicoda.com\/?p=1474"},"modified":"2014-03-24T15:10:38","modified_gmt":"2014-03-24T14:10:38","slug":"parsing-mongostat-data-with-logstash","status":"publish","type":"post","link":"https:\/\/www.unicoda.com\/?p=1474","title":{"rendered":"Parsing mongostat data with Logstash"},"content":{"rendered":"

On my way to a complete MongoDB monitoring solution, I’ve been playing with mongostat<\/a> to see what I can achieve with it. So I tested mongostat on a simple architecture made of two shards, each shard being a replica set composed of tree members. Of course, we also have tree configuration routers and one query router.<\/p>\n

First, I discovered a few bugs in the tool when using it with the option discover<\/em>. This parameter can be used to automatically retrieve statistics from all members of a replica set or a sharded cluster. Using it with version 2.4.9 of mongostat causes some other parameters to be ignored: rowcount<\/em> and noheaders<\/em>. So I dove in the code on Github to find that this bugs had been already corrected. We just need modifications to come to the stable version.<\/p>\n

mongostat --host localhost:24000 --discover --noheaders -n 2 30 > mongostat.log<\/pre>\n
Here I’m connecting to my query router, asking mongostat to find other mongoDB instances by itself. Options noheaders<\/em> and n<\/em> don’t work at the moment but that’s not a problem. With this setup, I will receive logs every 30s.<\/p>\n
There are two types of log: the ones coming from the mongos and the ones coming from the other.<\/p>\n
localhost:21000\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 1|0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0 800m\u00a0 1.04g\u00a0\u00a0\u00a0 30m\u00a0\u00a0\u00a0\u00a0\u00a0 0 local:0.0%\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0 198b\u00a0\u00a0 924b\u00a0\u00a0\u00a0 13 rs0\u00a0 PRI\u00a0\u00a0 15:36:55\r\n localhost:21001\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 1|0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0 800m\u00a0 1.01g\u00a0\u00a0\u00a0 29m\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0 test:0.0%\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0 138b\u00a0\u00a0 359b\u00a0\u00a0\u00a0\u00a0 6 rs0\u00a0 SEC\u00a0\u00a0 15:36:55\r\n localhost:21002\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 1|0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0 800m\u00a0 1.01g\u00a0\u00a0\u00a0 29m\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0 test:0.0%\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0 138b\u00a0\u00a0 359b\u00a0\u00a0\u00a0\u00a0 6 rs0\u00a0 SEC\u00a0\u00a0 15:36:55\r\n localhost:21100\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 1|0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0 800m\u00a0 1.05g\u00a0\u00a0\u00a0 35m\u00a0\u00a0\u00a0\u00a0\u00a0 0 local:0.0%\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0 198b\u00a0\u00a0 924b\u00a0\u00a0\u00a0 13 rs1\u00a0 PRI\u00a0\u00a0 15:36:55\r\n localhost:21101\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 1|0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0 800m\u00a0 1.01g\u00a0\u00a0\u00a0 34m\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0 test:0.0%\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0 138b\u00a0\u00a0 359b\u00a0\u00a0\u00a0\u00a0 6 rs1\u00a0 SEC\u00a0\u00a0 15:36:55\r\n localhost:21102\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0 *0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0 1|0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0 800m\u00a0 1.01g\u00a0\u00a0\u00a0 34m\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0 test:0.0%\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0\u00a0\u00a0 0|0\u00a0\u00a0 138b\u00a0\u00a0 359b\u00a0\u00a0\u00a0\u00a0 6 rs1\u00a0 SEC\u00a0\u00a0 15:36:55\r\n localhost:24000\u00a0\u00a0 \u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 174m\u00a0\u00a0\u00a0\u00a0 5m\u00a0\u00a0\u00a0\u00a0\u00a0 0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0\u00a0 2b\u00a0\u00a0\u00a0 23b\u00a0\u00a0\u00a0\u00a0 2\u00a0\u00a0\u00a0\u00a0\u00a0 RTR\u00a0\u00a0 15:36:55<\/pre>\n
Output from mongostat.<\/p>\n
 <\/p>\n
The last line coming from the mongos has empty fields. So we will need to deal with it when parsing the log. After having understood how mongostat works, it is now time to see if we can easily plug it in logstash. Let’s take a look at our logstash configuration file.<\/p>\n
input {\r\n  file {\r\n    type => \"mongostat\"\r\n    path => [\"\/path\/to\/mongostat.log\"]\r\n  }\r\n}<\/pre>\n
We define where to find the log file.<\/p>\n
 <\/p>\n
filter {\r\n  if [type] == \"mongostat\" {\r\n    grok {\r\n      patterns_dir => \".\/patterns\"\r\n      match => [\"message\", \"%{HOSTNAME:host}:%{INT:port}%{SPACE}%{METRIC:insert}%{SPACE}%{METRIC:query}%{SPACE}%{METRIC:update}%{SPACE}%{METRIC:delete}%{SPACE}%{METRIC:getmore}%{SPACE}%{COMMAND:command}%{MONGOTYPE1}%{SIZE:vsize}%{SPACE}%{SIZE:res}%{SPACE}%{NUMBER:fault}%{MONGOTYPE2}%{SIZE:netIn}%{SPACE}%{SIZE:netOut}%{SPACE}%{NUMBER:connections}%{SPACE}%{USERNAME:replicaset}%{SPACE}%{WORD:replicaMember}%{SPACE}%{TIME:time}\"]\r\n    }\r\n  }\r\n  if [tags] == \"_grokparsefailure\" {\r\n    drop { }\r\n  }\r\n  if [message] == \"\" {\r\n    drop { }\r\n  }\r\n}<\/pre>\n
We apply filter on each message received if it comes from mongostat. If the message is empty or grok fails to parse it, we drop the log.<\/p>\n
 <\/p>\n
output {\r\n  stdout { }\r\n  elasticsearch_http {\r\n    host => \"127.0.0.1\"\r\n  }\r\n}<\/pre>\n
Simple output. Logs are stored in Elasticsearch so that we can use Kibana to examine them later and are written to stdout for immediate debugging and verification.<\/p>\n
Let’s consider a little bit the filter part. We give grok a directory for our personal patterns: .\/patterns<\/em>. This directory contains a file mongostat with the following patterns:<\/p>\n
METRIC (\\*%{NUMBER})|(%{NUMBER})\r\nCOMMAND (%{NUMBER}\\|%{NUMBER})|(%{NUMBER})\r\nSIZE (%{NUMBER}[a-z])|(%{NUMBER})\r\nLOCKEDDB (%{WORD}\\:%{NUMBER}%)\r\nMONGOTYPE2 (%{SPACE}%{LOCKEDDB:lockedDb}%{SPACE}%{NUMBER:indexMissedPercent}%{SPACE}%{COMMAND:QrQw}%{SPACE}%{COMMAND:ArAw}%{SPACE})|%{SPACE}\r\nMONGOTYPE1 (%{SPACE}%{NUMBER:flushes}%{SPACE}%{SIZE:mapped}%{SPACE})|%{SPACE}<\/pre>\n
The MONGOTYPE<\/em> patterns are used to deal with empty fields from mongos log line.
\nThe rest of the match directive is just about capturing each field from mongostat to produce a more readable and analysable output.<\/p>\n
To create this, I used an online Grok debugger<\/a> which is very useful because you needn’t to reload logstash every time you want to test your work. It also provides you instant feedback.<\/p>\n
I’m know waiting for the bugs to be fixed in stable so that this solution could be more useful to monitor mongo and maybe use it in production.<\/p>\n
List of available patterns<\/a> on Github.<\/p>\n","protected":false},"excerpt":{"rendered":"
On my way to a complete MongoDB monitoring solution, I’ve been playing with mongostat to see what I can achieve with it. So I tested mongostat on a simple architecture made of two shards, each shard being a replica set composed of tree members. Of course, we also have tree configuration routers and one query … Continuer la lecture de « Parsing mongostat data with Logstash »<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[126],"tags":[167,163,162,156,166],"class_list":["post-1474","post","type-post","status-publish","format-standard","hentry","category-logiciellibre","tag-grok","tag-logs","tag-logstash","tag-mongodb","tag-mongostat"],"_links":{"self":[{"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/posts\/1474","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=1474"}],"version-history":[{"count":8,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/posts\/1474\/revisions"}],"predecessor-version":[{"id":1500,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=\/wp\/v2\/posts\/1474\/revisions\/1500"}],"wp:attachment":[{"href":"https:\/\/www.unicoda.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=1474"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=1474"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.unicoda.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=1474"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}