Debug index template/mapping creation in Logstash and ElasticSearch

Logstash is not the best system at letting one know what is wrong or why it is giving you errors. In my case, I have set a Elasticsearch index template to be pushed during the creation process and Logstash was giving me this error at initialization.

Logstash output config, with the template push definition.

output {
   elasticsearch {
       hosts => "10.56.32.29:9200"
       index => "myindex-%{+YYYY.MM.dd}"
       template => "/myconfig/template.json"
       template_name => "myindex"
       template_overwrite => "true"
   }
}
[36254]: [2020-04-12T20:19:31,185][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/netflow
[36254]: [2020-04-12T20:19:31,294][ERROR][logstash.outputs.elasticsearch] Failed to install template.{:message=>"Got response code '400' contacting Elasticsearch at URL 'http://10.10.10.10:9200/_template/myindex'"

It didn’t prevent Logstash or Elasticsearch from working, but the fields mappings were not correct. You see, the error, says only that there was an error.. but doesnt tell you what it is, or where to check. Whenever this happens you are forced to try to reproduce the steps manually. Using curl I tried to push the template manually at Elasticsearch and check how it would receive it. Using curl you can debug and receive whatever Elastic has to say.

curl -i -X PUT "http://10.10.10.10:9200/_template/myindex?pretty" -H 'Content-Type: application/json' --data-binary "@/myconfig/template.json"

The command above with the -i, displays in the screen whatever the output is, and the “?pretty” make it show nicely. Another trick that is very important is to append the “@” at the beginning of the file path, otherwise curl will complain. after that Elasticsearch spilled the following:

HTTP/1.1 100 Continue
HTTP/1.1 400 Bad Request
content-type: application/json; charset=UTF-8
content-length: 545

{
"error" : {
   "root_cause" : [
                     {
                      "type" : "mapper_parsing_exception",
                      "reason" : "No handler for type [bytes] declared on field [src_to_dst_second_bytes]"
                     }
    ],
"type" : "mapper_parsing_exception",
"reason" : "Failed to parse mapping [_doc]: No handler for type [bytes] declared on field [src_to_dst_second_bytes]",
"caused_by" : {
       "type" : "mapper_parsing_exception",
       "reason" : "No handler for type [bytes] declared on field [src_to_dst_second_bytes]"
  }
},
"status" : 400
}

Now I know there is a field in my template mapping that has a type mismatch. Much better, to know where to fix.

Leave a Reply

Your email address will not be published. Required fields are marked *