Had a red flag on my ElasticSearch cluster these days and found that the reason was related to an unassigned shards between the nodes.
As the data I collect is not that sensitive I could easy delete it and recreate in case I need in the future. But first we need to find it. There is many articles on the internet to help one to understand the shards allocation but I offer here a simple solution which is – simply delete the bastard.
First, we check on the cluster health and get the count of unassigned shards.
curl -XGET http://<elastichost>:9200/_cluster/health?pretty | grep unassigned_shards
The list of unassigned shards can be retrieved using:
curl -XGET http://<elastichost>:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'}
And if you want to delete every unassigned shard in the list above, you could send it to xargs and do a DELETE.
curl -XGET http://<elastichost>:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE "http://<elastichost>:9200/{}"
In my case the I found the name of the index that was cause problems and I deleted
curl -XDELETE "http://<elastichost>:9200/INDEXNAME
Done Cluster health is green again.