Skip to content

标签:ElasticSearch

ElasticSearch 7.x 解决 TooManyBucketsException 问题

ElasticSearch 7.x 版本出现如下提示: Caused by: org.elasticsearch.search.aggregations.MultiBucketConsumerService$TooManyBucketsException: Trying to create too many buckets. Must be less than or equal to: [10000] but was [10314]. This limit can be set by changing the [search.max_buckets] cluster level setting.

分析: 这是6.x以后版本的特性, 目的是限制大批量聚合操作, 规避性能风险.

解决:

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{"persistent": { "search.max_buckets": 50000 }}'

参考文档:
Increasing max_buckets for specific Visualizations
ElasticSearch search_phase_execution_exception

 

2020/05/28 Published in Linux运维 | Leave a Comment

ElasticSearch 7.x 版本提示 too_many_buckets_exception

Grafana中使用Elasticsearch的数据源时, 提示type:”too_many_buckets_exception”
reason:”Trying to create too many buckets. Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max_buckets] cluster level setting.”
max_buckets:10000

解决办法:

To minimize these either add change min time interval on datasource or panel level or either add min doc count on date histogram to 1.

参考文档:
ElasticSearch 7.x too_many_buckets_exception #17327

2020/05/27 Published in Linux运维 | Leave a Comment

ElasticSearch 解决 UNASSIGNED SHARDS

ElasticSearch出现UNASSIGNED SHARDS的解决办法

首先可以查看集群里有多少个未分配的分片, 以及分片是否均匀.

$ curl -XGET "172.18.192.100:9200/_cat/allocation?v"
shards disk.indices disk.used disk.avail disk.total disk.percent host           ip             node
  2120        2.8tb     6.2tb     11.6tb     17.9tb           35 172.18.192.101 172.18.192.101 it-elk-node3
  3520        5.8tb     5.9tb       12tb     17.9tb           33 172.18.192.102 172.18.192.102 it-elk-node4
   764          1tb       2tb      9.3tb     11.3tb           17 172.18.192.100 172.18.192.100 it-elk-node2
  1707                                                                                         UNASSIGNED

一般来说, ES会自动将未分配的shards, 分配到各node上. 使用以下命令确定自动分配分片的功能是打开的

$ curl -XGET http://172.18.192.100:9200/_cluster/settings?pretty
{
  "persistent" : {
    "cluster" : {
      "max_shards_per_node" : "20000"    # 一个node可以拥有最大20000个shards
    },
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all"    # 只要cluster.routing.allocation.enable是all的状态, ES就会自动分配shards
        }
      }
    }
  }
}

如果自动分配分片功能没有打开, 使用如下命令打开之

2020/05/19 Published in Linux运维 | Leave a Comment

ElasticSearch提示too many open files

ElasticSearch提示too many open files, 如何去分析定位?

$ curl -XGET "172.18.192.100:9200/_nodes/stats/process?pretty"
{
  "_nodes" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "cluster_name" : "it-elk",
  "nodes" : {
    "rBm53XWOTk-2v3MHPa2FDA" : {
      "timestamp" : 1589854287039,
      "name" : "it-elk-node3",
      "transport_address" : "172.18.192.101:9300",
      "host" : "172.18.192.101",
      "ip" : "172.18.192.101:9300",
      "roles" : [
        "ingest",
        "master",
        "data"
      ],
      "attributes" : {
        "ml.machine_memory" : "134778376192",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true"
      },
      "process" : {
        "timestamp" : 1589854286789,
        "open_file_descriptors" : 59595,    # 当前打开的文件
        "max_file_descriptors" : 65535,     # 系统允许打开的最大文件
        "cpu" : {
          "percent" : 3,
          "total_in_millis" : 86105320
        },
        "mem" : {
          "total_virtual_in_bytes" : 1669361537024
        }
      }
    }

当然, 也可以从系统层面, 看一下当前限制

$ ps -ef | grep elasticsearch    # 找到进程的PID
elastic+ 128967      1 99 5月18 ?       1-13:22:07 /usr/share/elasticsearch/jdk/bin/java -Xms32g -Xmx32g -XX:+UseConcMarkSweepGC

$ cat /proc/128967/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             4096                 4096                 processes
Max open files            65535                65535                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       514069               514069               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

参考文档:
https://www.elastic.co/guide/en/elasticsearch/guide/master/_file_descriptors_and_mmap.html
ElasticSearch: Unassigned Shards, how to fix?

2020/05/19 Published in Linux运维 | Leave a Comment

kibana使用的lucene查询语法

kibana使用的是lucene查询语法, 使用该语法不仅可以在kibana上使用, 也可以在Grafana中使用.

下面简单介绍一下使用方法.

全文搜索

在搜索栏输入login, 会返回所有字段值中包含login的文档
使用双引号包起来作为一个短语搜索

"like Gecko"

字段(Field)

也可以按页面左侧显示的字段搜索

field:value      # 限定字段全文搜索
filed:"value"    # 精确搜索, 关键字加上双引号
http_code:404    # 搜索http状态码为404的文档

字段本身是否存在

_exists_:http_host    # 返回结果中需要有http_host字段
_missing_:http_host   # 不能含有http_host字段
2020/01/19 Published in Linux运维 | Leave a Comment

Elasticsearch常用命令

查看所有index
curl -X GET 'localhost:9200/_cat/indices?v'

查看某些名字的index
curl -X GET "localhost:9200/_cat/indices/INDEX_PATTERN-*?v&s=index"

查看按容量大小排序的index
curl -X GET "localhost:9200/_cat/indices?v&s=docs.count:desc"

删除某些index
curl -XDELETE "localhost:9200/INDEX_NAME"

查看某个INDEX的文档数量
curl -X GET "localhost:9200/INDEX_NAME/_count"

查看节点状态(参考这篇文章.)

curl -X GET "localhost:9200/_cat/nodes?v"    # 查看各节点内存使用状态及负载情况
ip             heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.18.192.101           37          80   4    0.41    0.49     0.52 dim       -      it-elk-node3
172.18.192.102           69          90   4    1.19    1.53     1.56 dim       *      it-elk-node4
172.18.192.100           36         100   1    2.91    2.60     2.86 dim       -      it-elk-node2

curl -X GET "localhost:9200/_nodes/stats"
curl -X GET "localhost:9200/_nodes/nodeId1,nodeId2/stats"

# return just indices
curl -X GET "localhost:9200/_nodes/stats/indices"
# return just os and process
curl -X GET "localhost:9200/_nodes/stats/os,process"
# return just process for node with IP address 10.0.0.1
curl -X GET "localhost:9200/_nodes/10.0.0.1/stats/process"

# return just process
curl -X GET "localhost:9200/_nodes/process"
# same as above
curl -X GET "localhost:9200/_nodes/_all/process"
# return just jvm and process of only nodeId1 and nodeId2
curl -X GET "localhost:9200/_nodes/nodeId1,nodeId2/jvm,process"
# same as above
curl -X GET "localhost:9200/_nodes/nodeId1,nodeId2/info/jvm,process"
# return all the information of only nodeId1 and nodeId2
curl -X GET "localhost:9200/_nodes/nodeId1,nodeId2/_all"


# Fielddata summarised by node
curl -X GET "localhost:9200/_nodes/stats/indices/fielddata?fields=field1,field2"
# Fielddata summarised by node and index
curl -X GET "localhost:9200/_nodes/stats/indices/fielddata?level=indices&fields=field1,field2"
# Fielddata summarised by node, index, and shard
curl -X GET "localhost:9200/_nodes/stats/indices/fielddata?level=shards&fields=field1,field2"
# You can use wildcards for field names
curl -X GET "localhost:9200/_nodes/stats/indices/fielddata?fields=field*"

License相关

# 查看License
curl -XGET 'http://127.0.0.1:9200/_license'

# 删除License
curl -X DELETE "localhost:9200/_license"

# 导入License(本地的License文件为aaa.json), 如果启用了用户名/密码, 这里需要加上用户, 例如-u elastic
curl -XPUT 'http://127.0.0.1:9200/_xpack/license' -H "Content-Type: application/json" -d @aaa.json

shards管理

curl -XGET "127.0.0.1:9200/_cluster/settings?pretty"    # 查看集群最大分片数量
{
  "persistent" : {
    "cluster" : {
      "max_shards_per_node" : "30000"    # 单个节点能容纳3000个shards
    }
    "xpack" : 
      "monitoring" : 
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  }
}


curl -u elastic:HMEaQXtLiJaD4zn1ZxzM -XGET "127.0.0.1:9200/_cluster/health?pretty"    # 查看当前shards使用数量
{
  "cluster_name" : "it-elk",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 3,
  "number_of_data_nodes" : 3,
  "active_primary_shards" : 7233,
  "active_shards" : 7248,
  "relocating_shards" : 0,
  "initializing_shards" : 12,
  "unassigned_shards" : 5323,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 2085,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 3167844,
  "active_shards_percent_as_number" : 57.601525868234916        #数据的正常率,100表示一切ok
}

curl -XGET "127.0.0.1:9200/_cat/shards?h=index,shard,prirep,state,unassigned.*&pretty"|grep UNASSIGNED | wc -l  # 查看未分配的分片

参考文档:
https://www.datadoghq.com/blog/elasticsearch-unassigned-shards/#reason-2-too-many-shards-not-enough-nodes

2019/04/25 Published in 电脑网络 | Leave a Comment