ElasticSearch 解决 UNASSIGNED SHARDS

ElasticSearch出现UNASSIGNED SHARDS的解决办法

首先可以查看集群里有多少个未分配的分片, 以及分片是否均匀.

$ curl -XGET "172.18.192.100:9200/_cat/allocation?v"
shards disk.indices disk.used disk.avail disk.total disk.percent host           ip             node
  2120        2.8tb     6.2tb     11.6tb     17.9tb           35 172.18.192.101 172.18.192.101 it-elk-node3
  3520        5.8tb     5.9tb       12tb     17.9tb           33 172.18.192.102 172.18.192.102 it-elk-node4
   764          1tb       2tb      9.3tb     11.3tb           17 172.18.192.100 172.18.192.100 it-elk-node2
  1707                                                                                         UNASSIGNED

一般来说, ES会自动将未分配的shards, 分配到各node上. 使用以下命令确定自动分配分片的功能是打开的

$ curl -XGET http://172.18.192.100:9200/_cluster/settings?pretty
{
  "persistent" : {
    "cluster" : {
      "max_shards_per_node" : "20000"    # 一个node可以拥有最大20000个shards
    },
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : {
    "cluster" : {
      "routing" : {
        "allocation" : {
          "enable" : "all"    # 只要cluster.routing.allocation.enable是all的状态, ES就会自动分配shards
        }
      }
    }
  }
}

如果自动分配分片功能没有打开, 使用如下命令打开之 Continue reading “ElasticSearch 解决 UNASSIGNED SHARDS”

ElasticSearch提示too many open files

ElasticSearch提示too many open files, 如何去分析定位?

$ curl -XGET "172.18.192.100:9200/_nodes/stats/process?pretty"
{
  "_nodes" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "cluster_name" : "it-elk",
  "nodes" : {
    "rBm53XWOTk-2v3MHPa2FDA" : {
      "timestamp" : 1589854287039,
      "name" : "it-elk-node3",
      "transport_address" : "172.18.192.101:9300",
      "host" : "172.18.192.101",
      "ip" : "172.18.192.101:9300",
      "roles" : [
        "ingest",
        "master",
        "data"
      ],
      "attributes" : {
        "ml.machine_memory" : "134778376192",
        "ml.max_open_jobs" : "20",
        "xpack.installed" : "true"
      },
      "process" : {
        "timestamp" : 1589854286789,
        "open_file_descriptors" : 59595,    # 当前打开的文件
        "max_file_descriptors" : 65535,     # 系统允许打开的最大文件
        "cpu" : {
          "percent" : 3,
          "total_in_millis" : 86105320
        },
        "mem" : {
          "total_virtual_in_bytes" : 1669361537024
        }
      }
    }

当然, 也可以从系统层面, 看一下当前限制

$ ps -ef | grep elasticsearch    # 找到进程的PID
elastic+ 128967      1 99 5月18 ?       1-13:22:07 /usr/share/elasticsearch/jdk/bin/java -Xms32g -Xmx32g -XX:+UseConcMarkSweepGC

$ cat /proc/128967/limits
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             4096                 4096                 processes
Max open files            65535                65535                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       514069               514069               signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us

参考文档:
https://www.elastic.co/guide/en/elasticsearch/guide/master/_file_descriptors_and_mmap.html
ElasticSearch: Unassigned Shards, how to fix?

Ubuntu使用socat进行端口转发

以前写过一篇使用iptables进行端口转发的文章, 今天写一下使用socat将本地端口的流量转发到远程机上的过程. 不要问我这样做有什么用, 我也不知道.

安装

$ sudo apt install socat

转发TCP端口

$ sudo vim /etc/systemd/system/socat.service    # 写入如下内容
[Unit]
Description=socat (https://www.zhukun.net)
After=network-online.target
Wants=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/socat TCP4-LISTEN:本地端口,reuseaddr,fork TCP4:远程IP:远程端口
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target

转发UDP端口

$ sudo vim /etc/systemd/system/socat_udp.service    # 写入如下内容
[Unit]
Description=socat_udp (https://www.zhukun.net)
After=network-online.target
Wants=network-online.target

[Service]
User=root
Group=root
ExecStart=/usr/bin/socat -T5 UDP4-LISTEN:本地端口,reuseaddr,fork UDP4:远程IP:远程端口
Restart=always
RestartSec=2

[Install]
WantedBy=multi-user.target

启动服务

$ sudo systemctl daemon-reload
$ sudo systemctl start socat.service
$ sudo systemctl start socat_udp.service
$ sudo systemctl enable socat.service
$ sudo systemctl enable socat_udp.service

在VMware中快速构建Ubuntu虚拟机

在VMware中快速构建ubuntu虚拟机, 除了本博客先前介绍的Vagrant工具以外, 今天再介绍一个办法.

OVA (Open Virtualization Appliance, 开放虚拟化设备)是一种通用的虚拟机文件, 可以在VMware/Virtualbox等常见的虚拟机中打开. 今天我们的这种方法就是下载一个Ubuntu的ova文件并导入进VMware.

1, 下载如下ova镜像

https://cloud-images.ubuntu.com/releases/bionic/release/ubuntu-18.04-server-cloudimg-amd64.ova

2, 从VMware Workstation或者VMware Player中选择"打开虚拟机", 然后选择刚下载好的.ova文件, VMware会弹出选择虚拟机存放位置, 以及如下设定界面:
在VMware中快速构建Ubuntu虚拟机

3, 等待虚拟机初始化完成, 大约需要1-3分钟时间. 然后就可以在VMware Workstation/Player中登陆了. 默认登陆用户名是ubuntu, 密码即为刚才设定的密码. 首次登陆会要求修改密码

4, 设置ssh远程登陆
如果需要使用Xshell/Putty等工具远程登陆, 可能需要做如下操作

$ sudo vim /etc/ssh/sshd_config    # 修改如下配置, 将no改为yes
...
PasswordAuthentication yes
...

$ sudo systemctl restart sshd

Continue reading “在VMware中快速构建Ubuntu虚拟机”

Logstash对Field进行简单数学计算

Logstash解析出Field以后, 可以使用filter的ruby插件进行简单数学计算/大小写转换等操作(官方介绍地址), 下面是配置

input {
  kafka{
    bootstrap_servers => ["www.hizy.net:6667,www.xpdo.net:6667","www.zhukun.net:6667"]
    client_id => "logstash_www.xpdo.net"
    group_id => "www.zhukun.net"
    auto_offset_reset => "latest"
    consumer_threads => 10
    decorate_events => false
    topics => ["www.zhukun.net"]
  }
}

filter {
    mutate {
        gsub =>["message",'\\"','"']
        gsub =>["message",'\\"','\\\\"']
    }
    json {
        source => "message"
        target => "aduser"
    }

    # 将[aduser][action][info][timestamp]映射为@timestamp
    # 需要注意的是, 即使是UNIX时间戳, 也有带毫秒和不带毫秒的, 可能是UNIX或者UNIX_MS
    date {
        match => [ "[aduser][action][info][timestamp]", "UNIX_MS" ]
        target => "@timestamp"
        locale => "cn"
    }

    # 如果这2个Field都存在, 则对它们进行相加, 形成一个新的Field
    if [aduser][action][param][vast][during_time] and [aduser][action][param][resource][during_time] {
        ruby {
            code => 'event.set("[aduser][action][param][vast_resource_during_time]", event.get("[aduser][action][param][vast][during_time]") + event.get("[aduser][action][param][resource][during_time]") )'
        }
    } else {
        drop  { }
    }
    mutate {
        remove_field => [ "message" ]
    }
}

output {
    stdout {
       codec => rubydebug {
    #       metadata => true
        }
    }
}

最后解析出来的样子是这样的
Logstash对Field进行简单数学计算

参考文档:
官方介绍地址
Simple Math Functions with Ruby in Logstash 5.3
Logstash中的数学函数