CentOS 7解决arp欺骗

某天发现一台CentOS机器网络一切正常, 但是却无法正常上网. 后经查询, 发现该机器的网关mac地址与实际网关的mac地址不符.  下面是查看和解决办法.

查看arp

$ cat /proc/net/arp 
IP address       HW type     Flags       HW address            Mask     Device
192.168.43.62    0x1         0x2         24:6e:96:93:c9:7d     *        eth0
192.168.43.61    0x1         0x2         24:6e:96:8c:e0:65     *        eth0
192.168.43.154   0x1         0x0         00:00:00:00:00:00     *        eth0
192.168.43.153   0x1         0x2         52:54:00:fa:bb:fc     *        eth0
192.168.43.60    0x1         0x0         00:00:00:00:00:00     *        eth0
192.168.43.59    0x1         0x0         00:00:00:00:00:00     *        eth0
192.168.43.151   0x1         0x2         52:54:00:aa:73:e2     *        eth0
192.168.43.11    0x1         0x0         00:00:00:00:00:00     *        eth0
192.168.43.31    0x1         0x2         00:be:75:c7:47:ea     *        eth0
192.168.43.111   0x1         0x0         00:00:00:00:00:00     *        eth0
192.168.43.27    0x1         0x2         52:54:00:b4:3f:a3     *        eth0
192.168.43.224   0x1         0x0         00:00:00:00:00:00     *        eth0
192.168.43.26    0x1         0x2         52:54:00:33:50:7e     *        eth0
192.168.43.223   0x1         0x0         00:00:00:00:00:00     *        eth0
192.168.43.25    0x1         0x2         52:54:00:b4:3f:a3     *        eth0
192.168.43.1     0x1         0x2         3c:f5:cc:91:79:87     *        eth0
192.168.43.163   0x1         0x2         52:54:00:78:13:7d     *        eth0
192.168.43.162   0x1         0x2         52:54:00:7f:83:a4     *        eth0
192.168.43.104   0x1         0x2         52:54:00:34:0c:fc     *        eth0
192.168.43.21    0x1         0x0         00:00:00:00:00:00     *        eth0
192.168.43.44    0x1         0x2         24:6e:96:93:a3:c4     *        eth0
192.168.43.43    0x1         0x2         24:6e:96:8c:df:64     *        eth0
192.168.43.158   0x1         0x2         52:54:00:9a:ff:9f     *        eth0
192.168.43.122   0x1         0x2         52:54:00:99:e5:5e     *        eth0


$ arp -a
? (192.168.43.62) at 24:6e:96:93:c9:7d [ether] on eth0
? (192.168.43.61) at 24:6e:96:8c:e0:65 [ether] on eth0
? (192.168.43.154) at <incomplete> on eth0
? (192.168.43.153) at 52:54:00:fa:bb:fc [ether] on eth0
? (192.168.43.60) at <incomplete> on eth0
? (192.168.43.59) at <incomplete> on eth0
? (192.168.43.151) at 52:54:00:aa:73:e2 [ether] on eth0
? (192.168.43.11) at <incomplete> on eth0
? (192.168.43.31) at 00:be:75:c7:47:ea [ether] on eth0
? (192.168.43.111) at <incomplete> on eth0
? (192.168.43.27) at 52:54:00:b4:3f:a3 [ether] on eth0
? (192.168.43.224) at <incomplete> on eth0
? (192.168.43.26) at 52:54:00:33:50:7e [ether] on eth0
? (192.168.43.223) at <incomplete> on eth0
? (192.168.43.25) at 52:54:00:b4:3f:a3 [ether] on eth0
gateway (192.168.43.1) at 3c:f5:cc:91:79:87 [ether] on eth0
? (192.168.43.163) at 52:54:00:78:13:7d [ether] on eth0
? (192.168.43.162) at 52:54:00:7f:83:a4 [ether] on eth0
? (192.168.43.104) at 52:54:00:34:0c:fc [ether] on eth0
? (192.168.43.21) at <incomplete> on eth0
? (192.168.43.44) at 24:6e:96:93:a3:c4 [ether] on eth0
? (192.168.43.43) at 24:6e:96:8c:df:64 [ether] on eth0
? (192.168.43.158) at 52:54:00:9a:ff:9f [ether] on eth0

绑定arp

绑定arp的过程在某些国外网站称之为Create a Static ARP Table. 下面演示手动绑定网关192.168.43.1的mac地址为74:ea:c8:2d:9f:f6

arp -s 192.168.43.1 74:ea:c8:2d:9f:f6

 

使用python判断IP段可用IP及数量

使用python判断IP段可用IP及数量, 很简单.几个命令就可以(本文基于python3).

>>> import ipaddress
>>> for ip in ipaddress.ip_network('192.168.0.0/28'):
...     print(ip)
...
192.168.0.0
192.168.0.1
192.168.0.2
192.168.0.3
192.168.0.4
192.168.0.5
192.168.0.6
192.168.0.7
192.168.0.8
192.168.0.9
192.168.0.10
192.168.0.11
192.168.0.12
192.168.0.13
192.168.0.14
192.168.0.15
>>>
>>> ipaddress.ip_network('192.168.0.0/28').num_addresses
16

批量计算

$ cat 2
172.16.128.0/18
172.16.32.0/20
172.16.64.0/19
172.19.192.0/19
172.16.240.0/21
172.16.48.0/20
172.16.192.0/19
172.19.160.0/19
172.19.64.0/18
172.16.24.0/21
172.16.96.0/19
172.19.128.0/19


$ python3
>>> import ipaddress
>>> with open("./2", "r") as f:
...     for i in f.readlines():
...         print(ipaddress.ip_network(i.rstrip()).num_addresses)
...
16384
4096
8192
8192
2048
4096
8192
8192
16384
2048
8192
8192

 

ElasticSearch DSL聚合查询语句

本来像聚合(aggregation)这种东西, 在Grafana中可以轻易的实现, 但是偶尔会有需求, 需要自己写DSL脚本实现一些功能, 于是, 只好自己动手了.

例子1

查询serverName=”dns-server-1″结果里, 按hostip的数量进行排序, 取前5

GET /my-service-2020.07.22/_search
{
  "query": {
    "term": { "serverName.keyword": "dns-server-1" }
  },
  "size" : 0,
  "aggs": {
    "top-10-hostip": {
      "terms": {
      	"field": "hostip.keyword",
        "size": 5
      }
    }
  }
}

结果 Continue reading “ElasticSearch DSL聚合查询语句”

ElasticSearch索引/数据定期清理

关于定期清理ElasticSearch索引, 最简单粗暴的方法是写一个shell脚本, 实现定理删除INDEX. 但其实ElasticSearch官网也提供了一些工具来做这些事, 比如下面2个方法.

1, ILM: Manage the index lifecycle

应该是最简单有用的清理INDEX的办法了(官方文档在此, 一个简单的范例在此), 是X-Pack自带的功能, 不需要安装额外工具. ILM的主要功能有

  1. 当index容量达到一定数值(例如50G), 或者其中的日志数量达到一定数值以后, 开启一个新index
  2. 定期把旧index移动到旧的硬件节点上
  3. 指定什么情况下可以修改replicas数量, 或者修改一个index的主分片数量, 或者指定什么情况可以Force merge segments
  4. 定期删除旧index

2, Curator

也是ElasticSearch官方的工具, 需要额外安装(下载地址).  这个工具最早是clearESindices.py演化而来的, 最早的目的就是清理删除Index, 再后来, 随着作者被Elasticsearch公司聘用, 这个工具也被更名为Elasticsearch Curator. 它使用yaml作为基础配置语法, 官网提供了一堆Example配置可以参考.

$ cat /etc/elasticsearch/curator-cfg.yml
client:
  hosts:
    - 172.29.4.158
    - 172.29.4.157
    - 172.29.4.156
  port: 9200
  use_ssl: False
  http_auth: "elastic:MyPassword"
  timeout: 30

logging:
  loglevel: INFO
  logformat: default
  logfile: /var/log/elasticsearch/curator.log
$ cat /etc/elasticsearch/curator-del.yml
actions:
  1:
    action: delete_indices
    description: >-
      Delete old system indexes.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: prefix
      value: .monitoring-kibana-7-
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 3
  2:
    action: delete_indices
    description: >-
      Delete old indexes.
    options:
      ignore_empty_list: True
      timeout_override:
      continue_if_exception: False
      disable_action: False
    filters:
    - filtertype: pattern
      kind: regex
      value: '^(office_dns_log-|office_dns_log_failover-|mail-|mail_failover-).*$'
    - filtertype: age
      source: name
      direction: older
      timestring: '%Y.%m.%d'
      unit: days
      unit_count: 180

dry-run运行试一下

/usr/bin/curator --config /etc/elasticsearch/curator-cfg.yml --dry-run /etc/elasticsearch/curator-del.yml

然后可以观察下/var/log/elasticsearch/curator.log文件里的提示. 确认没问题后, 将–dry-run去掉并写入crontab即可.

参考文档:
Automatically removing index

转载: 使用tcpdump抓包

1, tcpdump常用选项介绍

-n 禁止IP名称解析
-nn 禁止IP和端口名称解析
-i 指定捕获哪个网卡的网络数据包。
-w 指定将包写入哪个文件,如果文件不存在则创建该文件;如果存在则覆盖其内容
-f 指定过滤表达式,例如指定捕获哪个端口,哪个协议等
-r 指定从哪个文件读取网络数据包文件
-F 指定使用哪个文件的过滤表达式抓包
-D 列出所有可以使用tcpdump抓包的网卡
-c 指定捕获或者读取包的个数,-c后面直接接数字即可
-l 抓包时保存到文件的同时查看包的内容
-t 不打印时间戳
-tt 秒级时间戳
-ttt 打印时间戳到微秒或者纳秒,取决于 –time-stamp-precision option 选项
-s 指定每个包捕获的字节数
-S 打印绝对的tcp序列号,而不是相对的序列号
-v/-vv/-vvv 打印详细信息,v的个数越多, 打印内容越详细

上面是常用的选项, 更多的选项请参考tcpdump官方文档, 下面将对使用过滤条件抓包进行基本的介绍

2, tcpdump常用命令

#协议为tcp, 目标端口或源端口为80的包, 并将其写入packets.pcap文件中
tcpdump -nni ens33 -w packets.pcap 'tcp port 80'

#协议为tcp, 目标端口为80
tcpdump -nni ens33 -w packets.pcap 'tcp dst port 80' -c10

#协议类型为tcp, 源端口为80
tcpdump -nni ens33 -w packets.pcap 'tcp src port 80' -c10

#读取文件中协议类型为tcp, 目标端口为80的包
tcpdump -nnr packets.pcap 'tcp dst port 80' -c10

#将packets.pcap文件中目标端口为443的包转存到dst_port_443.pcap中
tcpdump -r packets.pcap 'dst port 443' -w dst_port_443.pcap 

#指定IP地址为14.215.177.39
tcpdump -nni ens33 host 14.215.177.39 -c5

#源IP地址为192.168.248.134
tcpdump -nni ens33 src 192.168.248.134 -c5

#目标IP地址为192.168.248.134
tcpdump -nni ens33 dst 192.168.248.134 -c5

#通往网络192.168.248.0/24
tcpdump -nni ens33 net 192.168.248.0/24 -c5

本文来源:
tcpdump使用过滤条件抓包(基础篇)

解决访问kibana monitoring 被提示Access Denied

在Kibana中使用”Stack Monitoring”时, 提示

Access Denied

You are not authorized to access Monitoring. To use Monitoring, you need the privileges granted by both the `kibana_user` and `monitoring_user` roles.

If you are attempting to access a dedicated monitoring cluster, this might be because you are logged in as a user that is not configured on the monitoring cluster.

解决办法: 停用Elasticsearch集群的remote.cluster功能, 将现有remote.cluster全部清除即可.

# 查看现有的 remote cluster
curl -XGET "127.0.0.1:9200/_cluster/settings?pretty"
{
  "persistent" : {
    "cluster" : {
      "remote" : {
        "aaa" : {
          "skip_unavailable" : "true",
          "seeds" : [
            "172.29.4.168:9300"
          ]
        },
        "leader" : {
          "seeds" : [
            "172.29.4.168:9300"
          ]
        },
        "hello-elk" : {
          "skip_unavailable" : "false",
          "seeds" : [
            "127.0.0.1:9300"
          ]
        }
      }
    },
    "xpack" : {
      "monitoring" : {
        "collection" : {
          "enabled" : "true"
        }
      }
    }
  },
  "transient" : { }
}

# 清除其中一个 remote cluster 节点
curl -X PUT "127.0.0.1:9200/_cluster/settings" -H 'Content-Type: application/json' -d'{
  "persistent" : {
    "cluster" : {
      "remote" : {
        "leader" : {
          "seeds" : null
        }
      }
    }
  }
}'

提示: 如果一个remote cluster节点设置了”skip_unavailable” : “true”信息, 直接清除可能会提示Cannot configure setting [cluster.remote.hello-elk.skip_unavailable] if remote cluster is not enabled. 解决办法为, 先将skip_unavailable设置为null, 再将seeds设置为null

Migrate data for Elasticsearch cluster

迁移ElasticSearch集群的数据, 最好用的是用到ElasticSearch的CCR(Cross-cluster replication, 跨集群复制)功能(官方文档在此). 但无奈今天配置了一天, 怎么也没有成功. 其实CCR存在的意义不仅仅是迁移数据, 更重要的是保证ElasticSearch集群的多副本/高可用状态. 比如, 如果你的主ES集群不能对外暴露, 那么可以设置一个readonly的对外暴露集群(数据通过CCR功能与主集群保持同步, 等. 而如果仅仅是迁移数据的话, 只用到ES的reindex功能即可完成.

将旧集群(172.29.4.168:9200)里的mail-w3svc1-2020.06.06索引数据迁移过来, 仅需要在新集群上执行如下命令即可.

curl -X POST "localhost:9200/_reindex?pretty" -H 'Content-Type: application/json' -d'
{
  "source": {
    "remote": {
      "host": "http://172.29.4.168:9200",
      "username": "elastic",
      "password": "MyPassword"
    },
    "index": "mail-w3svc1-2020.06.06"
  },
  "dest": {
    "index": "mail-2020.06.06"
  }
}'

参考文档:
Reindex API