Press "Enter" to skip to content

Category: 电脑网络

电脑知识,网络资讯

logstash的drop过滤器插件

logstash在filter段对日志进行解析的时候, 可以直接筛选出我们想要的日志内容, 如果日志内容里不包括某些字段, 我们可以把整条日志直接扔掉, 下面是配置.

input {
    kafka {
        bootstrap_servers => k1.zhukun.net:6687 k2.zhukun.net:6687
        topics => ["com.prod.feedengine","com.prod.feedgateway"]
        # 如果收取多个kafaka topic里的消息也可以用下面的写法
        # topics_pattern => "zhukun.net.log.rms-api.*"
        group_id => logstash-mp-ops
        consumer_threads => 10
        decorate_events => true
        auto_offset_reset => "latest"
    }
}

filter {

    # 如果message里不以2019/2020/2021开头, 则直接丢弃整条日志
    if [message] !~ /^[2020|2021|2019]/ {
        drop { }
    }

    # 直接打印出来原始日志看看
    #ruby {
    #    code => 'puts event(message)'
    #}

    # grop正则匹配
    grok {
        match => { message => '%{TIMESTAMP_ISO8601:time_local}\s*\[%{DATA:service}\]\s*%{LOGLEVEL:loglevel}\s*%{DATA:message}$' }
        overwrite => [message]
        tag_on_failure => ["_invalid_log_format"]    # 如果解析失败则加上这个tag
    }

    # 如果日志解析成功,那么
    if !("_invalid_log_format" in [tags]) {
        mutate {
            # 如果把整条日志都解析出来以后(已经解析到各个tag之中), 原始日志应该也没什么用了, 可以考虑直接扔掉原始日志
            remove_field => [ "message" ]
            # 将kafka topic的名字作为oootype字段
            add_field => { "oootype" => "%{[@metadata][kafka][topic]}" }
            gsub => [
                "logInfo", "\t\t", ""
            ]
        }

        # 日期处理
        date {
            # 将time_local赋给@timestamp字段, 右侧是time_local的实际格式, 例如2019-03-18 08:12:45.006
            match => ["time_local", "yyyy-MM-dd HH:mm:ss.SSS"]
            # match => [ "logTime", "ISO8601" ]
            # timezone => "Asia/Shanghai"
            target => "@timestamp"    # 默认target就是@timestamp
            tag_on_failure => [ "_dateparsefailure" ]
            # remove_field => [ "time_local" ]
        }

    }
}

output {
    elasticsearch {
        hosts => [10.18.4.24:9200,10.18.4.25:9200,10.18.4.77:9200,10.18.4.78:9200, 10.11.149.69:9200,10.16.22.149:9200]
        index => zhukun.net_console.log-%{+yyyy.MM.dd}
    }
    #stdout {
    #   codec => rubydebug {
    #       metadata => true
    #    }
    #}
}

补充:
logstash可以使用条件判断来控制filter的执行。官方说明见Accessing Event Data and Fields in the Configuration。支持的运算符包括:

相等: ==, !=, <, >, <=, >=
正则: =~(匹配正则), !~(不匹配正则)
包含: in(包含), not in(不包含)
布尔操作: and(与), or(或), nand(非与), xor(非或)
一元运算: !(取反), ()(复合表达式), !()(对复合表达式结果取反)

参考文档
Drop filter plugin
Logstash Grep and Drop
Missing grep filter in logstash

1 Comment

WordPress post from xmlrpc

从Wordpress 3.5版本开始, xmlrpc默认开启, 更早期的版本可能在设置中手动开启. 以下是一个通过xmlrpc发布文章的脚本(python版本):

vim post.py
#!/usr/bin/python
# -*- coding: utf-8 -*-

import datetime, xmlrpclib

wp_url = "http://www.example.net/xmlrpc.php"
wp_username = "USER"
wp_password = "passwd"
wp_blogid = ""
status_draft = 0
status_published = 1

server = xmlrpclib.ServerProxy(wp_url)

title = "article title"
content = "article content"
#date_created = xmlrpclib.DateTime(datetime.datetime.strptime("2011-10-20 21:08", "%Y-%m-%d %H:%M"))
categories = ["Beijing"]
tags = ["sometag", "othertag"]
data = {'title': title, 'description': content, 'categories': categories, 'mt_keywords': tags}

post_id = server.metaWeblog.newPost(wp_blogid, wp_username, wp_password, data, status_published)

请注意: 如果脚本里的categories不存在, 则会把文章post到默认分为中.

3 Comments

systemd service log check

systemd服务一般使用systemctl命令来启动, 那么如果服务启动不正常, 如何查看它的日志呢?

journalctl -u myapp.service

如果想查看实时日志输出( follow logs in real time ), 可以加一个-f参数:

journalctl -u -f myapp.service

另外2个有用的参数:

-e          直接打印到日志末尾;
--no-pager  打印全部日志(无需再翻页).

其实systemctl也是可以直接查看日志的:

systemctl -l status myapp.service
1 Comment

ansible use timestamp as vars

ansible 使用时间戳作为变量

- hosts: app_group
  remote_user: root
  gather_facts: yes    # 需要设定为yes(也可以省略),不然会提示ansible_date_time' is undefined
  vars:
    - VERSION:  v1.0.0
    - GIT_APP_NAME: HelloWorld
    - GIT_ADDR: [email protected]/yourname/helloworld.git
    - LOCAL_CODE_DIR: '/opt/ansible/yourname/code'
    - CMD_TIME: "{{ ansible_date_time.iso8601_basic_short }}"    # 获取系统时间戳
  tasks:

    - name: sync code to build server
      synchronize:
        src: "{{ LOCAL_CODE_DIR }}/{{ GIT_APP_NAME }}/" 
        dest: "/opt/src-{{ VERSION }}-{{ CMD_TIME }}"

    - name: 编译
      shell: "cd /opt/src-{{ VERSION }}-{{ CMD_TIME }}; make clean && make -j12;"

    - name: 同步新版本文件到本地
      synchronize
        src: "/opt/src-{{ VERSION }}-{{ CMD_TIME }}/helloworld"
        dest: "{{ LOCAL_CODE_DIR }}/helloworld_online/"
        mode: pull

    - name: 同步新版本文件到线上
      synchronize:
        src: "{{ LOCAL_CODE_DIR }}/helloworld_online/helloworld"
        dest: "/opt/helloworld/helloworld"

    - name: 计算线上md5值
      shell: "md5sum /opt/helloworld/helloworld |awk '{print $1}'"
      register: helloworld_online

    - name: 计算本地md5值
      shell: "md5sum {{ LOCAL_CODE_DIR }}/helloworld_online/helloworld | awk '{print $1}'"
      register: helloworld_local
      connection: local    # 表示在本地执行相关命令

    - name: md5 check
      debug:
        msg: 'remote: {{ helloworld_online.stdout }} local: {{ helloworld_local.stdout }}'

    - name: 本地与线上md5值对比成功重启服务(随机sleep 5秒,确保服务可用率)
      when: helloworld_online.stdout == helloworld_local.stdout
      shell: "sleep $((RANDOM % 5)) && cd /opt/helloworld/ && sh ./start.sh"
      args:
        executable: /bin/bash

1 Comment

OpenWrt路由器获取系统信息

我的OpenWrt路由器信息

cat /etc/openwrt_release
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='23.05.3'
DISTRIB_REVISION='r23809-234f1a2efa'
DISTRIB_TARGET='mediatek/filogic'
DISTRIB_ARCH='aarch64_cortex-a53'
DISTRIB_DESCRIPTION='OpenWrt 23.05.3 r23809-234f1a2efa'


cat /etc/os-release
NAME="OpenWrt"
VERSION="21.02-SNAPSHOT"
ID="openwrt"
ID_LIKE="lede openwrt"
PRETTY_NAME="OpenWrt 21.02-SNAPSHOT"
VERSION_ID="21.02-snapshot"
HOME_URL="https://openwrt.org/"
BUG_URL="https://bugs.openwrt.org/"
SUPPORT_URL="https://forum.openwrt.org/"
BUILD_ID="r15812+885-46b6ee7ffc"
OPENWRT_BOARD="mediatek/mt7981"
OPENWRT_ARCH="aarch64_cortex-a53"
OPENWRT_TAINTS="no-all busybox"
OPENWRT_DEVICE_MANUFACTURER="OpenWrt"
OPENWRT_DEVICE_MANUFACTURER_URL="https://openwrt.org/"
OPENWRT_DEVICE_PRODUCT="Generic"
OPENWRT_DEVICE_REVISION="v0"
OPENWRT_RELEASE="OpenWrt 21.02-SNAPSHOT r15812+885-46b6ee7ffc"


. /etc/openwrt_release && echo "$DISTRIB_RELEASE"
21.02-SNAPSHOT


. /etc/openwrt_release && echo "$DISTRIB_ARCH"
aarch64_cortex-a53


# Get repo link
echo "https://downloads.openwrt.org/releases/$(. /etc/openwrt_release && echo "$DISTRIB_RELEASE")/packages/$(. /etc/openwrt_release && echo "$DISTRIB_ARCH")/packages"


# Write repo
echo "
src/gz openwrt_base https://downloads.openwrt.org/releases/$(. /etc/openwrt_release && echo "$DISTRIB_RELEASE")/packages/$(. /etc/openwrt_release && echo "$DISTRIB_ARCH")/base
src/gz openwrt_luci https://downloads.openwrt.org/releases/$(. /etc/openwrt_release && echo "$DISTRIB_RELEASE")/packages/$(. /etc/openwrt_release && echo "$DISTRIB_ARCH")/luci
src/gz openwrt_packages https://downloads.openwrt.org/releases/$(. /etc/openwrt_release && echo "$DISTRIB_RELEASE")/packages/$(. /etc/openwrt_release && echo "$DISTRIB_ARCH")/packages
src/gz openwrt_routing https://downloads.openwrt.org/releases/$(. /etc/openwrt_release && echo "$DISTRIB_RELEASE")/packages/$(. /etc/openwrt_release && echo "$DISTRIB_ARCH")/routing
" | tee -a "/etc/opkg/customfeeds.conf"

我的OpenWrt使用的是哪种架构的CPU?

opkg print-architecture | awk '{print $2}' | grep -v all | grep -v noarch   # 精确的命令
mips_24kc

$ uname -m    # 只能得到大致架构信息
mips

$ . /etc/openwrt_release ; echo $DISTRIB_ARCH
mips_24kc

我的OpenWrt是32位的还是64位的?

cat /proc/cpuinfo
Processor       : ARMv7 Processor rev 10 (v7l)

uname -m
armv7l    # 如果是64位系统, 这里会有64位字样出现

提示: ARMv7 都是32位CPU, armv8以上才是64位CPU.

$ echo $SHELL
/bin/ash

$ file /bin/ash
/bin/ash: symbolic link to busybox

$ which busybox
/bin/busybox

$ file /bin/busybox 
/bin/busybox: ELF 32-bit MSB executable, MIPS, MIPS32 rel2 version 1...
1 Comment

可以代替Dropbox的云存储服务

连续使用了3年的Dropbox收费套餐, 自从本月再次从信用卡自动扣费99美元以后, 突然有了那么一丝丝的心疼, 觉得Dropbox的价格还是蛮贵的, 毕竟人民币700元/年的云存储服务, 还是过于奢侈. 于是又想着寻找一些Dropbox的替代品了.

1, Sync.com

sync.com是首先映入眼帘的, 国外的一些网站给了它很高的评价, 称之为替代Dropbox最好的选择.

优点:

1, 免费用户5G空间, 每邀请一个用户, 可以获得额外1GB的空间, 总共可以获得多少未知;
2, 客户端支持支持全面, Windows/MacOS及移动端Android/iOS均支持;
3, 经测试, Windows客户端支持使用代理, 很好的解决了服务在国内不可用的问题;
4, 界面非常简洁, 使用简单.

缺点:

1, 收费套餐最低是Personal Mini套餐, 200G容量60美元/年, 价格约为Dropbox的一半, 尚在可接受范围
2, 没有Linux客户端;

2, pcloud.com

由于我已经购买了pcloud.com家500G空间的永久空间(一次买断的方案), 所以这个云存储服务也在我的考虑范围内.

优点:

1, 官方经常推出500G/2TB等一次性买断空间的方案, 即一次买断500G/2TB可终身使用(不用每月付费);
2, 客户端支持非常全面, 连Linux版本都有;

2 Comments