Filebeat和Logstash都是ES套件(ES stack)中的组成部分, 其中, Filebeat还是beats家族的成员之一. Filebeat和Logstash都可以将日志文件输出到ElasticSearch, 且众所周知, Filebeat非常轻量级, 而Logstash由于使用JVM的原因性能堪忧, 那么是不是说我们可以抛弃笨重的Logstash了呢?
What is the difference between Logstash and Beats?
Beats are lightweight data shippers that you install as agents on your servers to send specific types of operational data to Elasticsearch. Beats have a small footprint and use fewer system resources than Logstash.
Logstash has a larger footprint, but provides a broad array of input, filter, and output plugins for collecting, enriching, and transforming data from a variety of sources.
翻译:
Logstash和Beats有什么区别？
Beats是轻量级数据托运者，您可以在服务器上将其作为代理安装，以将特定类型的操作数据发送到Elasticsearch。与Logstash相比，节拍占用空间小，使用的系统资源更少。
Logstash具有更大的占用空间，但提供了大量的输入，过滤和输出插件，用于收集，丰富和转换来自各种来源的数据。
If you’re using ELK as your logging solution, one way to ship these logs is using Filebeat to send the data directly into Elasticsearch. Since Filebeat ships data in JSON format, Elasticsearch should be able to parse the timestamp and message fields without too much hassle. Not only that, Filebeat also supports an Apache module that can handle some of the processing and parsing.
However, as of yet, advanced log enhancement — adding context to the log messages by parsing them up into separate fields, filtering out unwanted bits of data and enriching others — cannot be handled without Logstash.
Tough configuration, resource consumption, frequent crashes, and difficult troubleshooting have all made Logstash the weakest link in the ELK Stack. Yet despite these flaws, Logstash still remains a crucial component of the stack.
Elastic has made big steps in trying to alleviate these pains by introducing Beats (and adding a visual element to Logstash pipelines in the future version 6.0), which has enabled users to build and configure multiple and resilient logging pipelines — and ultimately make logging with ELK much more reliable.
翻译:
如果您使用ELK作为日志记录解决方案，那么发送这些日志的一种方法是使用Filebeat将数据直接发送到Elasticsearch。由于Filebeat以JSON格式发送数据(新版Filebeat使用YAML格式)，因此Elasticsearch应该能够解析时间戳和消息字段而不会有太多麻烦。不仅如此，Filebeat还支持可以处理一些处理和解析的Apache模块。
但是，到目前为止，高级日志增强 – 通过将日志消息分解为单独的字段来添加上下文，过滤掉不需要的数据并丰富其他内容 – 如果没有Logstash，则无法处理。
严格的配置，资源消耗，频繁的崩溃和难以排除故障都使得Logstash成为ELK Stack中最薄弱的环节。尽管有这些缺陷，Logstash仍然是堆栈的关键组成部分。
Elastic通过引入Beats（以及在未来版本6.0中向Logstash管道添加可视元素）尝试缓解这些痛苦，已经迈出了重要的一步，这使得用户能够构建和配置多个弹性日志管道 – 并最终使用ELK进行日志记录更可靠。
Logstash is useful if you need to aggregate logs from many servers and apply some common transformations and filtering to your events.
If your log events are already structured and you are ok with indexing them directly, then you can definitely have Filebeat send them directly to ES. If ES goes down (e.g. for maintenance), Filebeat will retry until it can successfully send the events.
翻译:
如果您需要聚合来自许多服务器的日志并应用一些常见转换并过滤到您的事件，则Logstash非常有用。
如果您的日志事件已经结构化并且您可以直接索引它们，那么您肯定可以将Filebeat直接发送给ES。如果ES关闭（例如进行维护），Filebeat将重试，直到它可以成功发送事件。
参考文档:
What is the difference between Logstash and Beats?
Filebeat vs. Logstash — The Evolution of a Log Shipper
FileBeat directly to ELS or via LogStash?