ELK环境准备与安装

清峰大约 5 分钟ELK日志收集技术ELK部署安装

ELK环境准备与安装

环境准备

1. 系统要求

操作系统

Linux：CentOS 7+/Ubuntu 16.04+/SLES 12+
Windows：Windows 10/Windows Server 2016+
macOS：10.14+ (仅用于开发测试)

硬件要求

组件	最小配置	推荐配置
CPU	2核	4核以上
内存	4GB	8GB以上
磁盘	20GB	100GB以上
Java	JDK 8+	JDK 11+

2. 软件依赖

Java环境

# 检查Java版本
java -version

# 安装OpenJDK (CentOS)
sudo yum install java-11-openjdk-devel

# 安装OpenJDK (Ubuntu)
sudo apt-get install openjdk-11-jdk

系统配置优化

# 增加文件描述符限制
echo "* soft nofile 65536" >> /etc/security/limits.conf
echo "* hard nofile 65536" >> /etc/security/limits.conf

# 调整虚拟内存设置
echo "vm.max_map_count=262144" >> /etc/sysctl.conf
sysctl -p

# 禁用swap（可选但推荐）
swapoff -a

Elasticsearch安装

1. 下载安装包

使用包管理器安装（推荐）

CentOS/RHEL:

# 导入Elasticsearch GPG密钥
sudo rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch

# 创建yum仓库文件
cat <<EOF | sudo tee /etc/yum.repos.d/elasticsearch.repo
[elasticsearch]
name=Elasticsearch repository for 7.x packages
baseurl=https://artifacts.elastic.co/packages/7.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
EOF

# 安装Elasticsearch
sudo yum install elasticsearch

Ubuntu/Debian:

# 下载并安装公钥
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -

# 安装apt-transport-https
sudo apt-get install apt-transport-https

# 保存仓库定义
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

# 安装Elasticsearch
sudo apt-get update && sudo apt-get install elasticsearch

手动下载安装

# 下载tar.gz包
wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.15.0-linux-x86_64.tar.gz

# 解压
tar -xzf elasticsearch-7.15.0-linux-x86_64.tar.gz

# 移动到指定目录
sudo mv elasticsearch-7.15.0 /usr/share/elasticsearch

2. 配置Elasticsearch

基本配置

# /etc/elasticsearch/elasticsearch.yml
cluster.name: my-elk-cluster
node.name: node-1
network.host: 0.0.0.0
http.port: 9200
discovery.seed_hosts: ["127.0.0.1"]
cluster.initial_master_nodes: ["node-1"]

内存配置

# /etc/elasticsearch/jvm.options
-Xms2g
-Xmx2g

3. 启动Elasticsearch

# 启动服务
sudo systemctl daemon-reload
sudo systemctl enable elasticsearch
sudo systemctl start elasticsearch

# 验证启动
curl -X GET "localhost:9200/?pretty"

Logstash安装

1. 下载安装包

使用包管理器安装

CentOS/RHEL:

# 安装Logstash
sudo yum install logstash

Ubuntu/Debian:

# 安装Logstash
sudo apt-get install logstash

手动下载安装

# 下载tar.gz包
wget https://artifacts.elastic.co/downloads/logstash/logstash-7.15.0-linux-x86_64.tar.gz

# 解压
tar -xzf logstash-7.15.0-linux-x86_64.tar.gz

# 移动到指定目录
sudo mv logstash-7.15.0 /usr/share/logstash

2. 配置Logstash

基本配置

# /etc/logstash/logstash.yml
http.host: "0.0.0.0"
path.config: /etc/logstash/conf.d/*.conf

JVM配置

# /etc/logstash/jvm.options
-Xms1g
-Xmx1g

3. 启动Logstash

# 启动服务
sudo systemctl enable logstash
sudo systemctl start logstash

# 验证启动
sudo systemctl status logstash

Kibana安装

1. 下载安装包

使用包管理器安装

CentOS/RHEL:

# 安装Kibana
sudo yum install kibana

Ubuntu/Debian:

# 安装Kibana
sudo apt-get install kibana

手动下载安装

# 下载tar.gz包
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.15.0-linux-x86_64.tar.gz

# 解压
tar -xzf kibana-7.15.0-linux-x86_64.tar.gz

# 移动到指定目录
sudo mv kibana-7.15.0-linux-x86_64 /usr/share/kibana

2. 配置Kibana

# /etc/kibana/kibana.yml
server.port: 5601
server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]

3. 启动Kibana

# 启动服务
sudo systemctl enable kibana
sudo systemctl start kibana

# 验证启动
sudo systemctl status kibana

Beats安装

1. Filebeat安装

使用包管理器安装

CentOS/RHEL:

# 安装Filebeat
sudo yum install filebeat

Ubuntu/Debian:

# 安装Filebeat
sudo apt-get install filebeat

手动下载安装

# 下载rpm包（CentOS）
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.15.0-x86_64.rpm
sudo rpm -vi filebeat-7.15.0-x86_64.rpm

# 下载deb包（Ubuntu）
wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.15.0-amd64.deb
sudo dpkg -i filebeat-7.15.0-amd64.deb

2. 配置Filebeat

# /etc/filebeat/filebeat.yml
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/*.log

output.elasticsearch:
  hosts: ["localhost:9200"]

3. 启动Filebeat

# 启动服务
sudo systemctl enable filebeat
sudo systemctl start filebeat

# 验证启动
sudo systemctl status filebeat

Docker部署方式

1. Docker Compose部署

# docker-compose.yml
version: '3.7'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:7.15.0
    container_name: elasticsearch
    environment:
      - discovery.type=single-node
      - ES_JAVA_OPTS=-Xms1g -Xmx1g
    ports:
      - "9200:9200"
      - "9300:9300"
    volumes:
      - esdata:/usr/share/elasticsearch/data
    networks:
      - elk

  logstash:
    image: docker.elastic.co/logstash/logstash:7.15.0
    container_name: logstash
    ports:
      - "5044:5044"
      - "9600:9600"
    volumes:
      - ./logstash/pipeline:/usr/share/logstash/pipeline
    networks:
      - elk
    depends_on:
      - elasticsearch

  kibana:
    image: docker.elastic.co/kibana/kibana:7.15.0
    container_name: kibana
    ports:
      - "5601:5601"
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
    networks:
      - elk
    depends_on:
      - elasticsearch

volumes:
  esdata:

networks:
  elk:

2. 启动Docker环境

# 启动所有服务
docker-compose up -d

# 查看服务状态
docker-compose ps

# 查看日志
docker-compose logs -f

集群部署

1. Elasticsearch集群配置

主节点配置

# /etc/elasticsearch/elasticsearch.yml
cluster.name: my-elk-cluster
node.name: master-node
node.master: true
node.data: false
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["192.168.1.101", "192.168.1.102", "192.168.1.103"]
cluster.initial_master_nodes: ["master-node"]

数据节点配置

# /etc/elasticsearch/elasticsearch.yml
cluster.name: my-elk-cluster
node.name: data-node-1
node.master: false
node.data: true
network.host: 0.0.0.0
http.port: 9200
transport.port: 9300
discovery.seed_hosts: ["192.168.1.101", "192.168.1.102", "192.168.1.103"]

2. Logstash集群配置

# logstash.conf
input {
  beats {
    port => 5044
  }
}

output {
  elasticsearch {
    hosts => ["192.168.1.101:9200", "192.168.1.102:9200", "192.168.1.103:9200"]
    index => "logstash-%{+YYYY.MM.dd}"
  }
}

安装验证

1. 验证Elasticsearch

# 检查集群状态
curl -X GET "localhost:9200/_cluster/health?pretty"

# 检查节点信息
curl -X GET "localhost:9200/_nodes?pretty"

# 创建测试索引
curl -X PUT "localhost:9200/test-index?pretty"

# 添加测试文档
curl -X POST "localhost:9200/test-index/_doc/1?pretty" -H 'Content-Type: application/json' -d'
{
  "title": "Test Document",
  "content": "This is a test document"
}
'

2. 验证Logstash

# 测试配置文件
sudo /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/test.conf

# 启动测试管道
sudo /usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/test.conf

3. 验证Kibana

# 访问Kibana界面
curl -I "http://localhost:5601"

# 检查Kibana状态
curl -X GET "http://localhost:5601/api/status"

常见问题及解决方案

1. 启动失败

Elasticsearch启动失败

# 检查日志
sudo journalctl -u elasticsearch

# 检查配置文件语法
sudo /usr/share/elasticsearch/bin/elasticsearch -t

# 检查端口占用
netstat -tlnp | grep :9200

Logstash启动失败

# 检查配置文件
sudo /usr/share/logstash/bin/logstash --config.test_and_exit -f /etc/logstash/conf.d/logstash.conf

# 查看详细日志
sudo tail -f /var/log/logstash/logstash-plain.log

2. 内存不足

# 调整JVM内存设置
# Elasticsearch: /etc/elasticsearch/jvm.options
-Xms1g
-Xmx1g

# Logstash: /etc/logstash/jvm.options
-Xms512m
-Xmx512m

3. 网络连接问题

# 检查网络连通性
ping elasticsearch-host

# 检查端口连通性
telnet elasticsearch-host 9200

# 检查防火墙设置
sudo firewall-cmd --list-all

总结

ELK Stack的安装部署需要根据实际环境和需求选择合适的安装方式。对于开发测试环境，可以使用单机部署或Docker部署；对于生产环境，建议使用集群部署以确保高可用性和性能。在安装过程中需要注意系统配置优化、安全设置和监控配置，确保ELK Stack能够稳定运行并满足业务需求。