环境:被监控系统版本redhat7.9 splunk版本9.X.X splunk服务器地址10.10.10.10
前提条件:splunk服务器已经配置好了告警邮件服务器,接收9997端口数据,已新建索引testdata,终端可以访问splunk服务器的9997/tcp端口
一、被监控redhat系统安装客户端
1、将安装包上传到redhat系统/tmp目录
2、安装splunk客户端
rpm -ivh splunkforwarder-9.3.2-d8bb32809498.x86_64.rpm安装过程中出现的下述错误对客户端基本功能使用不会造成影响
[root@XXX tmp]# rpm -ivh splunkforwarder-9.3.2-d8bb32809498.x86_64.rpm
warning: splunkforwarder-9.3.2-d8bb32809498.x86_64.rpm: Header V4 RSA/SHA256 Signature, key ID b3cd4420: NOKEY
Preparing... ################################# [100%]
Updating / installing...
1:splunkforwarder-9.3.2-d8bb3280949################################# [100%]
find: ‘/opt/splunkforwarder/lib/python3.7/site-packages’: No such file or directory
find: ‘/opt/splunkforwarder/lib/python3.9/site-packages’: No such file or directory
complete
3、配置客户端日志外发
新建并编辑outputs.conf文件
vi /opt/splunkforwarder/etc/system/local/outputs.conf内容为:
[tcpout] defaultGroup = primary_splunkservers [tcpout:primary_splunkservers] server = 10.10.10.10:9997 compressed = true4、配置客户端日志收集
新建并编辑inputs.conf文件
vi /opt/splunkforwarder/etc/system/local/inputs.conf内容为:
[monitor:///var/log/messages] sourcetype = linux_messages_log index = testdata disabled = 05、运行splunk客户端
启动客户端服务
/opt/splunkforwarder/bin/splunk start[root@xxxx tmp]# /opt/splunkforwarder/bin/splunk start
Warning: Attempting to revert the SPLUNK_HOME ownership
Warning: Executing "chown -R splunkfwd:splunkfwd /opt/splunkforwarder"
SPLUNK GENERAL TERMS
Last Updated: August 12, 2021
These Splunk General Terms ("General Terms") between Splunk Inc., a Delaware
corporation, with its principal place of business at 270 Brannan Street, San
Francisco, California 94107, U.S.A ("Splunk" or "we" or "us" or "our") and you
("Customer" or "you" or "your") apply to the purchase of licenses and
subscriptions for Splunk's Offerings. By clicking on the appropriate button,
or by downloading, installing, accessing or using the Offerings, you agree to
these General Terms. If you are entering into these General Terms on behalf of
Customer, you represent that you have the authority to bind Customer. If you
do not agree to these General Terms, or if you are not authorized to accept
the General Terms on behalf of the Customer, do not download, install, access,
or use any of the Offerings.
See the General Terms Definitions Exhibit attached for definitions of
capitalized terms not defined herein.
XXXXXXXXXXXXXXXXXXXXX中间有很多内容直接拖到底
"Splunk Preexisting IP" means, with respect to any C&I Services Materials, all
associated Splunk technology and all Intellectual Property Rights created or
acquired: (a) prior to the date of the Statement of Work that includes such
C&I Services Materials, or (b) after the date of such Statement of Work but
independently of the C&I Services provided under such Statement of Work.
"Statement of Work" means the statements of work and/or any and all applicable
Orders, that describe the specific services to be performed by Splunk,
including any materials and deliverables to be delivered by Splunk.
Do you agree with this license? [y/n]:
yThis appears to be your first time running this version of Splunk.
Splunk software must create an administrator account during startup. Otherwise, you cannot log in.
Create credentials for the administrator account.
Characters do not appear on the screen when you type in credentials.
Please enter an administrator username: admin
admin
Password must contain at least:
* 8 total printable ASCII character(s).
Please enter a new password:
Please confirm new password:
PasswordCreating unit file...
Important: splunk will start under systemd as user: splunkfwd
The unit file has been created.
Splunk> CSI: Logfiles.
Checking prerequisites...
Checking mgmt port [8089]: open
Creating: /opt/splunkforwarder/var/lib/splunk
Creating: /opt/splunkforwarder/var/run/splunk
Creating: /opt/splunkforwarder/var/run/splunk/appserver/i18n
Creating: /opt/splunkforwarder/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunkforwarder/var/run/splunk/upload
Creating: /opt/splunkforwarder/var/run/splunk/search_telemetry
Creating: /opt/splunkforwarder/var/run/splunk/search_log
Creating: /opt/splunkforwarder/var/spool/splunk
Creating: /opt/splunkforwarder/var/spool/dirmoncache
Creating: /opt/splunkforwarder/var/lib/splunk/authDb
Creating: /opt/splunkforwarder/var/lib/splunk/hashDb
Creating: /opt/splunkforwarder/var/run/splunk/sessions
New certs have been generated in '/opt/splunkforwarder/etc/auth'.
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunkforwarder/splunkforwarder-9.3.2-d8bb32809498-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Done
[ OK ]
客户端这边的账户密码正常情况下不会使用到,但建议记录到文档中留存。
二、将日志正文内容添加到message字段
1、在splunk平台查看收集到的日志
在新搜索中搜索index="testdata",即可搜索到/var/log/messages中的日志,此时正文内容不属于任何字段。
2、在当前页面右侧点击“+提取新字段”
3、在新出现的窗口中随便选择“我更喜欢自己编写正则表达式”
4、正则表达式填写 ^(?P<message>.+) 即获取日志的所有正文内容,点击预览查看是否所有的日志正文都变成message字段了,可以看到_raw字段中所有内容都变蓝了,且和message字段中的内容一致,这说明正则表达式的内容获取了日志的所有正文内容。
5、点击保存
6、在新出现的窗口查看信息是否正确,字段的提取名称默认即可
7、点击完成,会出现提示成功的窗口
8、再看之前的日志,已经多了一行message信息
三、根据message信息设置邮件告警
新建邮件告警搜索内容如下
根据上述内容设置系统日志中出现error字样发送告警效果如下