环境准备

本次实验共需要用到三台主机，两台部署elasticsearch、一台部署logstash和logstash，主机分配如下：
192.168.1.15 es-master
192.168.1.16 es-node
192.168.1.17 logstash-kibana

修改主机名

192.168.1.15主机

hostnamectl set-hostname es-master

192.168.1.16主机

hostnamectl set-hostname es-node

192.168.1.17主机

hostnamectl set-hostname logstash-kibana

下载软件包

192.168.1.15和192.168.1.16主机执行如下命令下载elasticsearch软件包

wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-7.14.1-linux-x86_64.tar.gz

192.168.1.17主机执行如下命令下载logstash和kibana软件包

wget https://artifacts.elastic.co/downloads/logstash/logstash-7.14.1-linux-x86_64.tar.gz
wget https://artifacts.elastic.co/downloads/kibana/kibana-7.14.1-linux-x86_64.tar.gz

部署elasticsearch集群

解压elasticsearch压缩包到指定目录

192.168.1.15和192.168.1.16主机执行如下命令

tar -xf /root/elasticsearch-7.14.1-linux-x86_64.tar.gz -C /usr/local/

修改配置文件

192.168.1.15主机执行如下命令

vi /usr/local/elasticsearch-7.14.1/config/elasticsearch.yml
cluster.name: es-test    #elasticsearch集群名称
node.name: es-master  #结群节点名称建议写主机名
path.data: /elasticsearch/data  #elasticsearch索引数据存放路径
path.logs: /elasticsearch/logs   #elasticsearch日志存放路径
network.host: 0.0.0.0   #设置所有地址监听访问
http.port: 9200   #elasticsearch服务端口
discovery.seed_hosts: ["192.168.1.15", "192.168.1.16"]  #集群地址池若配置了主机名称解析该地方也可以写主机名
cluster.initial_master_nodes: ["192.168.1.15"]  #声明主节点

192.168.1.16主机执行如下命令

vi /usr/local/elasticsearch-7.14.1/config/elasticsearch.yml
cluster.name: es-test    #elasticsearch集群名称
node.name: es-node  #结群节点名称建议写主机名
path.data: /elasticsearch/data  #elasticsearch索引数据存放路径
path.logs: /elasticsearch/logs   #elasticsearch日志存放路径
network.host: 0.0.0.0   #设置所有地址监听访问
http.port: 9200   #elasticsearch服务端口
discovery.seed_hosts: ["192.168.1.15", "192.168.1.16"]  #集群地址池若配置了主机名称解析该地方也可以写主机名
cluster.initial_master_nodes: ["192.168.1.15"]  #声明主节点

配置elasticsearch集群认证

在主节点192.168.1.15主机操作

生成证书

cd /usr/local/elasticsearch-7.14.1/bin/
./elasticsearch-certutil ca

查看证书文件

设置连接证书私钥文件

cd /usr/local/elasticsearch-7.14.1/bin/
./elasticsearch-certutil cert --ca elastic-stack-ca.p12

查看证书私钥文件

将证书文件和私钥文件拷贝到node节点192.168.1.16主机

scp  /usr/local/elasticsearch-7.14.1/elastic-* 192.168.1.16:/usr/local/elasticsearch-7.14.1/

将证书文件加入到elasticsearch集群

对elasticsearch安装目录进行授权192.168.1.15和192.168.1.16主机操作

useradd es
chown -R es:es /usr/local/elasticsearch-7.14.1

证书加入集群，192.168.1.15和192.168.1.16主机操作

./elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password

./elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

证书文件加入完成后，重新对elasticsearch安装目录授权es用户

chown -R es:es /usr/local/elasticsearch-7.14.1

elasticsearch证书文件移动到对应目录，192.168.1.15和192.168.1.16操作

如果不断移动到安装目录的config目录下，在启动elasticsearch服务实会报错，会提示证书文件应该放在elasticsearch安装目录的config目录下

mv /usr/local/elasticsearch-7.14.1/elastic-* /usr/local/elasticsearch-7.14.1/config/

elasticsearch添加xpack认证，192.168.1.15和192.168.1.16主机操作，在配置文件末尾添加如下几行

vi /usr/local/elasticsearch-7.14.1/config/elasticsearch.yml
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: none
xpack.security.transport.ssl.keystore.path: /usr/local/elasticsearch-7.14.1/config/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /usr/local/elasticsearch-7.14.1/config/elastic-certificates.p12
ingest.geoip.downloader.enabled: false

创建elasticsearch配置文件指定的索引日志目录和日志文件目录，192.168.1.15和192.168.1.16主机操作如下

mkdir -p /elasticsearch/data /elasticsearch/logs
chown -R es:es /elasticsearch

启动elasticsearch服务，先启动主节点192.168.1.15再启动192.168.1.16从节点

su es
cd /usr/local/elasticsearch-7.14.1/bin/
 ./elasticsearch -d

如果启动报如下错误

编辑如下配置文件，文件末尾添加如下几行

vi /etc/security/limits.conf
* soft nofile 65536
* hard nofile 131072
* soft nproc 2048
* hard nproc 4096

执行如下一行操作命令

sudo sysctl -w vm.max_map_count=262144

修改上面的配置之后，启动elasticsearch服务（先启动192.168.1.15，再启动192.168.1.16）

./elasticsearch -d

授权连接访问elasticsearch服务账号以及密码，在主节点192.168.1.15主机操作

./elasticsearch-setup-passwords interactive -u 'http://192.168.1.15:9200'

验证账号及其密码，验证成功后此时elasticsearch集群部署完成

curl --user elastic:111qqq 192.168.1.15:9200/_cluster/health?pretty

部署logstash服务

解压logstash软件包到对应目录，192.168.1.17主机操作

 tar -xf logstash-7.14.1-linux-x86_64.tar.gz -C /usr/local/logstash-7.14.1/

被分析的java日志，这里截图一部分作为演示

2023-02-16 16:01:50.568  INFO 12845 --- [qtp1338958728-7681] r.h.app.handler.file.LocalFileHandler    : Uploading file: [1676534461460.jpg] to directory: [upload/2023/02/1676534461460.jpg]
2023-02-16 16:01:50.576  WARN 12845 --- [qtp1338958728-7681] r.h.app.handler.file.LocalFileHandler    : Failed to fetch image meta data

javax.imageio.IIOException: Not a JPEG file: starts with 0x89 0x50
        at java.desktop/com.sun.imageio.plugins.jpeg.JPEGImageReader.readImageHeader(Native Method) ~[na:na]
        at java.desktop/com.sun.imageio.plugins.jpeg.JPEGImageReader.readNativeHeader(JPEGImageReader.java:731) ~[na:na]
        at java.desktop/com.sun.imageio.plugins.jpeg.JPEGImageReader.checkTablesOnly(JPEGImageReader.java:347) ~[na:na]
        at java.desktop/com.sun.imageio.plugins.jpeg.JPEGImageReader.gotoImage(JPEGImageReader.java:496) ~[na:na]
        at java.desktop/com.sun.imageio.plugins.jpeg.JPEGImageReader.readHeader(JPEGImageReader.java:724) ~[na:na]
        at java.desktop/com.sun.imageio.plugins.jpeg.JPEGImageReader.getWidth(JPEGImageReader.java:838) ~[na:na]
        at run.halo.app.handler.file.FileHandler.handleImageMetadata(FileHandler.java:88) ~[classes!/:1.5.3]
        at run.halo.app.handler.file.LocalFileHandler.upload(LocalFileHandler.java:125) ~[classes!/:1.5.3]
        at run.halo.app.handler.file.FileHandlers.upload(FileHandlers.java:53) ~[classes!/:1.5.3]
        at run.halo.app.service.impl.AttachmentServiceImpl.upload(AttachmentServiceImpl.java:116) ~[classes!/:1.5.3]
        at run.halo.app.service.impl.AttachmentServiceImpl$$FastClassBySpringCGLIB$$2d9c80b9.invoke(<generated>) ~[classes!/:1.5.3]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:689) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at run.halo.app.service.impl.AttachmentServiceImpl$$EnhancerBySpringCGLIB$$a01d20fb.upload(<generated>) ~[classes!/:1.5.3]
        at run.halo.app.controller.admin.api.AttachmentController.uploadAttachment(AttachmentController.java:84) ~[classes!/:1.5.3]
        at run.halo.app.controller.admin.api.AttachmentController$$FastClassBySpringCGLIB$$8f3ff968.invoke(<generated>) ~[classes!/:1.5.3]
        at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218) ~[spring-core-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.invokeJoinpoint(CglibAopProxy.java:783) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:89) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at run.halo.app.core.ControllerLogAop.controller(ControllerLogAop.java:48) ~[classes!/:1.5.3]
        at jdk.internal.reflect.GeneratedMethodAccessor360.invoke(Unknown Source) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:634) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:624) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:72) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:753) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:698) ~[spring-aop-5.3.18.jar!/:5.3.18]
        at run.halo.app.controller.admin.api.AttachmentController$$EnhancerBySpringCGLIB$$7caad889.uploadAttachment(<generated>) ~[classes!/:1.5.3]
        at jdk.internal.reflect.GeneratedMethodAccessor525.invoke(Unknown Source) ~[na:na]
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
        at java.base/java.lang.reflect.Method.invoke(Method.java:566) ~[na:na]
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117) ~[spring-webmvc-5.3.18.jar!/:5.3.18]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895) ~[spring-webmvc-5.3.18.jar!/:5.3.18]
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808) ~[spring-webmvc-5.3.18.jar!/:5.3.18]
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.3.18.jar!/:5.3.18]
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067) ~[spring-webmvc-5.3.18.jar!/:5.3.18]
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963) ~[spring-webmvc-5.3.18.jar!/:5.3.18]
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006) ~[spring-webmvc-5.3.18.jar!/:5.3.18]
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:909) ~[spring-webmvc-5.3.18.jar!/:5.3.18]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:517) ~[jakarta.servlet-api-4.0.4.jar!/:4.0.4]
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883) ~[spring-webmvc-5.3.18.jar!/:5.3.18]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:584) ~[jakarta.servlet-api-4.0.4.jar!/:4.0.4]
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1631) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.websocket.server.WebSocketUpgradeFilter.doFilter(WebSocketUpgradeFilter.java:230) ~[websocket-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at run.halo.app.security.filter.AdminAuthenticationFilter.doAuthenticate(AdminAuthenticationFilter.java:120) ~[classes!/:1.5.3]
        at run.halo.app.security.filter.AbstractAuthenticationFilter.doFilterInternal(AbstractAuthenticationFilter.java:229) ~[classes!/:1.5.3]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:102) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at run.halo.app.filter.CorsFilter.doFilter(CorsFilter.java:53) ~[classes!/:1.5.3]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96) ~[spring-boot-actuator-2.5.12.jar!/:2.5.12]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at run.halo.app.filter.LogFilter.doFilterInternal(LogFilter.java:40) ~[classes!/:1.5.3]
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:117) ~[spring-web-5.3.18.jar!/:5.3.18]
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600) ~[jetty-security-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) ~[jetty-servlet-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) ~[jetty-server-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) ~[jetty-io-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) ~[jetty-io-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) ~[jetty-io-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) ~[jetty-util-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) ~[jetty-util-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) ~[jetty-util-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) ~[jetty-util-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) ~[jetty-util-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) ~[jetty-util-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) ~[jetty-util-9.4.45.v20220203.jar!/:9.4.45.v20220203]
        at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

编辑logstash.conf配置文件

第一种分析方法使用自定义正则分析日志

vi /usr/local/logstash-7.14.1/config/logstash.conf
filter {
     grok {
         match => {
             "message" => "(?<list1>^(\d{4}).*?\d{3})\s+(?<list2>[A-Z]{4,5})\s+(?<list3>\d+)\s+(?<list4>\-\-\-)\s+(?<list5>\[\w+\-\d+\])\s+(?<list6>[\.|\w+]+)\s+(?<list7>\:)\s+(?<list8>\w+[\s+|\:].*)"    #自定义正则写法“(?<自定义名称>正则表达式)正则表达式”，其中“<>”为显示标头格式。
         }
     }
}
input {  #输入日志
     file {
         path => "/logstash/java-test-log/spring.log.2023-02-16.0"  #日志存放目录
         codec => multiline{   #启用cedec插件
             pattern => "^%{TIMESTAMP_ISO8601}"  #输入日志以正则宏时间戳作为分割每一行
             negate => true   #ture若输入的行匹配失败进行合并；flase匹配失败，则不合并
             what => previous  #previous表示合并到前面一行，next表示合并到后面一行
         }
         start_position => "beginning"    #每次启动logstash从开头读取日志此处写end就是从上次结束的标识处开始 ，注意如果不写下面的 sincedb_path => /dev/null每次重启还是从上次结束的标识处开始，若需要每次从文件的行首读取需要加入sincedb_path => /dev/null参数
         sincedb_path => "/dev/null"    
         stat_interval => "3"
     }
}
output {
    stdout { codec => "rubydebug" }  #开启调试模式，分析内容不输出到logstash
}

第二种分析方法使用官方的正则宏分析java日志示例

vi /usr/local/logstash-7.14.1/config/logstash.conf
filter {        #分析日志
     grok {
         match => { "message" => "\s*%{TIMESTAMP_ISO8601:time}" }   #调用正则宏
     }
     date {
         match => ["time", "yyyy-MM-dd HH:mm:ss.SSS"]       #按照时间戳为分隔符区分每一条日志（不通日志需要注意在此处的时间戳格式）
             target => "@timestamp"
     }
}
input {       #输入日志
     file {
         type => "test-java-log"  #类型可以在output调用同时区分不同的日志环境
         path => "/logstash/java-test-log/spring.log.2023-02-16.0"  #读入日志文件的路径
         codec => multiline{     #把当前行的数据放在前面一行的后面
             pattern => "^%{TIMESTAMP_ISO8601}"  #输入日志以正则宏时间戳作为分割每一行
             negate => true #ture若输入的行匹配失败进行合并；flase匹配失败，则不合并
             what => previous   #previous表示合并到前面一行，next表示合并到后面一行
         }
         ignore_older => 86400   #logstash每隔多久监听一下文件状态
         start_position => "beginning"    #每次启动logstash从开头读取日志此处写end就是从上次结束的标识处开始 ，注意如果不写下面的 sincedb_path => /dev/null每次重启还是从上次结束的标识处开始，若需要每次从文件的行首读取需要加入sincedb_path => /dev/null参数
         sincedb_path => “/dev/null”       
         stat_interval => "3"  
     }
}
output {    #输出 将日志输出到elasticsearch索引库
    if [type] == "test-java-log"  {  #此处可以调用input的type
        elasticsearch {    #elasticsearch信息
            hosts => ["http://192.168.1.15:9200"]   #elasticsearch主机地址和端口
            index => "test-java-%{+YYYY.MM.dd}"  #生成的索引名称格式
            user => "elastic"     #访问elasticsearch的账号
            password => "111qqq"  #访问elasticsearch的elastic的密码
        }
    }
}

启动logstash并进行测试

./logstash -f /usr/local/logstash-7.14.1/config/logstash.conf

由于上面使用官方正则宏分析没有开启调试模式，不在这里截图展示，可自己开启调试模式进行测试，这里作为示例看一下使用自定义正则分析出来的日志结果

启动成功后查看elasticsearch索引，若失败检查处理分析日志文件logstash.conf是否正确，如果logstash分析没有问题，则执行启动放入后台运行，操作如下

mkdir /logstash/logstash-log/   #用于存放logstash日志文件
nohup /usr/local/logstash-7.14.1/bin/logstash -f /usr/local/logstash-7.14.1/config/logstash.conf  > /logstash/logstash-log/logstash.log 2>&1 &

查看elasticsearch是否生成索引

curl --user elastic:111qqq 192.168.1.15:9200/_cat/indices?v

出现此条索引表示日志分析成功

部署kibana

解压kibana到指定目录

tar -xf kibana-7.14.1-linux-x86_64.tar.gz -C /usr/local/

修改kibana配置文件

vi /usr/local/kibana-7.14.1-linux-x86_64/config/kibana.yml
server.port: 5601     #kibana启动和访问端口
server.host: "0.0.0.0"  #允许所有地址监听访问
elasticsearch.hosts: ["http://192.168.1.15:9200"]   #elasticsearch地址获取索引
elasticsearch.username: "kibana_system"     #连接elasticsearch的账号不建议写elastic，elastic权限过大
elasticsearch.password: "111qqq"   #账号kibana_system密码
pid.file: /run/kibana/kibana.pid   #pid文件路径

mkdir /run/kibana  #上面配置文件指定的pid存放目录不存在需要进行创建

启动kibana服务

/usr/local/kibana-7.14.1-linux-x86_64/bin/kibana --allow-root  > /var/log/kibana7.log 2>&1 &

浏览器访问kibana服务

登录kibana

账号:elastic
密码:111qqq

kibana添加数据

点击 “Explore on my own”

点击左侧菜单栏

点击"Management"

点击“Stack Managemeng”

点击”Index Patterns“

点击”Create index pattern“

可以看到指针指向地方kibana获取的elasticsearch的索引源

在图中指针指向地方为正则匹配索引，对不同的日志进行区分

点击Next Step

点击下拉框，选择”@timestamp“

点击”create index pattern“

看到下图则创建成功

查看创建的索引信息，点击左侧菜单栏，点击”Discover“

点击”metrices-“，选择创建的”test-java“

查看日志区间范围按照下图操作即可

常见错误类型分析

elasticsearch运行主机磁盘使用率达到90%会出现报错

解决方法：释放磁盘，清理日志索引存储

eleaticserch由于jvm内存不足无法启动问题

解决方法修改如下配置文件修改jav参数，注意修改此参数会分掉内存，实际情况按照主机内存合理进行分配jvm参数，配置文件默认为4g

vi /usr/local/elasticsearch-7.14.1/config/jvm.options
 -Xms8g     
 -Xmx8g

添加elasticsearch索引分片不足报错

解决方法增加索引分片数，执行如下操作

 curl --user elastic:111qqq -XPUT -H "Content-Type:application/json" -d '{"transient":{"cluster":{"max_shards_per_node":50000}}}' 'http://192.168.1.15:9200/_cluster/settings'

其中50000为索引分片数，实际需求根据实际情况合理分配

内存限制导致elasticsearch报错

解决方法执行如下操作

curl --user elastic:111qqq -XPUT "192.168.1.15:9200/_cluster/settings" -H 'Content-Type: application/json' -d '{ "persistent" : { "indices.breaker.fielddata.limit" : "40%" } }'

其中40%为内存限制

集群频繁进行主节点选举

查看日志，发现节点每过固定时间后会丢失对其他节点的连接，集群重新进行主节点选举，选举期间客户端连接超时
原因：elasticsearch节点之间是tcp长连接，当集群长时间无操作时，tcp连接可能被防火墙中断，但elasticsearch节点仍然尝试使用该tcp进行通信，导致节点间连接失败直到使用新的tcp连接时，集群才逐渐恢复使用
解决方法执行如下操作
1、修改tcp的keepalive参数vi /etc/sysctl.conf （如果部署了keeplived方案可参考）
net.ipv4.tcp_keepalive_time = 600 #keepalive间隔时间应小于防火墙的中断时间
net.ipv4.tcp_keepalive_intvl = 60
net.ipv4.tcp_keepalive_probes = 6
保存后执行sysctl -p生效
2、重启es节点