分步部署-持续交付部署--部署gitlab service总是上线不了,报错

  • Choerodon平台版本: 0.18

  • 遇到问题的执行步骤:
    分步部署持续交付中gitlab service,部署正常,查看pod也是running,但使用验证部署等好久返回500错误

  • 文档地址:
    http://choerodon.io/zh/docs/installation-configuration/steps/install/parts/choerodon-devops/

  • 环境信息(如:节点信息):
    4个节点8核16G

  • 报错日志:
    查看了 gitlab-service 的日志:报如下错误
    2019-07-17 14:51:14.028 ERROR 8 — [ XNIO-2 task-12] io.undertow.request : UT005023: Exception handling request to /actuator/health

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is io.choerodon.core.exception.CommonException: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection timed out (Connection timed out)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209) ~[spring-security-web-5.0.9.RELEASE.jar!/:5.0.9.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) ~[spring-security-web-5.0.9.RELEASE.jar!/:5.0.9.RELEASE]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:65) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336) [undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_202]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_202]
at java.lang.Thread.run(Thread.java:813) [na:1.8.0_202]
Caused by: io.choerodon.core.exception.CommonException: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection timed out (Connection timed out)
at io.choerodon.gitlab.domain.config.GitlabHealthy.health(GitlabHealthy.java:33) ~[classes!/:0.18.0]
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:50) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_202]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_202]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_202]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_202]
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:223) ~[spring-core-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:61) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$$Lambda$558.00000000D1156FA0.invoke(Unknown Source) ~[na:na]
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:274) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:330) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_202]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_202]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_202]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_202]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:891) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.boot.actuate.autoconfigure.web.servlet.CompositeHandlerAdapter.handle(CompositeHandlerAdapter.java:58) ~[spring-boot-actuator-autoconfigure-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
… 38 common frames omitted
Caused by: org.gitlab4j.api.GitLabApiException: java.net.ConnectException: Connection timed out (Connection timed out)
at org.gitlab4j.api.AbstractApi.handle(AbstractApi.java:352) ~[choerodon-gitlab4j-api-0.11.0.RELEASE.jar!/:0.11.0.RELEASE]
at org.gitlab4j.api.AbstractApi.get(AbstractApi.java:90) ~[choerodon-gitlab4j-api-0.11.0.RELEASE.jar!/:0.11.0.RELEASE]
at org.gitlab4j.api.UserApi.getCurrentUser(UserApi.java:452) ~[choerodon-gitlab4j-api-0.11.0.RELEASE.jar!/:0.11.0.RELEASE]
at io.choerodon.gitlab.domain.config.GitlabHealthy.health(GitlabHealthy.java:26) ~[classes!/:0.18.0]
… 64 common frames omitted
Caused by: javax.ws.rs.ProcessingException: java.net.ConnectException: Connection timed out (Connection timed out)
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:284) ~[jersey-client-2.26.jar!/:na]
at org.glassfish.jersey.client.ClientRuntime.invoke(ClientRuntime.java:278) ~[jersey-client-2.26.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation.lambda$invoke$0(JerseyInvocation.java:753) ~[jersey-client-2.26.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation$$Lambda$686.00000000D43D2B70.call(Unknown Source) ~[na:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:316) ~[jersey-common-2.26.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:298) ~[jersey-common-2.26.jar!/:na]
at org.glassfish.jersey.internal.Errors.process(Errors.java:229) ~[jersey-common-2.26.jar!/:na]
at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:414) ~[jersey-common-2.26.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation.invoke(JerseyInvocation.java:752) ~[jersey-client-2.26.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.method(JerseyInvocation.java:419) ~[jersey-client-2.26.jar!/:na]
at org.glassfish.jersey.client.JerseyInvocation$Builder.get(JerseyInvocation.java:319) ~[jersey-client-2.26.jar!/:na]
at org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:322) ~[choerodon-gitlab4j-api-0.11.0.RELEASE.jar!/:0.11.0.RELEASE]
at org.gitlab4j.api.GitLabApiClient.get(GitLabApiClient.java:310) ~[choerodon-gitlab4j-api-0.11.0.RELEASE.jar!/:0.11.0.RELEASE]
at org.gitlab4j.api.AbstractApi.get(AbstractApi.java:88) ~[choerodon-gitlab4j-api-0.11.0.RELEASE.jar!/:0.11.0.RELEASE]
… 66 common frames omitted
Caused by: java.net.ConnectException: Connection timed out (Connection timed out)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[na:1.8.0_202]
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[na:1.8.0_202]
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[na:1.8.0_202]
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[na:1.8.0_202]
at java.net.Socket.connect(Socket.java:589) ~[na:1.8.0_202]
at sun.net.NetworkClient.doConnect(NetworkClient.java:175) ~[na:1.8.0_202]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:463) ~[na:1.8.0_202]
at sun.net.www.http.HttpClient.openServer(HttpClient.java:558) ~[na:1.8.0_202]
at sun.net.www.http.HttpClient.(HttpClient.java:242) ~[na:1.8.0_202]
at sun.net.www.http.HttpClient.New(HttpClient.java:339) ~[na:1.8.0_202]
at sun.net.www.http.HttpClient.New(HttpClient.java:357) ~[na:1.8.0_202]
at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1220) ~[na:1.8.0_202]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1156) ~[na:1.8.0_202]
at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1050) ~[na:1.8.0_202]
at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:984) ~[na:1.8.0_202]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1564) ~[na:1.8.0_202]
at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) ~[na:1.8.0_202]
at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) ~[na:1.8.0_202]
at org.glassfish.jersey.client.internal.HttpUrlConnector._apply(HttpUrlConnector.java:390) ~[jersey-client-2.26.jar!/:na]
at org.glassfish.jersey.client.internal.HttpUrlConnector.apply(HttpUrlConnector.java:282) ~[jersey-client-2.26.jar!/:na]
… 79 common frames omitted

  • 原因分析:

    一直报连接超时 也不知道哪里错了,GITLAB_PRIVATETOKEN也是按照社区中的流程产生的,名称取的是root。

  • 疑问:

    望指导下 谢谢了

请大神帮忙啊 进行不下去了

进入到gitlab-service的容器中,curl一下gitlab的域名地址试下看看

很是奇怪,我进入了gitlab-service的容器中 curl百度可以访问 gitlab域名就不能访问(使用的是免费域名nip.io),但是我浏览器里都可以直接访问
望指点

curl gitlab域名的时候是返回的什么状态码或者什么错误

找到原因了:是我们内网访问域名指向的内网地址的时候访问不了,防火墙不支持内环访问,我部署gitlab-service中的–set env.open.GITLAB_URL=“http://10.233.64.58” 地址修改成了内部容器地址,部署成功了,查看状态up 不知道以后使用过程会不会有问题。

这样处理的话,以后gitlab容器重启了,那么容器的ip就变了,到时候又访问不了了。建议防火墙过滤掉
这个dns解析域名的ip地址

嗯 谢谢啊,我用的是kubectl edit svc -n c7n-system gitlab中的配置地址,重启后不会变的。

哦 那行,我以为是gitlab pod的ip地址

我也是类似的问题,到这配置都是对的呢,但问题还存在。日志中反复显示如下的报错:
2020-12-02 13:16:41.518 INFO 7 — [ XNIO-2 task-6] i.c.g.infra.common.config.GitlabHealthy : 健康检查探测
2020-12-02 13:16:41.582 ERROR 7 — [ XNIO-2 task-6] io.undertow.request : UT005023: Exception handling request to /actuator/health

curl: (6) Could not resolve host:

curl: (6) Could not resolve host:
显示这个,解析不了

你好,这是网络的问题,请开新贴子讨论这个问题