部署 gitlab service，日志有报错。

Choerodon Installation management
#1

  • Choerodon平台版本: 0.25.0

  • 遇到问题的执行步骤:
    helm upgrade --install安装后，服务异常

  • 文档地址:
    部署 gitlab service章节
    https://choerodon.io/zh/docs/installation-configuration/steps/install/parts/choerodon-devops/

  • 环境信息(如:节点信息):
    master节点，centos7.9 4核16G内存150G磁盘

  • 报错日志:
    Events:
    Type Reason Age From Message

    Warning Unhealthy 7m41s (x1811 over 11h) kubelet Readiness probe failed:
    Warning BackOff 2m46s (x1188 over 11h) kubelet Back-off restarting failed container

  • 原因分析:

    提出您分析问题的过程,以便我们能更准确的找到问题所在
    按照官方文件到GITLAB生成token后，写入gitlab-service.yaml文件GITLAB_PRIVATETOKEN字段，然后helm upgrade --install 。查看验证命令发现是500

  • 疑问:

    提出您对于遇到和解决该问题时的疑问

#2

请使用 kubectl logs 命令获取 gitlab-service pod 的日志

#3

使用kubectl describe pod gitlab-service-6b754f765f-4glg4 -n c7n-system 命令查看的报错信息

image
image751×211 8.16 KB

#4

用 kubectl describe 得到的信息看不出问题，还请用 kubectl logs 获取日志

#5

image
image961×84 4.24 KB

pod没有起来，所以只能通过kubectl describe pod gitlab-service-6b754f765f-4glg4 -n c7n-system命令查看。

#6

pod没起来也可以看日志的，请试一试

#7

2021-05-10 16:12:16.678 INFO 7 — [ XNIO-3 task-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet ‘dispatcherServlet’: initialization completed in 85 ms
2021-05-10 16:12:26.164 INFO 7 — [ XNIO-2 task-1] io.undertow.servlet : Initializing Spring FrameworkServlet ‘dispatcherServlet’
2021-05-10 16:12:26.165 INFO 7 — [ XNIO-2 task-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet ‘dispatcherServlet’: initialization started
2021-05-10 16:12:26.169 INFO 7 — [ XNIO-2 task-1] o.s.web.servlet.DispatcherServlet : FrameworkServlet ‘dispatcherServlet’: initialization completed in 4 ms
2021-05-10 16:12:26.204 INFO 7 — [ XNIO-2 task-1] i.c.g.infra.common.config.GitlabHealthy : 健康检查探测
2021-05-10 16:12:28.130 ERROR 7 — [ XNIO-2 task-1] io.undertow.request : UT005023: Exception handling request to /actuator/health

org.springframework.web.util.NestedServletException: Request processing failed; nested exception is io.choerodon.core.exception.CommonException: org.gitlab4j.api.GitLabApiException: Service Temporarily Unavailable
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:982) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:866) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:851) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar!/:3.1.0]
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209) ~[spring-security-web-5.0.9.RELEASE.jar!/:5.0.9.RELEASE]
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178) ~[spring-security-web-5.0.9.RELEASE.jar!/:5.0.9.RELEASE]
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:65) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:132) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77) ~[undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43) ~[undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:292) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:81) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:138) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:135) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:272) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:81) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:104) [undertow-servlet-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:336) [undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:830) [undertow-core-1.4.26.Final.jar!/:1.4.26.Final]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_275]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_275]
at java.lang.Thread.run(Thread.java:748) [na:1.8.0_275]
Caused by: io.choerodon.core.exception.CommonException: org.gitlab4j.api.GitLabApiException: Service Temporarily Unavailable
at io.choerodon.gitlab.infra.common.config.GitlabHealthy.health(GitlabHealthy.java:40) ~[classes!/:0.25.0]
at org.springframework.boot.actuate.health.CompositeHealthIndicator.health(CompositeHealthIndicator.java:68) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:50) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_275]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_275]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_275]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_275]
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:223) ~[spring-core-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:76) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:61) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:274) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:330) ~[spring-boot-actuator-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.8.0_275]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[na:1.8.0_275]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.8.0_275]
at java.lang.reflect.Method.invoke(Method.java:498) ~[na:1.8.0_275]
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:209) ~[spring-web-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136) ~[spring-web-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:891) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.boot.actuate.autoconfigure.web.servlet.CompositeHandlerAdapter.handle(CompositeHandlerAdapter.java:58) ~[spring-boot-actuator-autoconfigure-2.0.6.RELEASE.jar!/:2.0.6.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:991) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:925) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:974) ~[spring-webmvc-5.0.10.RELEASE.jar!/:5.0.10.RELEASE]
… 38 common frames omitted
Caused by: org.gitlab4j.api.GitLabApiException: Service Temporarily Unavailable
at org.gitlab4j.api.AbstractApi.validate(AbstractApi.java:386) ~[choerodon-gitlab4j-api-0.19.1.RELEASE.jar!/:0.19.1.RELEASE]
at org.gitlab4j.api.AbstractApi.get(AbstractApi.java:126) ~[choerodon-gitlab4j-api-0.19.1.RELEASE.jar!/:0.19.1.RELEASE]
at org.gitlab4j.api.UserApi.getCurrentUser(UserApi.java:452) ~[choerodon-gitlab4j-api-0.19.1.RELEASE.jar!/:0.19.1.RELEASE]
at io.choerodon.gitlab.infra.common.config.GitlabHealthy.health(GitlabHealthy.java:33) ~[classes!/:0.25.0]
… 63 common frames omitted

#8

你的 gitlab 不正常

#9

gitlab的界面和登陆都没有问题，怎么看正不正常?

#10

@Vista