手动部署时创建网络失败 create or update service failed

hudcan · 2020 年4 月 9 日 08:35

Choerodon平台版本：0.21
运行环境：自主搭建
问题描述：

手动创建网络失败

devops-service 报错

gitlab-service 报错

hudcan · 2020 年4 月 9 日 11:57

重启gitlab后正常。。。。感觉一点都不稳定啊

ZmfCn · 2020 年4 月 9 日 13:57

可能是你的gitlab-service访问gitlab不稳定吧？

hudcan · 2020 年4 月 10 日 01:40

重启后还是会出现网络创建失败的情况。。

我觉得应该不是 gitlab-service访问gitlab不稳定

因为每次pod的yaml 在gitlab中都能创建成功，但是网络的 yaml 就不一定了。

而且手动在资源视图中创建网络是没有问题的，感觉还是代码哪里有问题。

ZmfCn · 2020 年4 月 10 日 01:44

你的意思是在资源视图创建网络可以，但是实例视图创建不行？

hudcan · 2020 年4 月 10 日 01:53

我是手动部署的时候不是要创建网络吗?

创建这个网络失败，但是这个pod能成功

ZmfCn · 2020 年4 月 10 日 02:36

你去项目层的事务管理看看失败的事务实例

hudcan · 2020 年4 月 10 日 02:49

全部是成功的？
事务：

任务：

管理中心的也是全部成功
事务：

任务：

ZmfCn · 2020 年4 月 10 日 02:52

我这边没有复现诶

ZmfCn · 2020 年4 月 10 日 02:52

这个是同样输入百分百复现吗？

hudcan · 2020 年4 月 10 日 02:55

不是的，昨天下午就一直失败，重启gitlab后正常了一会，但是后面又不行了。今天好像又好了，不是必现

ZmfCn · 2020 年4 月 10 日 02:59

好的，我记录下这个问题，之后排查下。
如果还出现这个问题，请再次将gitlab-service的异常日志贴一下

hudcan · 2020 年4 月 10 日 03:14

现在好像单单部署pod有不成功了。日志：

2020-04-10 11:01:17.433  INFO 7 --- [nio-8060-exec-2] i.c.d.a.s.i.AgentMsgHandlerServiceImpl   : env 9 receive git ops msg :
{"id":0,"resourceIDs":["c7nhelmrelease/aaa-7b30e"],"type":"sync","startedAt":"2020-04-10T03:01:17.192742634Z","endedAt":"2020-04-10T03:01:17.192742634Z","metadata":{"commit":"6559f5018f0a0722ab0887d572fb02b78a8d90b4","filesCommit":[{"file":"release-aaa-7b30e.yaml","commit":"6559f5018f0a0722ab0887d572fb02b78a8d90b4"}],"resourceCommits":[{"resourceId":"c7nhelmrelease/aaa-7b30e","file":"release-aaa-7b30e.yaml","commit":"6559f5018f0a0722ab0887d572fb02b78a8d90b4"}]}}
2020-04-10 11:01:45.089  INFO 7 --- [nio-8061-exec-9] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at : http://register-server.c7n-system:8000/
2020-04-10 11:01:45.104  INFO 7 --- [nio-8061-exec-9] c.c.c.ConfigServicePropertySourceLocator : Located environment: name=devops-service, profiles=[default], label=, version=0.21.1, state=
2020-04-10 11:02:39.582  INFO 7 --- [nio-8060-exec-4] i.c.d.a.s.impl.DevopsGitServiceImpl      : ````````````````````````````` 814768962dfcd1fea609782ef15b3e6ccd172c73
2020-04-10 11:02:39.608  INFO 7 --- [nio-8060-exec-4] i.c.d.a.s.impl.DevopsGitServiceImpl      : update devopsCommit successfully: 814768962dfcd1fea609782ef15b3e6ccd172c73
2020-04-10 11:02:40.701  INFO 7 --- [pool-7-thread-4] i.c.d.a.s.impl.DevopsGitServiceImpl      : The gitOps Repository ssh url: ssh://git@gitlab.192.168.200.173.nip.io:32222/operation-online-his-gitops/env-test.git
2020-04-10 11:02:42.030  INFO 7 --- [pool-7-thread-4] i.c.d.a.s.impl.DevopsGitServiceImpl      : 更新gitops库成功
2020-04-10 11:02:42.324  INFO 7 --- [pool-7-thread-4] i.c.d.a.s.impl.DevopsGitServiceImpl      : 序列化k8s对象成功
2020-04-10 11:02:42.336  INFO 7 --- [pool-7-thread-4] i.c.d.a.s.impl.DevopsGitServiceImpl      : k8s对象转换平台对象成功
2020-04-10 11:02:45.124  INFO 7 --- [pool-7-thread-4] i.c.d.a.s.impl.DevopsGitServiceImpl      : 发送gitops同步成功指令成功
2020-04-10 11:02:45.861  INFO 7 --- [nio-8060-exec-6] i.c.d.a.w.g.AgentGitOpsMessageHandler    : 找不到指令啊 git_ops_sync
2020-04-10 11:02:46.103  INFO 7 --- [nio-8060-exec-1] i.c.d.a.s.i.AgentMsgHandlerServiceImpl   : env 9 receive git ops msg :
{"id":0,"resourceIDs":[],"type":"sync","startedAt":"2020-04-10T03:02:45.870615321Z","endedAt":"2020-04-10T03:02:45.870615321Z","metadata":{"commit":"814768962dfcd1fea609782ef15b3e6ccd172c73"}}
2020-04-10 11:02:47.977  INFO 7 --- [nio-8060-exec-6] i.c.r.h.ControllerExceptionHandler       : exception info io.choerodon.core.exception.CommonException: error.resource.query
	at io.choerodon.devops.api.controller.v1.AppServiceInstanceController.lambda$listResourcesInHelmRelease$17(AppServiceInstanceController.java:503)
	at io.choerodon.devops.api.controller.v1.AppServiceInstanceController$$Lambda$1345/0000000020060D70.get(Unknown Source)
	at java.util.Optional.orElseThrow(Unknown Source)
	at io.choerodon.devops.api.controller.v1.AppServiceInstanceController.listResourcesInHelmRelease(AppServiceInstanceController.java:503)
	at sun.reflect.GeneratedMethodAccessor2205.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
	at java.lang.reflect.Method.invoke(Unknown Source)
	at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
	at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
	at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:892)
	at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:797)
	at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
	at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1039)
	at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
	at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
	at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:897)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
	at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.web.trace.servlet.HttpTraceFilter.doFilterInternal(HttpTraceFilter.java:88)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:320)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
	at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.oauth2.provider.authentication.OAuth2AuthenticationProcessingFilter.doFilter(OAuth2AuthenticationProcessingFilter.java:176)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:74)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
	at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:215)
	at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
	at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
	at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:99)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:92)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:93)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.filterAndRecordMetrics(WebMvcMetricsFilter.java:114)
	at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:104)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:200)
	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:109)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Unknown Source)

2020-04-10 11:03:10.118  INFO 7 --- [trap-executor-0] c.n.d.s.r.aws.ConfigClusterResolver      : Resolving eureka endpoints via configuration
2020-04-10 11:04:17.657  INFO 7 --- [pool-7-thread-5] io.choerodon.devops.infra.util.FileUtil  : 文件写入成功
2020-04-10 11:04:20.801  INFO 7 --- [nio-8060-exec-1] i.c.d.a.s.impl.DevopsGitServiceImpl      : ````````````````````````````` 6f299653d7b5eb9f193b8ad69620d784eda6cf23
2020-04-10 11:04:20.829  INFO 7 --- [nio-8060-exec-1] i.c.d.a.s.impl.DevopsGitServiceImpl      : update devopsCommit successfully: 6f299653d7b5eb9f193b8ad69620d784eda6cf23
2020-04-10 11:04:21.912  INFO 7 --- [pool-7-thread-6] i.c.d.a.s.impl.DevopsGitServiceImpl      : The gitOps Repository ssh url: ssh://git@gitlab.192.168.200.173.nip.io:32222/operation-online-his-gitops/env-test.git
2020-04-10 11:04:23.237  INFO 7 --- [pool-7-thread-6] i.c.d.a.s.impl.DevopsGitServiceImpl      : 更新gitops库成功
2020-04-10 11:04:23.561  INFO 7 --- [pool-7-thread-6] i.c.d.a.s.impl.DevopsGitServiceImpl      : 序列化k8s对象成功
2020-04-10 11:04:23.575  INFO 7 --- [pool-7-thread-6] i.c.d.a.s.impl.DevopsGitServiceImpl      : k8s对象转换平台对象成功
2020-04-10 11:04:26.364  INFO 7 --- [pool-7-thread-6] i.c.d.a.s.impl.DevopsGitServiceImpl      : 发送gitops同步成功指令成功
2020-04-10 11:04:27.136  INFO 7 --- [nio-8060-exec-4] i.c.d.a.w.g.AgentGitOpsMessageHandler    : 找不到指令啊 git_ops_sync
2020-04-10 11:04:27.423  INFO 7 --- [nio-8060-exec-9] i.c.d.a.s.i.AgentMsgHandlerServiceImpl   : env 9 receive git ops msg :
{"id":0,"resourceIDs":["c7nhelmrelease/aaa-f2d6c"],"type":"sync","startedAt":"2020-04-10T03:04:27.145219764Z","endedAt":"2020-04-10T03:04:27.145219764Z","metadata":{"commit":"6f299653d7b5eb9f193b8ad69620d784eda6cf23","filesCommit":[{"file":"release-aaa-f2d6c.yaml","commit":"6f299653d7b5eb9f193b8ad69620d784eda6cf23"}],"resourceCommits":[{"resourceId":"c7nhelmrelease/aaa-f2d6c","file":"release-aaa-f2d6c.yaml","commit":"6f299653d7b5eb9f193b8ad69620d784eda6cf23"}]}}
2020-04-10 11:05:34.986  INFO 7 --- [io-8060-exec-10] i.c.d.a.s.i.AgentMsgHandlerServiceImpl   : env 6 receive git ops msg :
{"id":0,"resourceIDs":[],"type":"sync","startedAt":"2020-04-10T03:05:34.951298047Z","endedAt":"2020-04-10T03:05:34.951298047Z","metadata":{"commit":"6f9d32c0a1d022c6b060deb2ee8922511972c927"}}
2020-04-10 11:06:45.100  INFO 7 --- [nio-8061-exec-9] c.c.c.ConfigServicePropertySourceLocator : Fetching config from server at : http://register-server.c7n-system:8000/
2020-04-10 11:06:45.117  INFO 7 --- [nio-8061-exec-9] c.c.c.ConfigServicePropertySourceLocator : Located environment: name=devops-service, profiles=[default], label=, version=0.21.1, state=

gitlab-service 无报错

ZmfCn · 2020 年4 月 10 日 03:17

你贴的这个异常，和创建实例失败没有关系。你看看项目层失败的事务实例，只要截图事务就可以了

hudcan · 2020 年4 月 10 日 03:29

好像没得失败

ZmfCn · 2020 年4 月 10 日 05:38

看看环境配置库中有创建失败的实例对应的文件吗？