流水线发布失败-timeout

yuanyblf · 2018 年11 月 30 日 04:45

Choerodon平台版本：0.11.0
运行环境：自主搭建
问题描述：

请尽量详细的描述您遇到的问题，以便我们能更快速的提供解决办法。

通过环境流水线部署应用，一直是失败；
显示 The deploy is time out!

image2840×1064 179 KB

image2826×1172 201 KB
执行的操作：
检查了devops_service 配置库都正常；
agent 日志

I1130 20:22:01.581031 1 agent.go:122] KubeClient init success.
I1130 20:22:01.581099 1 agent.go:124] Starting connect to tiller…
I1130 20:22:01.581381 1 agent.go:126] Tiller connect success
I1130 20:22:01.581397 1 agent.go:283] check k8s role binding…
I1130 20:22:01.636923 1 agent.go:289] k8s role binding succeed.
I1130 20:22:01.637033 1 agent.go:206] kubectl /usr/local/bin/kubectl
I1130 20:22:01.637197 1 client.go:79] Started agent
I1130 20:22:02.223122 1 sync.go:178] kubectl apply -f - , took 586.028713ms, err: , output: customresourcedefinition “c7nhelmreleases.choerodon.io” configured
I1130 20:22:02.223260 1 worker.go:140] get command: cluster:1/init_agent
I1130 20:22:02.223288 1 worker.go:140] get command: inter:inter/upgrade_cluster
I1130 20:22:02.233608 1 controller.go:98] Starting controllers for envs [dev service]
I1130 20:22:02.234185 1 controller_utils.go:1019] Waiting for caches to sync for endpoint controller
I1130 20:22:02.285483 1 upgrade.go:19] cluster agent upgrade: {“token”:“45877a0d-f872-4e65-b475-8608f5275f92”,“platformCode”:" ecfe18be"}
I1130 20:22:02.285576 1 client.go:173] send response key inter:inter, type upgrade
I1130 20:22:02.334369 1 controller_utils.go:1026] Caches are synced for endpoint controller
I1130 20:22:02.334768 1 client.go:173] send response key env:dev, type resource_sync
I1130 20:22:02.334838 1 client.go:173] send response key env:service, type resource_sync
I1130 20:22:08.335636 1 client.go:173] send response key none:none, type namespace_update
E1130 20:22:12.233953 1 daemon.go:94] dev do sync: git repo not ready
I1130 20:22:22.234128 1 client.go:173] send response key cluster:1, type init_agent_succeed
E1130 20:22:22.234127 1 daemon.go:94] service do sync: git repo not ready
W1130 20:22:50.468242 1 secret_controller.go:155] secret ‘dev/default-token-lngv7’ in work queue no longer exists
I1130 20:22:50.468302 1 client.go:173] send response key env:dev.Secret:default-token-lngv7, type resource_delete
W1130 20:22:50.497392 1 secret_controller.go:155] secret ‘dev/default-token-lngv7’ in work queue no longer exists
I1130 20:22:50.497447 1 client.go:173] send response key env:dev.Secret:default-token-lngv7, type resource_delete
I1130 20:23:15.014007 1 worker.go:140] get command: cluster:1.env:service.envId:3/git_ops_sync
E1130 20:23:15.014056 1 error.go:10] git repo not ready
I1130 20:23:15.014101 1 client.go:173] send response key cluster:1.env:service.envId:3, type git_ops_sync_failed
E1130 20:24:12.276218 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone848888145]: context deadline exceeded
E1130 20:24:22.288515 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone226717820]: context deadline exceeded
I1130 20:25:12.223745 1 client.go:173] send response key env:dev, type status_sync_event
I1130 20:25:12.223886 1 client.go:173] send response key env:service, type status_sync_event
I1130 20:25:12.230157 1 worker.go:140] get command: cluster:1.env:dev.envId:1/status_sync
I1130 20:25:12.233114 1 worker.go:140] get command: cluster:1.env:service.envId:3/status_sync
E1130 20:26:22.347780 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone400311211]: context deadline exceeded
E1130 20:26:32.315914 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone885159182]: context deadline exceeded
E1130 20:27:12.234136 1 daemon.go:94] dev do sync: git repo not ready
E1130 20:27:22.234562 1 daemon.go:94] service do sync: git repo not ready
I1130 20:28:12.224000 1 client.go:173] send response key env:dev, type status_sync_event
I1130 20:28:12.224162 1 client.go:173] send response key env:service, type status_sync_event
I1130 20:28:12.230839 1 worker.go:140] get command: cluster:1.env:dev.envId:1/status_sync
I1130 20:28:12.233419 1 worker.go:140] get command: cluster:1.env:service.envId:3/status_sync
I1130 20:28:12.237095 1 helm.go:326] sync response [{“resourceType”:“instance”,“resourceName”:“api-gateway-00a8a”,“id”:5}]
I1130 20:28:12.237143 1 client.go:173] send response key cluster:1.env:service.envId:3, type status_sync
E1130 20:28:32.429843 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone919300373]: context deadline exceeded
E1130 20:28:42.347623 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone777988976]: context deadline exceeded
E1130 20:30:42.512123 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone229796367]: context deadline exceeded
E1130 20:30:52.410310 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone822772258]: context deadline exceeded
I1130 20:31:12.224228 1 client.go:173] send response key env:dev, type status_sync_event
I1130 20:31:12.224360 1 client.go:173] send response key env:service, type status_sync_event
I1130 20:31:12.230729 1 worker.go:140] get command: cluster:1.env:dev.envId:1/status_sync
I1130 20:31:12.233560 1 worker.go:140] get command: cluster:1.env:service.envId:3/status_sync
E1130 20:32:12.234326 1 daemon.go:94] dev do sync: git repo not ready
E1130 20:32:22.234838 1 daemon.go:94] service do sync: git repo not ready
E1130 20:32:52.577667 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone043911961]: context deadline exceeded
E1130 20:33:02.436634 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone191378340]: context deadline exceeded
I1130 20:34:12.224469 1 client.go:173] send response key env:dev, type status_sync_event
I1130 20:34:12.224576 1 client.go:173] send response key env:service, type status_sync_event
I1130 20:34:12.231023 1 worker.go:140] get command: cluster:1.env:dev.envId:1/status_sync
I1130 20:34:12.233868 1 worker.go:140] get command: cluster:1.env:service.envId:3/status_sync
E1130 20:35:02.682456 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone450416307]: context deadline exceeded
E1130 20:35:12.467857 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone934658166]: context deadline exceeded
I1130 20:36:19.085402 1 worker.go:140] get command: cluster:1.env:service.envId:3/git_ops_sync
E1130 20:36:19.085452 1 error.go:10] git repo not ready
I1130 20:36:19.085490 1 client.go:173] send response key cluster:1.env:service.envId:3, type git_ops_sync_failed
I1130 20:37:12.224671 1 client.go:173] send response key env:dev, type status_sync_event
I1130 20:37:12.224791 1 client.go:173] send response key env:service, type status_sync_event
I1130 20:37:12.231358 1 worker.go:140] get command: cluster:1.env:dev.envId:1/status_sync
I1130 20:37:12.234427 1 worker.go:140] get command: cluster:1.env:service.envId:3/status_sync
E1130 20:37:12.234438 1 daemon.go:94] dev do sync: git repo not ready
E1130 20:37:12.760163 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone023503197]: context deadline exceeded
E1130 20:37:22.235119 1 daemon.go:94] service do sync: git repo not ready
E1130 20:37:22.500047 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone024325912]: context deadline exceeded
E1130 20:39:22.801242 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone985409943]: context deadline exceeded
E1130 20:39:32.527544 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone078868490]: context deadline exceeded
I1130 20:40:12.224922 1 client.go:173] send response key env:dev, type status_sync_event
I1130 20:40:12.225028 1 client.go:173] send response key env:service, type status_sync_event
I1130 20:40:12.231543 1 worker.go:140] get command: cluster:1.env:dev.envId:1/status_sync
I1130 20:40:12.234598 1 worker.go:140] get command: cluster:1.env:service.envId:3/status_sync
I1130 20:40:12.244885 1 helm.go:326] sync response [{“resourceType”:“instance”,“resourceName”:“api-gateway-00a8a”,“id”:6}]
I1130 20:40:12.244949 1 client.go:173] send response key cluster:1.env:service.envId:3, type status_sync
E1130 20:41:32.890950 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone451352545]: context deadline exceeded
E1130 20:41:42.562699 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone879052748]: context deadline exceeded
E1130 20:42:12.234657 1 daemon.go:94] dev do sync: git repo not ready
E1130 20:42:22.235311 1 daemon.go:94] service do sync: git repo not ready
I1130 20:43:12.225154 1 client.go:173] send response key env:dev, type status_sync_event
I1130 20:43:12.225325 1 client.go:173] send response key env:service, type status_sync_event
I1130 20:43:12.231585 1 worker.go:140] get command: cluster:1.env:dev.envId:1/status_sync
I1130 20:43:12.234667 1 worker.go:140] get command: cluster:1.env:service.envId:3/status_sync
E1130 20:43:42.983786 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone960335547]: context deadline exceeded
E1130 20:43:52.594195 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone986083038]: context deadline exceeded
E1130 20:45:53.031191 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone310359717]: context deadline exceeded
E1130 20:46:02.621119 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone229514688]: context deadline exceeded
I1130 20:46:12.225365 1 client.go:173] send response key env:dev, type status_sync_event
I1130 20:46:12.225501 1 client.go:173] send response key env:service, type status_sync_event
I1130 20:46:12.232444 1 worker.go:140] get command: cluster:1.env:dev.envId:1/status_sync
I1130 20:46:12.236277 1 worker.go:140] get command: cluster:1.env:service.envId:3/status_sync
E1130 20:47:12.234832 1 daemon.go:94] dev do sync: git repo not ready
E1130 20:47:22.235540 1 daemon.go:94] service do sync: git repo not ready
E1130 20:48:03.109816 1 repo.go:245] env: dev repo new: git clone --mirror: running git command: git [clone --mirror git@dev:operation-paas-front-gitops/dev.git /tmp/choerodon-git-clone112470047]: context deadline exceeded
E1130 20:48:12.686728 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone558422770]: context deadline exceeded

crockitwood · 2018 年11 月 30 日 06:10

[quote=“yuanyblf, post:1, topic:1241”]
E1130 20:33:02.436634 1 repo.go:245] env: service repo new: git clone --mirror: running git command: git [clone --mirror git@service:operation-paas-service-gitops/service.git /tmp/choerodon-git-clone191378340]: context deadline exceeded
[/quote]日志上看是agent克隆gitops库的时候超时了，这边最大超时时间是一分钟，可以先确认一下agent 容器中到gitlab网络是否通畅。

yuanyblf · 2018 年11 月 30 日 06:48

网络是通的，都是在同一个K8集群下，只是不同的命名空间。

命名空间未做网络隔离

crockitwood · 2018 年11 月 30 日 06:54

进pod中执行一下 git clone git@service:operation-paas-service-gitops/service.git 和cat一下 /etc/ssh/ssh_config这个文件

yuanyblf · 2018 年11 月 30 日 06:54

clone --mirror git@service:operation-paas-service-gitops/service.git

疑问：git@service --日志输出的是service，但这不是我真实的gitlab地址啊。

crockitwood · 2018 年11 月 30 日 06:55

这个是在ssh_config中配置的别名。

yuanyblf · 2018 年11 月 30 日 06:56

bash-4.4# cat /etc/ssh/ssh_config
Host dev
HostName gitlab.k8s.skyflyer.cn
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
IdentityFile /rsa-dev
LogLevel error
Host service
HostName gitlab.k8s.skyflyer.cn
StrictHostKeyChecking no
UserKnownHostsFile /dev/null
IdentityFile /rsa-service
LogLevel error

yuanyblf · 2018 年11 月 30 日 06:56

一直卡在这里

crockitwood · 2018 年11 月 30 日 06:59

试一下http的有没有响应

yuanyblf · 2018 年11 月 30 日 06:59

随便一个开源的库是可以clone下来的

crockitwood · 2018 年11 月 30 日 07:00

就克隆你们gitlab内部的东西

yuanyblf · 2018 年11 月 30 日 07:02

使用完整的git地址是可以的；
使用别名就不行

crockitwood · 2018 年11 月 30 日 07:03

别名是ssh下行才有用的。

crockitwood · 2018 年11 月 30 日 07:05

通过ssh地址克隆可以吗？

yuanyblf · 2018 年11 月 30 日 07:06

gitlab ssh 已经开启了

yuanyblf · 2018 年11 月 30 日 07:06

本地 git clone 都是正常的，端口也是 22

仓库的代码都是我在本地push上去的

yuanyblf · 2018 年11 月 30 日 07:09

是不是仓库是私有的问题

crockitwood · 2018 年11 月 30 日 07:20

不是这个问题。这个ssh_config别名应该是可以用，的重启一下agent的pod试试。

yuanyblf · 2018 年11 月 30 日 07:24

重启试过，不行。

yuanyblf · 2018 年11 月 30 日 07:27

本地git clone 是可以的