安装gitlab报错 Error syncing pod，反复重启

Choerodon Installation management
#1
#2

请教各位是什么情况？

#3

补充一下：
三台服务器都是8核16G内存
后台一直在刷

image
image1353×431 46.7 KB

#4

你好，请查看gitlab pod的日志

#5

[root@k8s-node1 ~]# kubectl describe pod gitlab-8685b9fb88-cbn5v -n c7n-system
Name: gitlab-8685b9fb88-cbn5v
Namespace: c7n-system
Node: k8s-node3/172.16.0.27
Start Time: Wed, 09 Jan 2019 09:01:39 +0800
Labels: choerodon.io/app=gitlab
choerodon.io/infra=gitlab
choerodon.io/release=gitlab
pod-template-hash=4241659644
Annotations: kubernetes.io/created-by={“kind”:“SerializedReference”,“apiVersion”:“v1”,“reference”:{“kind”:“ReplicaSet”,“namespace”:“c7n-system”,“name”:“gitlab-8685b9fb88”,“uid”:"1ed89833-13aa-11e9-b581-5254166244a…
Status: Running
IP: 10.233.66.9
Created By: ReplicaSet/gitlab-8685b9fb88
Controlled By: ReplicaSet/gitlab-8685b9fb88
Containers:
gitlab:
Container ID: docker://640ad9b71824fb64dd54a29f8759453a9f444d813e19a93cf0083600a4f30246
Image: registry.cn-hangzhou.aliyuncs.com/choerodon-tools/gitlab-ce:11.4.7-ce.0
Image ID: docker-pullable://registry.cn-hangzhou.aliyuncs.com/choerodon-tools/gitlab-ce@sha256:f886e5b86b20a5b84888d2f013043afcf585949d7a6484e847c6531d2fff36d4
Ports: 22/TCP, 80/TCP, 443/TCP
Command:
/bin/sh
-c
update-permissions && bash /entrypoint.sh
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Wed, 09 Jan 2019 11:23:38 +0800
Finished: Wed, 09 Jan 2019 11:24:05 +0800
Ready: False
Restart Count: 30
Limits:
memory: 4Gi
Requests:
memory: 3Gi
Readiness: http-get http://:80/help delay=120s timeout=15s period=10s #success=1 #failure=3
Environment:
GITLAB_BACKUP_SCHEDULE: daily
TZ: Asia/Shanghai
VALUES_ENV_CONFIG: map[SMTP_PORT:465 MYSQL_DATABASE:gitlabhq_production MYSQL_USERNAME:root SMTP_TLS:true CHOERODON_CLIENT_ID:gitlab CHOERODON_API_URL:http://api.example.choerodon.io REDIS_PORT:6379 GITLAB_EMAIL_FROM:git.sys@example.com NODE_EXPORTER_ENABLE:false SMTP_ENABLE:false SMTP_ENABLE_STARTTLS_AUTO:true SMTP_USER_NAME:git.sys@example.com GITLAB_TIMEZONE:Asia/Shanghai OTP_KEY_BASE:b719fe119132c7810908bba18315259ed12888d4f5ee5430c42a776d840a396799b0a5ef0a801348c8a357f07aa72bbd58e25a84b8f247a25c72f539c7a6c5fa CHOERODON_OMNIAUTH_ENABLED:false OMNIAUTH_BLOCK_AUTO_CREATED_USERS:false GITLAB_SHELL_SSH_PORT:2289 GITLAB_EXTERNAL_URL:http://gitlab.wei-pay.com SMTP_AUTHENTICATION:login MYSQL_PASSWORD:password PROMETHEUS_ENABLE:false REDIS_HOST:gitlab-redis.c7n-system.svc SMTP_PASSWORD:password PROMETHEUS_LISTEN_ADDRESS::9091 MYSQL_PORT:3306 GITLAB_DEFAULT_CAN_CREATE_GROUP:true MYSQL_HOST:gitlab-mysql.c7n-system.svc SMTP_ADDRESS:smtp.mxhichina.com SMTP_DOMAIN:smtp.mxhichina.com GITLAB_USERNAME_CHANGING_ENABLED:false SECRET_KEY_BASE:6e657410d57c71b4fc3ed0d694e7842b1895a8b401d812c17fe61caf95b48a6d703cb53c112bc01ebd197a85da81b18e29682040e99b4f26594772a4a2c98c6d DB_KEY_BASE:bf2e47b68d6cafaef1d767e628b619365becf27571e10f196f98dc85e7771042b9203199d39aff91fcb6837c8ed83f2a912b278da50999bb11a2fbc0fba52964 SECRET_TOKEN:fbfb19c355066a9afb030992231c4a363357f77345edd0f2e772359e5be59b02538e1fa6cae8f93f7d23355341cea2b93600dab6d6c3edcdced558fc6d739860]
Mounts:
/certs from gitlab-data (rw)
/etc/gitlab from gitlab-data (rw)
/opt/choerodon/paas/etc/gitlab.rb from gitlab-config (rw)
/var/log/gitlab from gitlab-data (rw)
/var/opt/gitlab from gitlab-data (rw)
/var/run/secrets/kubernetes.io/serviceaccount from default-token-28c82 (ro)
Conditions:
Type Status
Initialized True
Ready False
PodScheduled True
Volumes:
gitlab-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: gitlab-pvc
ReadOnly: false
gitlab-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: gitlab-cm
Optional: false
default-token-28c82:
Type: Secret (a volume populated by a Secret)
SecretName: default-token-28c82
Optional: false
QoS Class: Burstable
Node-Selectors:
Tolerations:
Events:
Type Reason Age From Message

Normal Pulled 20m (x28 over 2h) kubelet, k8s-node3 Container image “registry.cn-hangzhou.aliyuncs.com/choerodon-tools/gitlab-ce:11.4.7-ce.0” already present on machine
Warning FailedSync 5m (x570 over 2h) kubelet, k8s-node3 Error syncing pod
Warning BackOff 17s (x591 over 2h) kubelet, k8s-node3 Back-off restarting failed container

只有这些，没有其他的了。

#6

你好，请查看gitlab pod的日志的命令是 kubectl logs

#7

image
image1325×380 12.5 KB

bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received ‘1’
---- Begin output of “bash” “/tmp/chef-script20190109-85-1lvr9fr” ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

[execute] rake aborted!
          StandardError: An error has occurred, all later migrations canceled:
          
          Index name 'index_notes_on_updated_at' on table 'notes' already exists
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/schema_statements.rb:953:in `add_index_options'
          /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mysql_ignore_postgresql_options.rb:44:in `add_index_options'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:548:in `add_index'
          /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mysql_ignore_postgresql_options.rb:23:in `add_index'
          /opt/gitlab/embedded/service/gitlab-rails/config/initializers/mysql_set_length_for_binary_indexes.rb:19:in `add_index'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:665:in `block in method_missing'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:634:in `block in say_with_time'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:634:in `say_with_time'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:654:in `method_missing'
          /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20140428105831_add_notes_index_updated_at.rb:4:in `change'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:608:in `exec_migration'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:591:in `block in migrate'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:590:in `migrate'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:768:in `migrate'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:1023:in `block in execute_migration_in_transaction'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:1071:in `ddl_transaction'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:1022:in `execute_migration_in_transaction'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:984:in `block in migrate'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:980:in `each'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:980:in `migrate'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:823:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/migration.rb:801:in `migrate'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/tasks/database_tasks.rb:139:in `migrate'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/activerecord-4.2.10/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:52:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/service/gitlab-rails/vendor/bundle/ruby/2.4.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
#8

你好，根据观察日志，这个是在gitlab进行数据库初始化时出错了。可以卸载gitlab后重新进行安装以解决问题。

卸载命令如下：

helm delete --purge gitlab-mysql-pvc gitlab-mysql gitlab-redis-pvc gitlab-redis gitlab-pvc  gitlab
#9

好的，谢谢！