一键部署失败(MySql)

damage85 · 2018 年9 月 17 日 02:55

Choerodon平台版本: 0.6.0
遇到问题的执行步骤:
第四步：一键部署Choerodon
文档地址:
环境信息(如:节点信息):
报错日志:

[root@node1 ~]# curl -o choerodon-install.sh     https://file.choerodon.com.cn/choerodon-install/install-0.9.sh && sh choerodon-install.sh values.sh
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 60856  100 60856    0     0   181k      0 --:--:-- --:--:-- --:--:--  181k
customresourcedefinition "c7nhelmreleases.choerodon.io" configured

[Step 1]: checking installation parameters ...
✔ Installation parameters is OK.

[Step 2]: checking helm release ...

[Step 3]: checking job ...

[Step 4]: checking helm package and check its version ...
✔ Success check helm version.

[Step 5]: add helm repositories ...
"c7n" has been added to your repositories
Hang tight while we grab the latest from your chart repositories...
...Skip local chart repository
...Successfully got an update from the "c7n" chart repository
...Successfully got an update from the "stable" chart repository
Update Complete. ⎈ Happy Helming!⎈ 
✔ Success add helm repositories.

[Step 6]: checking tiller ready ...
✔ Tiller is ready.

[Step 7]: install Mysql for Choerodon ...
NAME:   choerodon-mysql-pv
LAST DEPLOYED: Mon Sep 17 10:54:07 2018
NAMESPACE: choerodon-devops-prod
STATUS: DEPLOYED

RESOURCES:
==> v1/PersistentVolume
NAME                CAPACITY  ACCESS MODES  RECLAIM POLICY  STATUS  CLAIM                                      STORAGECLASS  REASON  AGE
choerodon-mysql-pv  3Gi       RWO           Retain          Bound   choerodon-devops-prod/choerodon-mysql-pvc  0s

==> v1/PersistentVolumeClaim
NAME                 STATUS   VOLUME              CAPACITY  ACCESS MODES  STORAGECLASS  AGE
choerodon-mysql-pvc  Pending  choerodon-mysql-pv  0         0s


NAME:   choerodon-mysql
LAST DEPLOYED: Mon Sep 17 10:54:07 2018
NAMESPACE: choerodon-devops-prod
STATUS: DEPLOYED

RESOURCES:
==> v1/Pod(related)
NAME                             READY  STATUS             RESTARTS  AGE
choerodon-mysql-dcb9b88cd-k79p5  0/1    ContainerCreating  0         0s

==> v1/ConfigMap
NAME                    DATA  AGE
choerodon-mysql-config  1     0s

==> v1/Service
NAME             TYPE       CLUSTER-IP     EXTERNAL-IP  PORT(S)   AGE
choerodon-mysql  ClusterIP  10.233.26.140  <none>       3306/TCP  0s

==> v1beta1/Deployment
NAME             DESIRED  CURRENT  UP-TO-DATE  AVAILABLE  AGE
choerodon-mysql  1        1        1           0          0s


✔ Success of install Mysql for Choerodon.

[Step 8]: checking Mysql ready ...
➜ choerodon-mysql not ready,sleep 5s,check it.
➜ choerodon-mysql not ready,sleep 5s,check it.
➜ choerodon-mysql not ready,sleep 5s,check it.

原因分析:

提出您分析问题的过程,以便我们能更准确的找到问题所在
疑问:

choerodon-mysql not ready,sleep 5s,check it. 这句会一直出来，/u01目录里面并没有安装Mysql，/u01是NFS目录

[root@node1 ~]# cd /u01
[root@node1 u01]# ll
总用量 60
drwxr-xr-x. 2 root root 6 9月 14 17:50 aaa
-rw-r–r--. 1 root root 60856 9月 14 17:52 choerodon-install.sh
[root@node1 u01]#
[root@node1 u01]#
[root@node1 u01]#
[root@node1 u01]#
[root@node1 u01]#
[root@node1 u01]#

vinkdong · 2018 年9 月 17 日 03:01

你在nfs中创建对应的目录了吗

damage85 · 2018 年9 月 17 日 03:24

需要创建什么目录？我可以参考教程的哪部分？

vinkdong · 2018 年9 月 17 日 03:26

values 中定义的目录你需要在nfs中创建对应的目录

damage85 · 2018 年9 月 17 日 09:08

把目录全部建立了也不行，目录都是在NFS客户端建立的

[root@node1 io-choerodon]# ll
总用量 0
drwxr-xr-x. 2 root root 6 9月 17 17:02 chartmuseum
drwxr-xr-x. 2 root root 6 9月 17 17:02 gitlab
drwxr-xr-x. 5 root root 48 9月 17 17:02 harbor
drwxr-xr-x. 3 root root 15 9月 17 17:01 kafka
drwxr-xr-x. 2 root root 6 9月 17 17:02 minio
drwxr-xr-x. 2 root root 6 9月 17 17:01 mysql
drwxr-xr-x. 3 root root 15 9月 17 17:01 zookeeper
[root@node1 io-choerodon]# pwd
/u01/io-choerodon
[root@node1 io-choerodon]#

vinkdong · 2018 年9 月 17 日 09:31

执行下面命令看下

kubectl get po -n choerodon-devops-prod

damage85 · 2018 年9 月 17 日 10:12

[root@node1 io-choerodon]# kubectl get po -n choerodon-devops-prod
NAME READY STATUS RESTARTS AGE
choerodon-mysql-dcb9b88cd-wlgqv 0/1 ContainerCreating 0 1h
dnsmasq-5f5777c999-lznnl 1/1 Running 0 18d

顺便说一下，自建的mysql目录是空的

vinkdong · 2018 年9 月 17 日 11:26

kubectl describe po  choerodon-mysql-dcb9b88cd-wlgqv  -n choerodon-devops-prod

看一下pod的状态

damage85 · 2018 年9 月 18 日 01:41

[root@node1 ~]# kubectl describe po  choerodon-mysql-dcb9b88cd-wlgqv  -n choerodon-devops-prod
Name:           choerodon-mysql-dcb9b88cd-wlgqv
Namespace:      choerodon-devops-prod
Node:           node1/172.16.27.141
Start Time:     Mon, 17 Sep 2018 17:05:10 +0800
Labels:         app=mysql
                chart=mysql-0.1.0
                heritage=Tiller
                pod-template-hash=876564478
                release=choerodon-mysql
Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"choerodon-devops-prod","name":"choerodon-mysql-dcb9b88cd","uid":"c7968ead-ba58-11...
Status:         Pending
IP:             
Controlled By:  ReplicaSet/choerodon-mysql-dcb9b88cd
Containers:
  choerodon-mysql:
    Container ID:   
    Image:          mysql:5.7.22
    Image ID:       
    Port:           3306/TCP
    State:          Waiting
      Reason:       ContainerCreating
    Ready:          False
    Restart Count:  0
    Liveness:       tcp-socket :3306 delay=60s timeout=5s period=10s #success=1 #failure=3
    Environment:
      MYSQL_PASSWORD:       password
      MYSQL_ROOT_PASSWORD:  password
      MYSQL_USER:           choerodon
    Mounts:
      /etc/mysql/conf.d/my.cnf from config-volume (rw)
      /var/lib/mysql from mysql (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-dzc7n (ro)
Conditions:
  Type           Status
  Initialized    True 
  Ready          False 
  PodScheduled   True 
Volumes:
  mysql:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  choerodon-mysql-pvc
    ReadOnly:   false
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      choerodon-mysql-config
    Optional:  false
  default-token-dzc7n:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-dzc7n
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     <none>
Events:
  Type     Reason       Age                  From            Message
  ----     ------       ----                 ----            -------
  Warning  FailedSync   26m (x428 over 16h)  kubelet, node1  Error syncing pod
  Warning  FailedMount  5m (x335 over 16h)   kubelet, node1  Unable to mount volumes for pod "choerodon-mysql-dcb9b88cd-wlgqv_choerodon-devops-prod(c797733c-ba58-11e8-99f7-005056bf6ceb)": timeout expired waiting for volumes to attach/mount for pod "choerodon-devops-prod"/"choerodon-mysql-dcb9b88cd-wlgqv". list of unattached/unmounted volumes=[mysql]
  Warning  FailedMount  28s (x591 over 16h)  kubelet, node1  (combined from similar events): MountVolume.SetUp failed for volume "choerodon-mysql-pv" : mount failed: exit status 32
Mounting command: systemd-run
Mounting arguments: --description=Kubernetes transient mount for /var/lib/kubelet/pods/c797733c-ba58-11e8-99f7-005056bf6ceb/volumes/kubernetes.io~nfs/choerodon-mysql-pv --scope -- mount -t nfs nfs.example.choerodon.io:/u01/io-choerodon/mysql /var/lib/kubelet/pods/c797733c-ba58-11e8-99f7-005056bf6ceb/volumes/kubernetes.io~nfs/choerodon-mysql-pv
Output: Running scope as unit run-17352.scope.
mount.nfs: Failed to resolve server nfs.example.choerodon.io: Name or service not known
mount.nfs: Operation already in progress

vinkdong · 2018 年9 月 18 日 01:56

你的nfs服务器域名是 nfs.example.choerodon.io 吗？

damage85 · 2018 年9 月 18 日 02:10

api.example.choerodon.io
nfs.example.choerodon.io
等等，这些都没有配置，本机部署了example.choerodon.io这个

vinkdong · 2018 年9 月 18 日 02:13

通过其他主机能够访问 nfs.example.choerodon.io 吗，是否配置了域名解析？

damage85 · 2018 年9 月 18 日 02:21

我把NFS这个域名换成IP了，也不行
PS，我安装的是单机版k8s，NFS在另外一台服务器上

vinkdong · 2018 年9 月 18 日 02:22

根据刚才的步骤再贴一次 kubectl describe po xx 的结果

damage85 · 2018 年9 月 18 日 02:24

改了IP之后安装成功了，刚刚安装等待比较久。
现在走到这一步了

[Step 14]: install Gitlab ...
Error: Job failed: DeadlineExceeded
✖ create gitlab database failed

damage85 · 2018 年9 月 18 日 09:58

这个问题是算完成了？

vinkdong · 2018 年9 月 19 日 01:45

1.执行 kubectl describe get job -n choerodon-devops-prod 你应该能看到一个 create-database-xx 的job
2.执行 kubectl get job "create-database-xx" -n choerodon-devops-prod -o json | jq 'del(.spec.selector)' | jq 'del(.spec.template.metadata.labels)' | kubectl replace --force -f - 如果提示jq不存在先安装jq
3.kubectl describe get pod -n choerodon-devops-prod -w 查看是否有新启动的pod
4.如果有追踪下它的日志

damage85 · 2018 年9 月 20 日 02:05

[root@node1 ~]# kubectl describe get job -n choerodon-devops-prod
the server doesn’t have a resource type “get”

vinkdong · 2018 年9 月 20 日 03:03

sorry, 没有 describe,执行这句

kubectl  get job -n choerodon-devops-prod

damage85 · 2018 年9 月 25 日 07:17

请问一下，我用自动部署，走到了第14步，如果失败了，是不是之前的全部步骤都需要回滚？譬如mysql，需要自己手动删除，还是怎样处理？