c7n-mysql挂了

猪齿鱼所有平台都无法访问了
c7n-mysql,和harbor状态变成了CrashLoopBackOff
mysql的日志如下:

[root@env-sanitation-db ~]# kubectl  logs c7n-mysql-5c5dc67dbb-zzfdg -n c7n-system 
2021-07-22T13:57:43.985870Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
2021-07-22T13:57:44.624573Z 0 [Note] mysqld (mysqld 5.7.23) starting as process 1 ...
2021-07-22T13:57:45.296798Z 0 [Note] InnoDB: PUNCH HOLE support available
2021-07-22T13:57:45.296833Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-07-22T13:57:45.296837Z 0 [Note] InnoDB: Uses event mutexes
2021-07-22T13:57:45.296840Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-07-22T13:57:45.296843Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
2021-07-22T13:57:45.296846Z 0 [Note] InnoDB: Using Linux native AIO
2021-07-22T13:57:45.297948Z 0 [Note] InnoDB: Number of pools: 1
2021-07-22T13:57:45.298194Z 0 [Note] InnoDB: Using CPU crc32 instructions
2021-07-22T13:57:45.300783Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-07-22T13:57:45.309514Z 0 [Note] InnoDB: Completed initialization of buffer pool
2021-07-22T13:57:45.312052Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-07-22T13:57:48.621149Z 0 [ERROR] InnoDB: Unable to lock ./ib_logfile1 error: 11
2021-07-22T13:57:48.621180Z 0 [Note] InnoDB: Check that you do not already have another mysqld process using the same InnoDB data or log files.
2021-07-22 21:57:48 0x7fa9338ba740  InnoDB: Assertion failure in thread 140364690990912 in file fil0fil.cc line 896
InnoDB: Failing assertion: success
InnoDB: We intentionally generate a memory trap.
InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
InnoDB: If you get repeated assertion failures or crashes, even
InnoDB: immediately after the mysqld startup, there may be
InnoDB: corruption in the InnoDB tablespace. Please refer to
InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html
InnoDB: about forcing recovery.
13:57:48 UTC - mysqld got signal 6 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.
Attempting to collect some information that could help diagnose the problem.
As this is a crash and something is definitely wrong, the information
collection process might fail.

key_buffer_size=8388608
read_buffer_size=131072
max_used_connections=0
max_threads=1500
thread_count=0
connection_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 604254 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0 thread_stack 0x40000
mysqld(my_print_stacktrace+0x2c)[0x55cfab96fa6c]
mysqld(handle_fatal_signal+0x479)[0x55cfab29b709]
/lib/x86_64-linux-gnu/libpthread.so.0(+0x110c0)[0x7fa93349a0c0]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0xcf)[0x7fa931c26fff]
/lib/x86_64-linux-gnu/libc.so.6(abort+0x16a)[0x7fa931c2842a]
mysqld(+0x628c7b)[0x55cfab271c7b]
mysqld(+0xfa309b)[0x55cfabbec09b]
mysqld(_Z40fil_open_log_and_system_tablespace_filesv+0xeb)[0x55cfabbefdcb]
mysqld(_Z34innobase_start_or_create_for_mysqlv+0x2f20)[0x55cfabaea0b0]
mysqld(+0xd6d933)[0x55cfab9b6933]
mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x4f)[0x55cfab2e63df]
mysqld(+0xb143d6)[0x55cfab75d3d6]
mysqld(_Z40plugin_register_builtin_and_init_core_sePiPPc+0x2f0)[0x55cfab7605c0]
mysqld(+0x64ae16)[0x55cfab293e16]
mysqld(_Z11mysqld_mainiPPc+0xc71)[0x55cfab2959d1]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf1)[0x7fa931c142e1]
mysqld(_start+0x2a)[0x55cfab28c0ba]
The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains
information that should help you find out what is causing the crash.

说是缓存的问题
查看缓存

[root@env-sanitation-db ~]# free -h
              total        used        free      shared  buff/cache   available
Mem:            62G         17G        8.7G        3.4G         36G         41G
Swap:            0B          0B          0B

node信息:

[root@env-sanitation-db ~]# kubectl get nodes
NAME    STATUS   ROLES                              AGE   VERSION
node1   Ready    control-plane,etcd,master,worker   42d   v1.21.0

大佬帮忙看一下,要怎么弄~~~

你所有节点都是正常的?

这些pod一会running一会是CrashLoopBackOff

[root@env-sanitation-db ~]# kubectl get pod -n c7n-system 
NAME                                                 READY   STATUS             RESTARTS   AGE
agile-service-84c8496d77-kvpxt                       0/1     Running            17         42d
agile-service-init-db-jvcln                          0/1     Completed          0          42d
c7n-mysql-5c5dc67dbb-pht67                           0/1     Running            2          3m58s
c7n-redis-7487f9d79f-7s79m                           1/1     Running            0          42d
c7n-slaver-ct6rn                                     1/1     Running            0          42d
chartmuseum-chartmuseum-85b5548d74-qb5sz             1/1     Running            0          42d
choerodon-admin-68748594b9-5sqvw                     0/1     Running            19         42d
choerodon-asgard-6bbd744687-t8gj2                    0/1     Running            17         42d
choerodon-file-7fcc7f989c-qhtvx                      0/1     Running            17         42d
choerodon-front-7ff85d4cb-28lxf                      1/1     Running            0          42d
choerodon-front-hzero-7f56dc8866-qzjxn               1/1     Running            0          42d
choerodon-gateway-894594d47-h65p9                    0/1     Running            17         42d
choerodon-iam-5d67c446c8-l8725                       0/1     Running            17         42d
choerodon-message-68946b7679-vn9kd                   0/1     Running            17         42d
choerodon-monitor-5f949cb7f-sftmm                    0/1     CrashLoopBackOff   21         42d
choerodon-oauth-7f8847b-zwrmf                        0/1     Running            17         42d
choerodon-platform-7784b5cc49-v7mgj                  0/1     Running            17         42d
choerodon-register-0                                 1/1     Running            0          42d
choerodon-swagger-79f67c4d5b-n9p22                   0/1     Running            0          42d
code-repo-service-658659d948-st276                   0/1     Running            17         42d
code-repo-service-init-db-xc2wc                      0/1     Completed          0          42d
devops-service-64b6488c76-8bvvs                      0/1     Running            17         42d
elasticsearch-kb-594bc88ff8-w4764                    1/1     Running            0          42d
gitlab-gitlab-core-0                                 0/1     CrashLoopBackOff   17         42d
gitlab-gitlab-core-backup-scheduled-27111900-8g78p   0/1     Completed          0          2d22h
gitlab-gitlab-core-backup-scheduled-27113340-w8xcd   0/1     Completed          0          46h
gitlab-gitlab-core-backup-scheduled-27114780-47bbv   0/1     Completed          0          22h
gitlab-gitlab-database-0                             0/1     CrashLoopBackOff   25         42d
gitlab-gitlab-redis-0                                1/1     Running            0          42d
gitlab-runner-76d89f646b-mcfth                       1/1     Running            0          35d
gitlab-service-7cfc457745-r69qf                      0/1     Running            17         42d
harbor-harbor-core-56c8ccb944-lsw64                  0/1     Running            33         36d
harbor-harbor-database-0                             0/1     Running            16         36d
harbor-harbor-jobservice-7c75d4f48b-cxnzg            1/1     Running            0          36d
harbor-harbor-portal-659f6d6687-k4d6m                1/1     Running            0          36d
harbor-harbor-redis-0                                1/1     Running            0          36d
harbor-harbor-registry-55467bdfb5-wbmcq              2/2     Running            0          36d
harbor-harbor-trivy-0                                1/1     Running            0          36d
knowledgebase-service-7bc978647b-d7qzz               0/1     Running            18         42d
knowledgebase-service-init-db-52r8p                  0/1     Completed          0          42d
minio-0                                              1/1     Running            1          42d
minio-1                                              1/1     Running            1          42d
minio-2                                              1/1     Running            1          42d
minio-3                                              1/1     Running            1          42d
prod-repo-service-5dc765c5d7-qx4mp                   0/1     Running            19         42d
prod-repo-service-init-db-k6n68                      0/1     Completed          0          42d
sonarqube-postgresql-59f849764f-8d67r                0/1     CrashLoopBackOff   22         42d
sonarqube-sonarqube-77879f4d7c-hnv6c                 1/1     Running            0          42d
sonatype-nexus-f86d5745f-7z2c5                       1/1     Running            1          42d
test-manager-service-7965d4f854-bpc9z                0/1     Running            17         42d
test-manager-service-init-db-vm77j                   0/1     Completed          0          42d
workflow-service-df97d7556-rxjvr                     0/1     Running            17         42d
workflow-service-init-db-9nqk9                       0/1     Completed          0          42d