Sounds like I'm messed up the mount of overlay hostPath that stores the containers when updating from centos7 to 8-stream. I also noticed that 9-stream is in beta.
[root@ecs-t6-large-2-linux-20190912001402 ~]# podman ps
Error: kernel does not support overlay fs: 'overlay' is not supported over extfs at "/var/lib/containers/storage/overlay": backing file system is unsupported for this graph driver
For Kubernetes, we automatically apply the pod using yaml file like
metadata:
name: vo-hostpath-pod
spec:
containers:
- name: filebeat
image: ikubernetes/filebeat:5.6.7-alpine
env:
- name: REDIS_HOST
value: redis.ilinux.io:6379
- name: LOG_LEVEL
value: info
volumeMounts:
- name: varlog
mountPath: /var/log
- name: socket
mountPath: /var/run/docker.sock
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
volumes:
- name: varlog
hostPath:
path: /var/log
type: DirectoryOrCreate
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
type: Directory
- name: socket
hostPath:
path: /var/run/docker.sock
type: Socket
Debbug the command using strace
newfstatat(AT_FDCWD, "/root/bin/crun", 0xc00019e378, 0) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "/usr/bin/runc", {st_mode=S_IFREG|0755, st_size=11889776, ...}, 0) = 0
openat(AT_FDCWD, "/etc/selinux/refpolicy/contexts/lxc_contexts", O_RDONLY|O_CLOEXEC) = 9
epoll_ctl(4, EPOLL_CTL_ADD, 9, {events=EPOLLIN|EPOLLOUT|EPOLLRDHUP|EPOLLET, data={u32=1719281968, u64=140043422935344}}) = 0
fcntl(9, F_GETFL) = 0x8000 (flags O_RDONLY|O_LARGEFILE)
fcntl(9, F_SETFL, O_RDONLY|O_NONBLOCK|O_LARGEFILE) = 0
fstat(9, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
The open at /etc/selinux/refpolicy/contexts/lxc_contexts is wierd so I think there's sth about the selinux, so I remvoe container-selinux and everythin works fine.
[root@ecs-t6-large-2-linux-20190912001402 ~]# docker ps
Emulate Docker CLI using podman. Create /etc/containers/nodocker to quiet msg.
WARN[0000] Error validating CNI config file /etc/cni/net.d/10-flannel.conflist: [failed to find plugin "flannel" in path [/usr/local/libexec/cni /usr/libexec/cni /usr/local/lib/cni /usr/lib/cni /opt/cni/bin]]
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
Also after upgrade to CentOS 8, don't forget to disable firealld and selinux because it'll update the settings. I debug it through self ping success and couldn't get anything from browser.
[root@ecs-t6-large-2-linux-20190912001402 ~]# sudo ss -tulpn
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
udp UNCONN 0 0 0.0.0.0:5355 0.0.0.0:* users:(("systemd-resolve",pid=1045,fd=12))
udp UNCONN 0 0 127.0.0.53%lo:53 0.0.0.0:* users:(("systemd-resolve",pid=1045,fd=18))
udp UNCONN 0 0 192.168.0.173:123 0.0.0.0:* users:(("ntpd",pid=640,fd=21))
udp UNCONN 0 0 127.0.0.1:123 0.0.0.0:* users:(("ntpd",pid=640,fd=18))
udp UNCONN 0 0 0.0.0.0:123 0.0.0.0:* users:(("ntpd",pid=640,fd=16))
udp UNCONN 0 0 [::]:5355 [::]:* users:(("systemd-resolve",pid=1045,fd=14))
udp UNCONN 0 0 [fe80::f816:3eff:fe8b:cbe7]%eth0:123 [::]:* users:(("ntpd",pid=640,fd=22))
udp UNCONN 0 0 [::1]:123 [::]:* users:(("ntpd",pid=640,fd=19))
udp UNCONN 0 0 [::]:123 [::]:* users:(("ntpd",pid=640,fd=17))
tcp LISTEN 0 9 0.0.0.0:21 0.0.0.0:* users:(("pure-ftpd",pid=601,fd=5))
tcp LISTEN 0 511 0.0.0.0:888 0.0.0.0:* users:(("nginx",pid=2970,fd=20),("nginx",pid=2969,fd=20),("nginx",pid=2792,fd=20))
tcp LISTEN 0 128 0.0.0.0:8888 0.0.0.0:* users:(("BT-Panel",pid=764,fd=6))
tcp LISTEN 0 100 127.0.0.1:25 0.0.0.0:* users:(("master",pid=1030,fd=16))
tcp LISTEN 0 511 0.0.0.0:443 0.0.0.0:* users:(("nginx",pid=2970,fd=22),("nginx",pid=2969,fd=22),("nginx",pid=2792,fd=22))
tcp LISTEN 0 511 127.0.0.1:6379 0.0.0.0:* users:(("redis-server",pid=1706,fd=6))
tcp LISTEN 0 128 0.0.0.0:5355 0.0.0.0:* users:(("systemd-resolve",pid=1045,fd=13))
tcp LISTEN 0 1024 127.0.0.1:11211 0.0.0.0:* users:(("memcached",pid=734,fd=28))
tcp LISTEN 0 128 0.0.0.0:9999 0.0.0.0:* users:(("sshd",pid=1244,fd=5))
tcp LISTEN 0 511 0.0.0.0:80 0.0.0.0:* users:(("nginx",pid=2970,fd=21),("nginx",pid=2969,fd=21),("nginx",pid=2792,fd=21))
tcp LISTEN 0 9 [::]:21 [::]:* users:(("pure-ftpd",pid=601,fd=6))
tcp LISTEN 0 100 [::1]:25 [::]:* users:(("master",pid=1030,fd=17))
tcp LISTEN 0 150 *:3306 *:* users:(("mysqld",pid=2166,fd=19))
tcp LISTEN 0 128 [::]:5355 [::]:* users:(("systemd-resolve",pid=1045,fd=15))
tcp LISTEN 0 1024 [::1]:11211 [::]:* users:(("memcached",pid=734,fd=29))
tcp LISTEN 0 128 [::]:9999 [::]:* users:(("sshd",pid=1244,fd=6))
[root@ecs-t6-large-2-linux-20190912001402 ~]#
