k8s运行容器之Job（四）

k8s运行容器之Job（四）

Job

容器按照持续运行的时间可分为两类：服务类容器和工作类容器。

Kubernetes 的 Deployment、ReplicaSet 和 DaemonSet 都用于管理服务类容器；对于工作类容器，我们用 Job。

第一步：

先看一个简单的 Job 配置文件 myjob.yml：

① batch/v1 是当前 Job 的 apiVersion。

② 指明当前资源的类型为 Job。

③ restartPolicy 指定什么情况下需要重启容器。对于 Job，只能设置为 Never 或者 OnFailure。对于其他 controller（比如 Deployment）可以设置为 Always 。

第二步：通过 kubectl apply -f myjob.yml 启动 Job。

[root@ken ~]# kubectl apply -f myjob.ymljob.batch/job created

[root@ken ~]# kubectl apply -f myjob.ymljob.batch/job created

第三步：查看job的状态

[root@ken ~]# kubectl get jobNAME    COMPLETIONS   DURATION   AGEjob   1/1           4s         40s

[root@ken ~]# kubectl get jobNAME COMPLETIONS DURATION AGEjob 1/1 4s 40s

第四步：查看pod的状态

[root@ken ~]# kubectl get podNAME                                READY   STATUS      RESTARTS   AGEjob-8hczg                         0/1     Completed   0          83s

[root@ken ~]# kubectl get podNAME READY STATUS RESTARTS AGEjob-8hczg 0/1 Completed 0 83s

第五步：查看pod的标准输出

[root@ken ~]# kubectl logs myjob-8hczg hello world

[root@ken ~]# kubectl logs myjob-8hczg hello world

job失败的情况

讨论了job执行成功的情况，如果失败了会怎么样呢？

第一步：修改 myjob.yml，故意引入一个错误：

第二步：删除之前的job

[root@ken ~]# kubectl delete -f job.ymljob.batch "job" deleted[root@ken ~]# kubectl get jobNo resources found.

[root@ken ~]# kubectl delete -f job.ymljob.batch "job" deleted[root@ken ~]# kubectl get jobNo resources found.

第三步：运行新的job并查看状态

[root@ken ~]# kubectl apply -f job.ymljob.batch/job created[root@ken ~]# kubectl get jobNAME    COMPLETIONS   DURATION   AGEmyjob   0/1           6s         6s

[root@ken ~]# kubectl apply -f job.ymljob.batch/job created[root@ken ~]# kubectl get jobNAME COMPLETIONS DURATION AGEmyjob 0/1 6s 6s

可以发现完成为0

第四步：查看pod状态

[root@ken ~]# kubectl get podNAME                                READY   STATUS               RESTARTS   AGEmyjob-hc6ld                         0/1     ContainerCannotRun   0          64smyjob-hfblk                         0/1     ContainerCannotRun   0          60smyjob-t9f6v                         0/1     ContainerCreating    0          11smyjob-v2g7s                         0/1     ContainerCannotRun   0          31s

[root@ken ~]# kubectl get podNAME READY STATUS RESTARTS AGEmyjob-hc6ld 0/1 ContainerCannotRun 0 64smyjob-hfblk 0/1 ContainerCannotRun 0 60smyjob-t9f6v 0/1 ContainerCreating 0 11smyjob-v2g7s 0/1 ContainerCannotRun 0 31s

可以看到有多个 Pod，状态均不正常。kubectl describe pod 查看某个 Pod 的启动日志：

第五步：查看pod的启动日志

[root@ken-node1 ~]# kubectl describe pods job-r9lrlName:               job-r9lrlNamespace:          defaultPriority:           0PriorityClassName:  <none>Node:               ken-node3/192.168.163.134Start Time:         Sat, 03 Aug 2019 01:37:26 +0800Labels:             controller-uid=15d81b0a-b509-11e9-a9b7-000c29e2b20a                    job-name=jobAnnotations:        <none>Status:             FailedIP:                 10.244.2.41Controlled By:      Job/jobContainers:  job:    Container ID:  docker://4d4a0cf1698ba36266c8b59b21714a8547ec28a46e7e73ad1f0ce939cb3befc5    Image:         busybox    Image ID:      docker-pullable://busybox@sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70    Port:          <none>    Host Port:     <none>    Command:      echosssss      hello world    State:          Terminated      Reason:       ContainerCannotRun      Message:      OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"echosssss\": executable file not found in $PATH": unknown      Exit Code:    127      Started:      Sat, 03 Aug 2019 01:37:40 +0800      Finished:     Sat, 03 Aug 2019 01:37:40 +0800    Ready:          False    Restart Count:  0    Environment:    <none>    Mounts:      /var/run/secrets/kubernetes.io/serviceaccount from default-token-wsrwt (ro)Conditions:  Type              Status  Initialized       True   Ready             False   ContainersReady   False   PodScheduled      True Volumes:  default-token-wsrwt:    Type:        Secret (a volume populated by a Secret)    SecretName:  default-token-wsrwt    Optional:    falseQoS Class:       BestEffortNode-Selectors:  <none>Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s                 node.kubernetes.io/unreachable:NoExecute for 300sEvents:  Type     Reason     Age        From                Message  ----     ------     ----       ----                -------  Normal   Scheduled  60s        default-scheduler   Successfully assigned default/job-r9lrl to ken-node3  Normal   Pulling    <invalid>  kubelet, ken-node3  Pulling image "busybox"  Normal   Pulled     <invalid>  kubelet, ken-node3  Successfully pulled image "busybox"  Normal   Created    <invalid>  kubelet, ken-node3  Created container job  Warning  Failed     <invalid>  kubelet, ken-node3  Error: failed to start container "job": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"echosssss\": executable file not found in $PATH": unknown

[root@ken-node1 ~]# kubectl describe pods job-r9lrlName: job-r9lrlNamespace: defaultPriority: 0PriorityClassName: Node: ken-node3/192.168.163.134Start Time: Sat, 03 Aug 2019 01:37:26 +0800Labels: controller-uid=15d81b0a-b509-11e9-a9b7-000c29e2b20a job-name=jobAnnotations: Status: FailedIP: 10.244.2.41Controlled By: Job/jobContainers: job: Container ID: docker://4d4a0cf1698ba36266c8b59b21714a8547ec28a46e7e73ad1f0ce939cb3befc5 Image: busybox Image ID: docker-pullable://busybox@sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70 Port: Host Port: Command: echosssss hello world State: Terminated Reason: ContainerCannotRun Message: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"echosssss\": executable file not found in $PATH": unknown Exit Code: 127 Started: Sat, 03 Aug 2019 01:37:40 +0800 Finished: Sat, 03 Aug 2019 01:37:40 +0800 Ready: False Restart Count: 0 Environment: Mounts: /var/run/secrets/kubernetes.io/serviceaccount from default-token-wsrwt (ro)Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: default-token-wsrwt: Type: Secret (a volume populated by a Secret) SecretName: default-token-wsrwt Optional: falseQoS Class: BestEffortNode-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300sEvents: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled 60s default-scheduler Successfully assigned default/job-r9lrl to ken-node3 Normal Pulling kubelet, ken-node3 Pulling image "busybox" Normal Pulled kubelet, ken-node3 Successfully pulled image "busybox" Normal Created kubelet, ken-node3 Created container job Warning Failed kubelet, ken-node3 Error: failed to start container "job": Error response from daemon: OCI runtime create failed: container_linux.go:345: starting container process caused "exec: \"echosssss\": executable file not found in $PATH": unknown

日志显示没有可执行程序，符合我们的预期。

下面解释一个现象：为什么 kubectl get pod 会看到这么多个失败的 Pod？

原因是：当第一个 Pod 启动时，容器失败退出，根据 restartPolicy: Never，此失败容器不会被重启，但 Job DESIRED 的 Pod 是 1，目前 SUCCESSFUL 为 0，不满足，所以 Job controller 会启动新的 Pod，直到 SUCCESSFUL 为 1。对于我们这个例子，SUCCESSFUL 永远也到不了 1，所以 Job controller 会一直创建新的 Pod。为了终止这个行为，只能删除 Job。

[root@ken ~]# kubectl delete -f myjob.ymljob.batch "myjob" deleted[root@ken ~]# kubectl get podNAME                                READY   STATUS    RESTARTS   AGE

[root@ken ~]# kubectl delete -f myjob.ymljob.batch "myjob" deleted[root@ken ~]# kubectl get podNAME READY STATUS RESTARTS AGE

如果将 restartPolicy 设置为 OnFailure 会怎么样？下面我们实践一下，修改 myjob.yml 后重新启动。

[root@ken ~]# kubectl apply -f myjob.ymljob.batch/myjob created[root@ken ~]# kubectl get jobNAME    COMPLETIONS   DURATION   AGE

[root@ken ~]# kubectl apply -f myjob.ymljob.batch/myjob created[root@ken ~]# kubectl get jobNAME COMPLETIONS DURATION AGE

完成依然为0

再来查看一下pod的状态

[root@ken ~]# kubectl get podNAME          READY   STATUS             RESTARTS   AGEmyjob-5tbxw   0/1     CrashLoopBackOff   2          67s

[root@ken ~]# kubectl get podNAME READY STATUS RESTARTS AGEmyjob-5tbxw 0/1 CrashLoopBackOff 2 67s

这里只有一个 Pod，不过 RESTARTS 为 3，而且不断增加，说明 OnFailure 生效，容器失败后会自动重启。

并行执行 Job

有时，我们希望能同时运行多个 Pod，提高 Job 的执行效率。

第一步：这个可以通过 parallelism 设置。

[root@ken-master ~]# cat job.yml apiVersion: batch/v1kind: Jobmetadata:  name: ken1spec:  parallelism: 2  template:    spec:      containers:      - image: busybox        name: ken      restartPolicy: Never

[root@ken-master ~]# cat job.yml apiVersion: batch/v1kind: Jobmetadata: name: ken1spec: parallelism: 2 template: spec: containers: - image: busybox name: ken restartPolicy: Never

第二步：执行该yaml文件

[root@ken-master ~]# kubectl apply -f job.yml

[root@ken-master ~]# kubectl apply -f job.yml

第三步：查看job

[root@ken-master ~]# kubectl get poNAME                        READY   STATUS        RESTARTS   AGEken1-4w969                  0/1     Completed     0          69sken1-jhppx                  0/1     Completed     0          69s

[root@ken-master ~]# kubectl get poNAME READY STATUS RESTARTS AGEken1-4w969 0/1 Completed 0 69sken1-jhppx 0/1 Completed 0 69s

Job 一共启动了两个 Pod，而且 AGE 相同，可见是并行运行的。

第五步：查看相应的pod和job

[root@ken-master ~]# kubectl get poNAME                        READY   STATUS        RESTARTS   AGEken2-42c9k                  0/1     Completed     0          28sken2-6f4vc                  0/1     Completed     0          24sken2-n7pcl                  0/1     Completed     0          22sken2-ntznv                  0/1     Completed     0          21sken2-tnktd                  0/1     Completed     0          19sken2-xswmh                  0/1     Completed     0          28s[root@ken-master ~]# kubectl get jobNAME   COMPLETIONS   DURATION   AGEken2   6/6           17s        34s

[root@ken-master ~]# kubectl get poNAME READY STATUS RESTARTS AGEken2-42c9k 0/1 Completed 0 28sken2-6f4vc 0/1 Completed 0 24sken2-n7pcl 0/1 Completed 0 22sken2-ntznv 0/1 Completed 0 21sken2-tnktd 0/1 Completed 0 19sken2-xswmh 0/1 Completed 0 28s[root@ken-master ~]# kubectl get jobNAME COMPLETIONS DURATION AGEken2 6/6 17s 34s

上面的例子只是为了演示 Job 的并行特性，实际用途不大。不过现实中确实存在很多需要并行处理的场景。比如批处理程序，每个副本（Pod）都会从任务池中读取任务并执行，副本越多，执行时间就越短，效率就越高。这种类似的场景都可以用 Job 来实现。

定时执行job

Linux 中有 cron 程序定时执行任务，Kubernetes 的 CronJob 提供了类似的功能，可以定时执行 Job。

第一步：CronJob 配置文件示例如下：

① batch/v1beta1 是当前 CronJob 的 apiVersion。

② 指明当前资源的类型为 CronJob。

③ schedule 指定什么时候运行 Job，其格式与 Linux cron 一致。这里 */1 * * * * 的含义是每一分钟启动一次。

④ jobTemplate 定义 Job 的模板，格式与前面 Job 一致。

第二步：接下来通过 kubectl apply 创建 CronJob。

[root@ken ~]# kubectl apply -f myjob1.ymlcronjob.batch/hello created

[root@ken ~]# kubectl apply -f myjob1.ymlcronjob.batch/hello created

第三步：查看crontab的状态

[root@ken ~]# kubectl get cronjobNAME    SCHEDULE      SUSPEND   ACTIVE   LAST SCHEDULE   AGEhello   */1 * * * *   False     1        22s             3m12s

[root@ken ~]# kubectl get cronjobNAME SCHEDULE SUSPEND ACTIVE LAST SCHEDULE AGEhello */1 * * * * False 1 22s 3m12s

第四步：等待几分钟查看jobs的执行情况

[root@ken ~]# kubectl get jobNAME               COMPLETIONS   DURATION   AGEhello-1548766140   1/1           5s         2m24shello-1548766200   1/1           18s        83shello-1548766260   1/1           4s         23s

[root@ken ~]# kubectl get jobNAME COMPLETIONS DURATION AGEhello-1548766140 1/1 5s 2m24shello-1548766200 1/1 18s 83shello-1548766260 1/1 4s 23s

可以看到每隔一分钟就会启动一个 Job。

过段时间查看pod

第五步：执行 kubectl logs 可查看某个 Job 的pod运行日志：

[root@ken ~]# kubectl logs hello-1548766260-6s8lphello k8s job!

[root@ken ~]# kubectl logs hello-1548766260-6s8lphello k8s job!

查看pod会遗留很多已经完成的pod，只需要删除即可

kubectl delete -f myjob1.yml