使用kubectl管理作業(yè)
更新時(shí)間：
您可以在EMR on ACK控制臺(tái)管理您的作業(yè)，也可以通過Kubernetes工具或API直接管理您的作業(yè)。本文為您介紹如何通過kubectl管理Spark作業(yè)。
前提條件
已在E-MapReduce on ACK控制臺(tái)創(chuàng)建Spark集群，詳情請參見快速入門。
操作步驟
通過kubectl連接Kubernetes集群，詳情請參見獲取集群KubeConfig并通過kubectl工具連接集群。
您也可以通過API等方式連接Kubernetes集群，詳情請參見使用Kubernetes API。
執(zhí)行以下命令，管理作業(yè)。
您可以執(zhí)行以下命令，查看作業(yè)狀態(tài)。
kubectl describe SparkApplication <作業(yè)名> --namespace <集群對應(yīng)的namespace>返回信息如下所示。
Name: spark-pi-simple Namespace: c-48e779e0d9ad**** Labels: <none> Annotations: <none> API Version: sparkoperator.k8s.io/v1beta2 Kind: SparkApplication Metadata: Creation Timestamp: 2021-07-22T06:25:33Z Generation: 1 Resource Version: 7503740 UID: 930874ad-bb17-47f1-a556-55118c1d**** Spec: Arguments: 1000 Driver: Core Limit: 1000m Cores: 1 Memory: 4g Executor: Core Limit: 1000m Cores: 1 Instances: 1 Memory: 8g Memory Overhead: 1g Image: registry-vpc.cn-hangzhou.aliyuncs.com/emr/spark:emr-2.4.5-1.0.0 Main Application File: local:///opt/spark/examples/target/scala-2.11/jars/spark-examples_2.11-2.4.5.jar Main Class: org.apache.spark.examples.SparkPi Spark Version: 2.4.5 Type: Scala Status: Application State: State: RUNNING Driver Info: Pod Name: spark-pi-simple-driver Web UI Address: 172.16.230.240:4040 Web UI Ingress Address: spark-pi-simple.c-48e779e0d9ad4bfd.c7f6b768c34764c27ab740bdb1fc2a3ff.cn-hangzhou.alicontainer.com Web UI Ingress Name: spark-pi-simple-ui-ingress Web UI Port: 4040 Web UI Service Name: spark-pi-simple-ui-svc Execution Attempts: 1 Executor State: spark-pi-1626935142670-exec-1: RUNNING Last Submission Attempt Time: 2021-07-22T06:25:33Z Spark Application Id: spark-15b44f956ecc40b1ae59a27ca18d**** Submission Attempts: 1 Submission ID: d71f30e2-9bf8-4da1-8412-b585fd45**** Termination Time: <nil> Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal SparkApplicationAdded 17s spark-operator SparkApplication spark-pi-simple was added, enqueuing it for submission Normal SparkApplicationSubmitted 14s spark-operator SparkApplication spark-pi-simple was submitted successfully Normal SparkDriverRunning 13s spark-operator Driver spark-pi-simple-driver is running Normal SparkExecutorPending 7s spark-operator Executor spark-pi-1626935142670-exec-1 is pending Normal SparkExecutorRunning 6s spark-operator Executor spark-pi-1626935142670-exec-1 is running本文示例代碼中的
<集群對應(yīng)的namespace>，需要替換為集群的命名空間，您可以登錄E-MapReduce on ACK控制臺(tái)，在集群詳情頁面查看。本文示例代碼中的
<作業(yè)名>，您可以登錄E-MapReduce on ACK控制臺(tái)，在作業(yè)詳情頁面，查看已創(chuàng)建的作業(yè)名。您可以執(zhí)行以下命令，停止并刪除作業(yè)。
kubectl delete SparkApplication <作業(yè)名> -n <集群對應(yīng)的namespace>返回信息如下所示。
sparkapplication.sparkoperator.k8s.io "spark-pi-simple" deleted您可以執(zhí)行以下命令，查看作業(yè)日志。
kubectl logs <作業(yè)名-driver> -n <集群對應(yīng)的namespace>說明例如，作業(yè)名為spark-pi-simple，集群對應(yīng)的namespace為c-d2232227b951****，則對應(yīng)的命令為
kubectl logs spark-pi-simple-driver -n c-d2232227b951****。返回如下類似信息。
...... Pi is roughly 3.141488791414888 21/07/22 14:37:57 INFO SparkContext: Successfully stopped SparkContext 21/07/22 14:37:57 INFO ShutdownHookManager: Shutdown hook called 21/07/22 14:37:57 INFO ShutdownHookManager: Deleting directory /var/data/spark-b6a43b55-a354-44d7-ae5e-45b8b1493edb/spark-56aae0d1-37b9-4a7d-9c99-4e4ca12deb4b 21/07/22 14:37:57 INFO ShutdownHookManager: Deleting directory /tmp/spark-e2500491-6ed7-48d7-b94e-a9ebeb899320
文檔內(nèi)容是否對您有幫助？