Spark Connect server - Details for Stage 12 (Attempt 0)

Resource Profile Id: 0
Total Time Across All Tasks: 5 s
Locality Level Summary: Node local: 5; Process local: 45
Shuffle Read Size / Records: 5.3 KiB / 7
Shuffle Write Size / Records: 2.9 KiB / 50
Associated Job Ids: 11

digraph G { subgraph clusterstage_12 { label="Stage 12"; 37 [labelType="html" label="SQLExecutionRDD [37] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; 41 [labelType="html" label="SQLExecutionRDD [41] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; subgraph cluster86 { label="Exchange"; 32 [labelType="html" label="ShuffledRowRDD [32] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } subgraph cluster81 { label="WholeStageCodegen (2)"; 33 [labelType="html" label="MapPartitionsRDD [33] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } subgraph cluster80 { label="DeserializeToObject"; 34 [labelType="html" label="MapPartitionsRDD [34] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } subgraph cluster79 { label="MapPartitions"; 35 [labelType="html" label="MapPartitionsRDD [35] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } subgraph cluster76 { label="WholeStageCodegen (3)"; 36 [labelType="html" label="MapPartitionsRDD [36] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } subgraph cluster87 { label="map"; 38 [labelType="html" label="Delta Table State #0 - file:/home/jovyan/spark-warehouse/test.db/fhvhv_trip/_delta_log [38] [Cached] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } subgraph cluster89 { label="WholeStageCodegen (1)"; 39 [labelType="html" label="MapPartitionsRDD [39] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } subgraph cluster88 { label="Project"; 40 [labelType="html" label="MapPartitionsRDD [40] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } subgraph cluster96 { label="map"; 42 [labelType="html" label="Delta Table State with Stats #0 - file:/home/jovyan/spark-warehouse/test.db/fhvhv_trip/_delta_log [42] [Cached] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } subgraph cluster98 { label="WholeStageCodegen (1)"; 43 [labelType="html" label="MapPartitionsRDD [43] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } subgraph cluster97 { label="Exchange"; 44 [labelType="html" label="MapPartitionsRDD [44] [Unordered] Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"\nplan {\n command {\n sql_command {\n sql: "select count(*) from test..."]; } } 32->33; 33->34; 34->35; 35->36; 36->37; 37->38; 38->39; 39->40; 40->41; 41->42; 42->43; 43->44; }

31,32

Event Timeline

Enable zooming