Spark Connect server - Details for Query 36

Submitted Time: 2026/07/20 20:10:18
Duration: 3 s
Succeeded Jobs: 15 16

Show the Stage ID and Task ID that corresponds to the max metric

digraph G { 0 [labelType="html" label=" AdaptiveSparkPlan "]; 1 [labelType="html" label="ObjectHashAggregate number of output rows: 1 time in aggregation build: 77 ms spill size: 0.0 B number of sort fallback tasks: 0"]; 2 [labelType="html" label="Exchange shuffle records written: 50 local merged chunks fetched: 0 shuffle write time total (min, med, max (stageId: taskId)) 39 ms (0 ms, 0 ms, 8 ms (stage 22.0: task 121)) remote merged bytes read: 0.0 B local merged blocks fetched: 0 corrupt merged block chunks: 0 remote merged reqs duration: 0 ms remote merged blocks fetched: 0 records read: 50 local bytes read: 4.9 KiB fetch wait time: 0 ms remote bytes read: 0.0 B merged fetch fallback count: 0 local blocks read: 50 remote merged chunks fetched: 0 remote blocks read: 0 data size total (min, med, max (stageId: taskId)) 9.6 KiB (152.0 B, 152.0 B, 2.3 KiB (stage 22.0: task 160)) local merged bytes read: 0.0 B number of partitions: 1 remote reqs duration: 0 ms remote bytes read to disk: 0.0 B shuffle bytes written total (min, med, max (stageId: taskId)) 4.9 KiB (85.0 B, 85.0 B, 783.0 B (stage 22.0: task 160))"]; 3 [labelType="html" label="ObjectHashAggregate number of output rows: 50 time in aggregation build total (min, med, max (stageId: taskId)) 947 ms (10 ms, 13 ms, 153 ms (stage 22.0: task 118)) spill size total (min, med, max (stageId: taskId)) 0.0 B (0.0 B, 0.0 B, 0.0 B (stage 22.0: task 118)) number of sort fallback tasks: 0"]; subgraph cluster4 { isCluster="true"; label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n857 ms (9 ms, 12 ms, 131 ms (stage 22.0: task 118))"; 5 [labelType="html" label=" Project "]; 6 [labelType="html" label="Scan ExistingRDD Delta Table State #0 - file:/home/jovyan/spark-warehouse/test.db/fhvhv_trip/_delta_log number of output rows: 6"]; } 1->0; 2->1; 3->2; 5->3; 6->5; }

AdaptiveSparkPlan isFinalPlan=true

ObjectHashAggregate(keys=[], functions=[collect_set(domainMetadata#1130, 0, 0), collect_set(txn#1122, 0, 0), count(protocol#1126), sum(add#1123.size), last(metaData#1125, true), count(metaData#1125), last(protocol#1126, true), count(remove#1124), count(add#1123), count(txn#1122)])

Exchange SinglePartition, ENSURE_REQUIREMENTS, [plan_id=475]

ObjectHashAggregate(keys=[], functions=[partial_collect_set(domainMetadata#1130, 0, 0), partial_collect_set(txn#1122, 0, 0), partial_count(protocol#1126), partial_sum(add#1123.size), partial_last(metaData#1125, true), partial_count(metaData#1125), partial_last(protocol#1126, true), partial_count(remove#1124), partial_count(add#1123), partial_count(txn#1122)])

Project [txn#1122, add#1123, remove#1124, metaData#1125, protocol#1126, domainMetadata#1130]

Scan ExistingRDD Delta Table State #0 - file:/home/jovyan/spark-warehouse/test.db/fhvhv_trip/_delta_log[txn#1122,add#1123,remove#1124,metaData#1125,protocol#1126,cdc#1127,checkpointMetadata#1128,sidecar#1129,domainMetadata#1130,commitInfo#1131]

WholeStageCodegen (1)

Details

== Physical Plan ==
AdaptiveSparkPlan (10)
+- == Final Plan ==
   ObjectHashAggregate (6)
   +- ShuffleQueryStage (5), Statistics(sizeInBytes=9.6 KiB, rowCount=50)
      +- Exchange (4)
         +- ObjectHashAggregate (3)
            +- * Project (2)
               +- * Scan ExistingRDD Delta Table State #0 - file:/home/jovyan/spark-warehouse/test.db/fhvhv_trip/_delta_log (1)
+- == Initial Plan ==
   ObjectHashAggregate (9)
   +- Exchange (8)
      +- ObjectHashAggregate (7)
         +- Project (2)
            +- Scan ExistingRDD Delta Table State #0 - file:/home/jovyan/spark-warehouse/test.db/fhvhv_trip/_delta_log (1)


(1) Scan ExistingRDD Delta Table State #0 - file:/home/jovyan/spark-warehouse/test.db/fhvhv_trip/_delta_log [codegen id : 1]
Output [10]: [txn#1122, add#1123, remove#1124, metaData#1125, protocol#1126, cdc#1127, checkpointMetadata#1128, sidecar#1129, domainMetadata#1130, commitInfo#1131]
Arguments: [txn#1122, add#1123, remove#1124, metaData#1125, protocol#1126, cdc#1127, checkpointMetadata#1128, sidecar#1129, domainMetadata#1130, commitInfo#1131], Delta Table State #0 - file:/home/jovyan/spark-warehouse/test.db/fhvhv_trip/_delta_log MapPartitionsRDD[38] at Spark Connect - session_id: "58d45445-3af3-4104-9739-56b4041ec0f9"
plan {
  command {
    sql_command {
      sql: "select count(*) from test..., ExistingRDD, UnknownPartitioning(0)

(2) Project [codegen id : 1]
Output [6]: [txn#1122, add#1123, remove#1124, metaData#1125, protocol#1126, domainMetadata#1130]
Input [10]: [txn#1122, add#1123, remove#1124, metaData#1125, protocol#1126, cdc#1127, checkpointMetadata#1128, sidecar#1129, domainMetadata#1130, commitInfo#1131]

(3) ObjectHashAggregate
Input [6]: [txn#1122, add#1123, remove#1124, metaData#1125, protocol#1126, domainMetadata#1130]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#1130, 0, 0), partial_collect_set(txn#1122, 0, 0), partial_count(protocol#1126), partial_sum(add#1123.size), partial_last(metaData#1125, true), partial_count(metaData#1125), partial_last(protocol#1126, true), partial_count(remove#1124), partial_count(add#1123), partial_count(txn#1122)]
Aggregate Attributes [12]: [buf#2227, buf#2228, count#2229L, sum#2230L, last#2231, valueSet#2232, count#2233L, last#2234, valueSet#2235, count#2236L, count#2237L, count#2238L]
Results [12]: [buf#2239, buf#2240, count#2241L, sum#2242L, last#2243, valueSet#2244, count#2245L, last#2246, valueSet#2247, count#2248L, count#2249L, count#2250L]

(4) Exchange
Input [12]: [buf#2239, buf#2240, count#2241L, sum#2242L, last#2243, valueSet#2244, count#2245L, last#2246, valueSet#2247, count#2248L, count#2249L, count#2250L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=475]

(5) ShuffleQueryStage
Output [12]: [buf#2239, buf#2240, count#2241L, sum#2242L, last#2243, valueSet#2244, count#2245L, last#2246, valueSet#2247, count#2248L, count#2249L, count#2250L]
Arguments: 0

(6) ObjectHashAggregate
Input [12]: [buf#2239, buf#2240, count#2241L, sum#2242L, last#2243, valueSet#2244, count#2245L, last#2246, valueSet#2247, count#2248L, count#2249L, count#2250L]
Keys: []
Functions [10]: [collect_set(domainMetadata#1130, 0, 0), collect_set(txn#1122, 0, 0), count(protocol#1126), sum(add#1123.size), last(metaData#1125, true), count(metaData#1125), last(protocol#1126, true), count(remove#1124), count(add#1123), count(txn#1122)]
Aggregate Attributes [10]: [collect_set(domainMetadata#1130, 0, 0)#2134, collect_set(txn#1122, 0, 0)#2133, count(protocol#1126)#2132L, sum(add#1123.size)#2127L, last(metaData#1125) ignore nulls#2135, count(metaData#1125)#2131L, last(protocol#1126) ignore nulls#2136, count(remove#1124)#2130L, count(add#1123)#2129L, count(txn#1122)#2128L]
Results [14]: [collect_set(domainMetadata#1130, 0, 0)#2134 AS domainMetadata#2140, null AS fileSizeHistogram#2141, collect_set(txn#1122, 0, 0)#2133 AS setTransactions#2142, null AS numDeletedRecordsOpt#2143, count(protocol#1126)#2132L AS numOfProtocol#2144L, coalesce(sum(add#1123.size)#2127L, 0) AS sizeInBytes#2145L, last(metaData#1125) ignore nulls#2135 AS metadata#2146, count(metaData#1125)#2131L AS numOfMetadata#2147L, null AS deletedRecordCountsHistogramOpt#2148, null AS numDeletionVectorsOpt#2149, last(protocol#1126) ignore nulls#2136 AS protocol#2150, count(remove#1124)#2130L AS numOfRemoves#2151L, count(add#1123)#2129L AS numOfFiles#2152L, count(txn#1122)#2128L AS numOfSetTransactions#2153L]

(7) ObjectHashAggregate
Input [6]: [txn#1122, add#1123, remove#1124, metaData#1125, protocol#1126, domainMetadata#1130]
Keys: []
Functions [10]: [partial_collect_set(domainMetadata#1130, 0, 0), partial_collect_set(txn#1122, 0, 0), partial_count(protocol#1126), partial_sum(add#1123.size), partial_last(metaData#1125, true), partial_count(metaData#1125), partial_last(protocol#1126, true), partial_count(remove#1124), partial_count(add#1123), partial_count(txn#1122)]
Aggregate Attributes [12]: [buf#2227, buf#2228, count#2229L, sum#2230L, last#2231, valueSet#2232, count#2233L, last#2234, valueSet#2235, count#2236L, count#2237L, count#2238L]
Results [12]: [buf#2239, buf#2240, count#2241L, sum#2242L, last#2243, valueSet#2244, count#2245L, last#2246, valueSet#2247, count#2248L, count#2249L, count#2250L]

(8) Exchange
Input [12]: [buf#2239, buf#2240, count#2241L, sum#2242L, last#2243, valueSet#2244, count#2245L, last#2246, valueSet#2247, count#2248L, count#2249L, count#2250L]
Arguments: SinglePartition, ENSURE_REQUIREMENTS, [plan_id=465]

(9) ObjectHashAggregate
Input [12]: [buf#2239, buf#2240, count#2241L, sum#2242L, last#2243, valueSet#2244, count#2245L, last#2246, valueSet#2247, count#2248L, count#2249L, count#2250L]
Keys: []
Functions [10]: [collect_set(domainMetadata#1130, 0, 0), collect_set(txn#1122, 0, 0), count(protocol#1126), sum(add#1123.size), last(metaData#1125, true), count(metaData#1125), last(protocol#1126, true), count(remove#1124), count(add#1123), count(txn#1122)]
Aggregate Attributes [10]: [collect_set(domainMetadata#1130, 0, 0)#2134, collect_set(txn#1122, 0, 0)#2133, count(protocol#1126)#2132L, sum(add#1123.size)#2127L, last(metaData#1125) ignore nulls#2135, count(metaData#1125)#2131L, last(protocol#1126) ignore nulls#2136, count(remove#1124)#2130L, count(add#1123)#2129L, count(txn#1122)#2128L]
Results [14]: [collect_set(domainMetadata#1130, 0, 0)#2134 AS domainMetadata#2140, null AS fileSizeHistogram#2141, collect_set(txn#1122, 0, 0)#2133 AS setTransactions#2142, null AS numDeletedRecordsOpt#2143, count(protocol#1126)#2132L AS numOfProtocol#2144L, coalesce(sum(add#1123.size)#2127L, 0) AS sizeInBytes#2145L, last(metaData#1125) ignore nulls#2135 AS metadata#2146, count(metaData#1125)#2131L AS numOfMetadata#2147L, null AS deletedRecordCountsHistogramOpt#2148, null AS numDeletionVectorsOpt#2149, last(protocol#1126) ignore nulls#2136 AS protocol#2150, count(remove#1124)#2130L AS numOfRemoves#2151L, count(add#1123)#2129L AS numOfFiles#2152L, count(txn#1122)#2128L AS numOfSetTransactions#2153L]

(10) AdaptiveSparkPlan
Output [14]: [domainMetadata#2140, fileSizeHistogram#2141, setTransactions#2142, numDeletedRecordsOpt#2143, numOfProtocol#2144L, sizeInBytes#2145L, metadata#2146, numOfMetadata#2147L, deletedRecordCountsHistogramOpt#2148, numDeletionVectorsOpt#2149, protocol#2150, numOfRemoves#2151L, numOfFiles#2152L, numOfSetTransactions#2153L]
Arguments: isFinalPlan=true

SQL / DataFrame Properties

Name	Value
spark.databricks.delta.lastCommitVersionInSession	0
spark.remote	sc://spark-connect:15002
spark.sql.parquet.fieldId.read.enabled	true
spark.sql.parquet.fieldId.write.enabled	true