digraph G {
subgraph cluster0 {
isCluster="true";
label="WholeStageCodegen (1)\n \nduration: total (min, med, max (stageId: taskId))\n2.4 m (23.8 s, 39.1 s, 44.9 s (stage 9.0: task 12))";
1 [labelType="html" label="<b>ColumnarToRow</b><br><br>number of output rows: 20,405,666<br>number of input batches: 4,982"];
}
2 [labelType="html" label="<b>Scan parquet </b><br><br>number of files read: 1<br>scan time total (min, med, max (stageId: taskId))<br>8.4 s (1.5 s, 2.3 s, 2.4 s (stage 9.0: task 12))<br>metadata time: 0 ms<br>size of files read: 468.3 MiB<br>number of output rows: 20,405,666"];
2->1;
}
3
ColumnarToRow
WholeStageCodegen (1)
FileScan parquet [hvfhs_license_num#420,dispatching_base_num#421,originating_base_num#422,request_datetime#423,on_scene_datetime#424,pickup_datetime#425,dropoff_datetime#426,PULocationID#427,DOLocationID#428,trip_miles#429,trip_time#430L,base_passenger_fare#431,tolls#432,bcf#433,sales_tax#434,congestion_surcharge#435,airport_fee#436,tips#437,driver_pay#438,shared_request_flag#439,shared_match_flag#440,access_a_ride_flag#441,wav_request_flag#442,wav_match_flag#443,cbd_congestion_fee#444] Batched: true, DataFilters: [], Format: Parquet, Location: InMemoryFileIndex(1 paths)[file:/home/jovyan/Inbound/fhvhv_tripdata_2025-01.parquet], PartitionFilters: [], PushedFilters: [], ReadSchema: struct<hvfhs_license_num:string,dispatching_base_num:string,originating_base_num:string,request_d...
== Physical Plan ==
* ColumnarToRow (2)
+- Scan parquet (1)
(1) Scan parquet
Output [25]: [hvfhs_license_num#420, dispatching_base_num#421, originating_base_num#422, request_datetime#423, on_scene_datetime#424, pickup_datetime#425, dropoff_datetime#426, PULocationID#427, DOLocationID#428, trip_miles#429, trip_time#430L, base_passenger_fare#431, tolls#432, bcf#433, sales_tax#434, congestion_surcharge#435, airport_fee#436, tips#437, driver_pay#438, shared_request_flag#439, shared_match_flag#440, access_a_ride_flag#441, wav_request_flag#442, wav_match_flag#443, cbd_congestion_fee#444]
Batched: true
Location: InMemoryFileIndex [file:/home/jovyan/Inbound/fhvhv_tripdata_2025-01.parquet]
ReadSchema: struct<hvfhs_license_num:string,dispatching_base_num:string,originating_base_num:string,request_datetime:timestamp_ntz,on_scene_datetime:timestamp_ntz,pickup_datetime:timestamp_ntz,dropoff_datetime:timestamp_ntz,PULocationID:int,DOLocationID:int,trip_miles:double,trip_time:bigint,base_passenger_fare:double,tolls:double,bcf:double,sales_tax:double,congestion_surcharge:double,airport_fee:double,tips:double,driver_pay:double,shared_request_flag:string,shared_match_flag:string,access_a_ride_flag:string,wav_request_flag:string,wav_match_flag:string,cbd_congestion_fee:double>
(2) ColumnarToRow [codegen id : 1]
Input [25]: [hvfhs_license_num#420, dispatching_base_num#421, originating_base_num#422, request_datetime#423, on_scene_datetime#424, pickup_datetime#425, dropoff_datetime#426, PULocationID#427, DOLocationID#428, trip_miles#429, trip_time#430L, base_passenger_fare#431, tolls#432, bcf#433, sales_tax#434, congestion_surcharge#435, airport_fee#436, tips#437, driver_pay#438, shared_request_flag#439, shared_match_flag#440, access_a_ride_flag#441, wav_request_flag#442, wav_match_flag#443, cbd_congestion_fee#444]