mc-data-pipeline - Details for Query 36

Submitted Time: 2025/12/01 17:41:03
Duration: 1 s

Show the Stage ID and Task ID that corresponds to the max metric

Details

== Physical Plan ==
* Project (2)
+- MicroBatchScan (1)


(1) MicroBatchScan
Output [7]: [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13]
class org.apache.spark.sql.kafka010.KafkaSourceProvider$KafkaScan

(2) Project [codegen id : 1]
Output [2]: [topic#9, cast(value#8 as string) AS value#21]
Input [7]: [key#7, value#8, topic#9, partition#10, offset#11L, timestamp#12, timestampType#13]

SQL / DataFrame Properties

Name	Value
spark.sql.adaptive.enabled	false
spark.sql.cbo.enabled	false
spark.sql.requireAllClusterKeysForDistribution	false
spark.sql.streaming.aggregation.stateFormatVersion	2
spark.sql.streaming.flatMapGroupsWithState.stateFormatVersion	2
spark.sql.streaming.join.stateFormatVersion	2
spark.sql.streaming.multipleWatermarkPolicy	min
spark.sql.streaming.stateStore.compression.codec	lz4
spark.sql.streaming.stateStore.providerClass	org.apache.spark.sql.execution.streaming.state.HDFSBackedStateStoreProvider
spark.sql.streaming.stateStore.rocksdb.formatVersion	5
spark.sql.streaming.statefulOperator.useStrictDistribution	true