Evaluate log configurations to recommend tiered storage, retention policies, and access controls.

Design stream processing topologies that implement join patterns, aggregation windows, and state management for real-time data transformation.

Optimize real-time data flows by analyzing throughput bottlenecks, partition strategies, and resource allocation to meet SLAs within budget limits.

Explain core patterns for schema evolution (backward/forward/full compatibility, additive vs. breaking changes) and select the right strategy.

Implement versioned event/data contracts with Avro or Protobuf using a schema registry and enforce compatibility rules in CI/CD.

Orchestrate real‑time rollout plans across producers, consumers, and storage (Kafka topics, CDC sinks, warehouses) with monitoring and rollback.

Stream pipeline design by analyzing failure scenarios and business requirements to prevent data loss or duplication.

Implement exactly-once processing semantics across producer, processor, and sink layers using transactions, checkpoints, and idempotent operations.

Evaluate watermarking and windowing configurations to optimize the tradeoff between latency and data completeness.

Explain the execution model of Spark Structured Streaming and build a simple pipeline from a file source to a console sink.

Develop streaming pipelines that integrate with Kafka, apply event-time processing with watermarks, and write reliable outputs to Delta Lake.

Build an end-to-end Spark streaming pipeline that can be deployed in real-world production environments.

Explain Spark’s streaming model and produce a dashboard-ready table from a simple file source.

Construct a real-time pipeline that ingests from Kafka, processes with Spark, and stores result in Delta using event-time windows and watermarks.

Operate a production-oriented dashboard with refresh policies, monitoring, and failure recovery.

Build and schedule streaming and batch-adjacent workflows using a modern orchestrator, such as Airflow or Prefect.

IImplement reliability patterns like idempotence, checkpointing, DLQs, and backfills for fault-tolerant and exactly-once-ish processing.

Design multi-region recovery strategies (mirroring/replication) and run playbooks to restore pipelines after partial or regional failures.

Explain CDC fundamentals (binlog/WAL) and schema evolution strategies.

Configure a Schema Registry pipeline locally using Debezium and Kafka.

Use streaming SQL (Flink/ksqlDB) to map, cast, and merge divergent schemas into a canonical model.