Yashwanth Reddy

About

Data engineer at Microsoft who owned an end-to-end subscription analytics platform processing 7TB+ daily across 40+ pipelines, combining ADF batch ingestion with Kafka/Spark streaming and rigorous Great Expectations quality gates. Built a Fabric-based self-service ingestion platform with CI/CD and observability, plus a Databricks feature store serving near-real-time ML inference with Delta Lake reliability and versioning.

Key Strengths

• Owned end-to-end subscription analytics pipeline processing 7TB+ per day from 15+ sources
• Built batch + real-time ingestion (ADF + Kafka/Spark Structured Streaming) with <2 minute end-to-end streaming latency
• Implemented Great Expectations data quality gates with quarantine routing to prevent bad data reaching 10+ downstream teams
• Designed Synapse star schema models and performance tuning (partition pruning/broadcast joins) reducing processing latency by 40%
• Maintained 98% production uptime across 40+ pipelines
• Built near-real-time feature store pipeline serving ML inference with Delta Lake ACID guarantees and schema enforcement
• Enabled backward compatibility via Delta time travel and domain-specific gold tables for cleaner schema evolution
• Delivered an ambiguous, early-stage self-service ingestion platform on Fabric with CI/CD (Azure DevOps) and day-one observability/SLAs
• Strong external data ingestion resilience patterns: retries, dead-letter queues, schema drift detection, CDC-based backfills (Snowflake streams/tasks)

Experience

Education

Skills

Languages

Certifications