Vetted Apache Spark Professionals

“AI/ML engineer with CVS Health experience deploying production LLM systems in regulated healthcare settings, including a large-scale RAG solution (1M+ documents) built for compliance-grade, auditable policy/regulatory Q&A with strong anti-hallucination controls. Also delivered an NLP summarization system for physician notes/case narratives by partnering closely with non-technical care operations stakeholders and iterating via prototypes, dashboards, and feedback loops.”

“ML/NLP practitioner who built a retrieval-augmented generation (RAG) system for large financial and operational document sets using Sentence-Transformers (all-mpnet-base-v2) and a vector DB (e.g., Pinecone), with a strong focus on retrieval evaluation and chunking strategy optimization. Experienced in entity resolution (rules + embedding similarity with type-specific thresholds) and in productionizing scalable Python data workflows using Airflow/Dagster and Spark.”

“Built a secure, on-prem/private GPT assistant to replace manual SharePoint-style search across thousands of policies/SOPs/engineering docs, using a production RAG stack (LangChain/LangGraph, FAISS/Chroma, PyMuPDF+OCR, vLLM). Implemented layout-aware ingestion (including table-to-JSON) and a multi-agent retrieval/generation/verification workflow with strong observability and compliance guardrails, delivering ~70% reduction in search time.”

“Built and deployed an end-to-end product that brings a research-paper approach into production for large-scale time-series clustering, with attention to partitioning, latency, and scalability. Also designed a Python-based backend validation service (comparing outputs to database ground truths) and handled production reliability issues by reproducing dataset-specific crashes and hardening corner-case behavior with client-friendly errors.”

“Data engineer focused on AWS-based enterprise data platforms, owning end-to-end pipelines from multi-source batch/stream ingestion (Glue/Kinesis/StreamSets/Airflow) through PySpark transformations into curated datasets for Redshift/Snowflake. Emphasizes production reliability with strong monitoring/observability and data quality gates, and reports ~30% performance improvement plus improved SLAs and latency after optimization.”

“Data engineer with healthcare domain experience who has owned end-to-end pipelines and APIs at UnitedHealth Group, processing ~8M records per batch. Strong focus on data quality (multi-layer validation), reliability (monitoring/logging, retries/idempotency), and performance (Spark/SQL tuning, caching), with experience standing up early-stage systems using Python, Docker, and CI/CD.”

“Data engineer focused on reliable, production-grade pipelines and data services: has owned end-to-end ingestion-to-serving workflows processing millions of records/day, using Airflow, Python/SQL, and PySpark. Demonstrates strong operational rigor (monitoring, retries, idempotency, backfills) and measurable outcomes (98% stability, ~30% faster processing), plus experience exposing curated warehouse data via versioned REST APIs.”

“Data engineer in a banking context who has owned end-to-end Azure lakehouse pipelines ingesting financial/vendor data from APIs, Azure SQL, and flat files into Databricks/Delta (bronze-silver-gold). Emphasizes production reliability via schema-drift validation, data quality controls, monitoring/alerting, retries/checkpointing, and Spark/Delta performance tuning, with outputs served to BI/reporting teams (e.g., Tableau).”