Pavan Kumar Regalla, AI Engineer.

Open to AI Engineer roles across the U.S. — on-site, hybrid, or remote

Pavan Kumar Regalla

AI Engineer

I build AI systems to production standards: RAG retrieval with grounded, cited answers, ML anomaly detection and forecasting — on top of tested data pipelines and lakehouse platforms.

Professional profile

Pipelines you can trust, numbers you can use.

Most of my career — more than five years now — has been about making data trustworthy enough to build on. I started in data engineering at Deloitte, then AWS, where I build and run the pipelines behind service analytics. That work is what pulled me toward generative AI and retrieval: I build RAG systems and measure them, and the reliable batch and streaming pipelines, data-quality controls, and well-modeled datasets those systems run on.

Core stack: Python, embeddings and vector search, RAG and hybrid retrieval — standing on PySpark, Kafka, Databricks, Delta Lake and AWS, from streaming ingestion and lakehouse modeling to ML scoring for anomaly detection and forecasting. Working knowledge of LangChain and Amazon Bedrock; Azure OpenAI through AI-102 certification.

Featured projects

Case studies, measured.

Every metric below comes from an executed run in the linked public repository — code, tests and documentation included.

AI Engineering · RAG · Grounded QA

Cloud Analytics Knowledge Search — RAG & Hybrid Retrieval

Hybrid TF-IDF + dense retrieval over a synthetic enterprise corpus, with extractive answers, document citations and honest abstention.

Problem
Let analysts ask questions over enterprise documents and get answers they can trust — with citations, or an honest “not supported by the knowledge base”.
Architecture
Synthetic 300-document corpus → chunking → TF-IDF sparse + dense (sentence-transformers) hybrid fusion → extractive answer generation → Streamlit UI.
Implementation
A 40-question eval set measures all three paths (TF-IDF, dense, hybrid). The extractive generator copies answer sentences from retrieved context with document citations and abstains below the evidence threshold. No LLM is called anywhere.
  • 300 synthetic enterprise documents
  • 87.5% recall@1 — hybrid leads on the top answer; dense leads the full ranking (recall@5 97.5%, MRR 0.906)
  • Grounded answers with citations; abstains on unsupported questions
  • 24 passing tests, 40-question evaluation set
Streamlit UI showing a hybrid retrieval answer with document citations
Genuine output: Streamlit UI — hybrid retrieval answer with document citations (from the repository).
  • Python
  • RAG
  • Embeddings
  • Vector Search
  • sentence-transformers
  • scikit-learn
  • Streamlit

Design note: corpus and questions are synthetic. Answers are extractive by design — grounded in retrieved documents with citations, so every answer can be verified against its source. Bedrock/LangChain are documented integration points; the Docker image was not built in this environment.

View Repository

Streaming · Fraud detection · ML

Real-Time Retail Intelligence — Kafka, Spark & ML Platform

Streaming analytics over synthetic retail events: Spark Structured Streaming into Delta Lake, with fraud-velocity alerting, anomaly scoring and demand forecasting.

Problem
Spot fraud and forecast demand on fast-moving retail events — without late or bad data corrupting the results.
Architecture
Order events → Kafka → Spark Structured Streaming (bronze → silver → gold, 10-minute watermark, checkpointing and deduplication design for exactly-once processing) → Delta Lake + alerts topic. ML models run as a batch sidecar, not inside the stream.
Implementation
run.sh orchestrates the pipeline. Deterministic velocity guardrails run inline in the streaming job for near-real-time fraud alerts; IsolationForest and GradientBoosting models train and score in batch, writing flags to anomaly_scores.csv; dbt models and tests run on the gold layer.
  • 80,628 synthetic streaming events processed
  • 100% recall on injected fraud scenarios
  • 3.02% overall alert rate
  • 83.4% lower forecast MAE than the naïve baseline
  • 24/24 checks green — CI-tested file-replay path (12 dbt + 12 pytest)
Actual dbt build output showing models and schema tests passing
Genuine output: dbt build — models plus all schema tests passing (from the repository).
  • Apache Spark
  • Structured Streaming
  • Kafka
  • Delta Lake
  • Python
  • dbt
  • Machine Learning

Limitations: all data is synthetic. The Kafka path needs Docker and was not executed in this environment — CI tests the file-replay path, which runs the same transforms.

View Repository

Batch · Lakehouse · Data Pipelines

Retail Sales Analytics — Batch Lakehouse & Data Pipeline

Bronze-to-gold lakehouse on synthetic orders: medallion architecture, star schema, KPI queries and a tested PySpark port to Databricks — with a Power BI consumption layer documented in a build guide.

Problem
Turn messy raw e-commerce data into an executive dashboard the business can actually trust.
Architecture
Raw CSVs → bronze (as-is + lineage) → silver (cleansed) → gold star schema → KPI queries → Power BI.
Implementation
etl/run_local.py runs the full medallion pipeline in pandas with 10/10 data-quality checks; notebooks/databricks_etl.py ports the same logic to PySpark/Delta Lake; powerbi/BUILD_GUIDE.md builds the .pbix in about 10 minutes.
  • 60,600 synthetic raw orders in
  • 56,420 clean fact rows out — $80.96M modeled revenue
  • 10/10 data-quality checks passing
  • 2,113 bad rows quarantined, never silently dropped
  • Bronze, silver & gold architecture with star schema and KPI queries
Medallion architecture diagram: raw CSVs to bronze, silver, gold star schema, KPI queries and Power BI
Project diagram: medallion architecture from raw files to the Power BI dashboard (from the repository).
  • Python
  • PySpark
  • Databricks
  • Delta Lake
  • Power BI
  • DAX
  • SQL

Limitations: all data is synthetic. Dashboard images are matplotlib renders of the gold data, not Power BI screenshots — the genuine .pbix is built by following the guide. The Databricks notebook path was not executed in this environment.

View Repository

All portfolio datasets and injected incidents are synthetic. Repositories state exactly which paths were executed and which are documented-but-optional.

Technical skills

The working stack.

AI and Machine Learning

  • RAG & Hybrid Retrieval
  • Embeddings & Vector Search
  • LLM Retrieval & Grounding
  • Natural Language Processing (NLP)
  • Generative AI (LangChain, Bedrock, Azure OpenAI — working knowledge, AI-102)
  • scikit-learn
  • Anomaly Detection
  • Forecasting

Data Engineering

  • Apache Spark
  • PySpark
  • Apache Kafka
  • Delta Lake
  • dbt
  • Apache Airflow
  • ETL/ELT Pipelines
  • Dimensional Modeling

Cloud and Databases

  • AWS (Glue, S3, EMR, Redshift)
  • Databricks
  • Snowflake
  • PostgreSQL
  • SQL Server
  • Spark SQL

Languages and Libraries

  • Python
  • SQL
  • pandas
  • NumPy

Data Quality and Delivery

  • Data-Quality Checks
  • dbt Tests
  • CI/CD (GitHub Actions)
  • Data Modeling
  • Data Governance

BI (supporting)

  • Power BI
  • DAX
  • Tableau

Certifications

Credentialed, not just claimed.

Additional credentials

  • Databricks Fundamentals Accreditation — Databricks

Contact

Let’s talk AI.

I’m open to AI Engineer roles — full-time, on-site, hybrid or remote, anywhere in the U.S. If you’re hiring in this space, I’d be glad to connect.