Pavan Kumar Regalla, Data Engineer and Data Analyst.

Jersey City, NJ — Open to full-time roles across the U.S.

Pavan Kumar Regalla

Data Engineer | Data Analyst

I build tested data pipelines, analytics platforms and decision-ready reporting across streaming, batch and business-intelligence workloads.

View Projects GitHub LinkedIn Contact
Download Data Engineer Resume Download Data Analyst Resume

Professional profile

Pipelines you can trust, numbers you can use.

Data Engineer | Data Analyst with nearly five years across data engineering and analytics — currently at AWS, previously at Deloitte. I build reliable pipelines, enforce data-quality controls, and ship dashboards and analytics-ready datasets that teams actually make decisions on.

Core stack: Python, SQL, PySpark, Databricks, Kafka, AWS, and Power BI — from streaming ingestion and lakehouse modeling to the executive dashboard.

Featured projects

Case studies, measured.

Every metric below comes from an executed run in the linked public repository — code, tests and documentation included.

Streaming · Fraud detection · ML

Real-Time Retail Intelligence — Kafka, Spark & ML Platform

Streaming analytics over synthetic retail events: Spark Structured Streaming into Delta Lake, with fraud-velocity alerting, anomaly scoring and demand forecasting.

Problem
Spot fraud and forecast demand on fast-moving retail events — without late or bad data corrupting the results.
Architecture
Order events → Kafka → Spark Structured Streaming (bronze → silver → gold, 10-minute watermark, checkpointing and deduplication design for exactly-once processing) → Delta Lake + alerts topic. ML models run as a batch sidecar, not inside the stream.
Implementation
run.sh orchestrates the pipeline. Deterministic velocity guardrails run inline in the streaming job for near-real-time fraud alerts; IsolationForest and GradientBoosting models train and score in batch, writing flags to anomaly_scores.csv; dbt models and tests run on the gold layer.
  • 80,628 synthetic streaming events processed
  • 100% recall on injected fraud scenarios
  • 3.02% overall alert rate
  • 83.4% lower forecast MAE than the naïve baseline
  • 24/24 checks green — CI-tested file-replay path (12 dbt + 12 pytest)
Actual dbt build output showing models and schema tests passing
Genuine output: dbt build — models plus all schema tests passing (from the repository).
  • Apache Spark
  • Structured Streaming
  • Kafka
  • Delta Lake
  • Python
  • dbt
  • Machine Learning

Limitations: all data is synthetic. The Kafka path needs Docker and was not executed in this environment — CI tests the file-replay path, which runs the same transforms.

View Repository

Batch · Lakehouse · BI

Retail Sales Analytics — Batch Lakehouse & Power BI Pipeline

Bronze-to-gold lakehouse on synthetic orders: medallion architecture, star schema, KPI queries and a click-by-click Power BI build guide.

Problem
Turn messy raw e-commerce data into an executive dashboard the business can actually trust.
Architecture
Raw CSVs → bronze (as-is + lineage) → silver (cleansed) → gold star schema → KPI queries → Power BI.
Implementation
etl/run_local.py runs the full medallion pipeline in pandas with 10/10 data-quality checks; notebooks/databricks_etl.py ports the same logic to PySpark/Delta Lake; powerbi/BUILD_GUIDE.md builds the .pbix in about 10 minutes.
  • 60,600 synthetic raw orders in
  • 56,420 clean fact rows out — $80.96M modeled revenue
  • 10/10 data-quality checks passing
  • 2,113 bad rows quarantined, never silently dropped
  • Bronze, silver & gold architecture with star schema and KPI queries
Medallion architecture diagram: raw CSVs to bronze, silver, gold star schema, KPI queries and Power BI
Project diagram: medallion architecture from raw files to the Power BI dashboard (from the repository).
  • Python
  • PySpark
  • Databricks
  • Delta Lake
  • Power BI
  • DAX
  • SQL

Limitations: all data is synthetic. Dashboard images are matplotlib renders of the gold data, not Power BI screenshots — the genuine .pbix is built by following the guide. The Databricks notebook path was not executed in this environment.

View Repository

Search · Retrieval · Grounded QA

Cloud Analytics Knowledge Search — Hybrid Retrieval & Grounded QA

Hybrid TF-IDF + dense retrieval over a synthetic enterprise corpus, with extractive answers, document citations and honest abstention.

Problem
Let analysts ask questions over enterprise documents and get answers they can trust — with citations, or an honest “not supported by the knowledge base”.
Architecture
Synthetic 300-document corpus → chunking → TF-IDF sparse + dense (sentence-transformers) hybrid fusion → extractive answer generation → Streamlit UI.
Implementation
A 40-question eval set measures all three paths (TF-IDF, dense, hybrid). The extractive generator copies answer sentences from retrieved context with document citations and abstains below the evidence threshold. No LLM is called anywhere.
  • 300 synthetic enterprise documents
  • 87.5% recall@1 — hybrid beats TF-IDF (75.0%) and dense (85.0%) alone
  • Grounded answers with citations; abstains on unsupported questions
  • 24 passing tests, 40-question evaluation set
Streamlit UI showing a hybrid retrieval answer with document citations
Genuine output: Streamlit UI — hybrid retrieval answer with document citations (from the repository).
  • Python
  • sentence-transformers
  • scikit-learn
  • Streamlit
  • Information Retrieval

Limitations: corpus and questions are synthetic. No LLM was executed — answers are extractive, not generated. Bedrock/LangChain are optional integration points, not run here; the Docker image was not built in this environment.

View Repository

All portfolio datasets and injected incidents are synthetic. Repositories state exactly which paths were executed and which are documented-but-optional.

Technical skills

The working stack.

Data Engineering

  • Apache Spark
  • PySpark
  • Apache Kafka
  • Delta Lake
  • dbt
  • Apache Airflow

Cloud and Databases

  • AWS (Glue, S3)
  • Azure Databricks
  • PostgreSQL
  • SQL Server
  • Spark SQL

Analytics and BI

  • Power BI
  • DAX
  • Tableau
  • Dimensional Modeling
  • KPI Design

Languages and Libraries

  • Python
  • SQL
  • pandas
  • scikit-learn

AI and Machine Learning

  • Anomaly Detection
  • Forecasting
  • Dense Retrieval (MiniLM)
  • Streamlit

Data Quality and Delivery

  • Data-Quality Checks
  • dbt Tests
  • CI/CD (GitHub Actions)
  • Data Modeling

Certifications

Credentialed, not just claimed.

AWS Certified Data Engineer – Associate

Amazon Web Services

Microsoft Certified: Power BI Data Analyst Associate

Microsoft · PL-300

Databricks Certified Data Analyst Associate

Databricks

Additional credentials

  • Microsoft Certified: Azure AI Engineer Associate (AI-102) — Microsoft
  • Databricks Fundamentals Accreditation — Databricks

Contact

Let’s talk data.

I’m open to Data Engineer, Data Analyst, Analytics Engineer and BI Engineer opportunities — full-time, on-site, hybrid or remote. If you’re hiring or working in this space, I’d be glad to connect.

Email Me Connect on LinkedIn