Details
Skills
Programming: Python, SQL, PySpark, Scala, Java, Bash/Shell Scripting
Data Engineering: ETL/ELT Development, Batch & Streaming Pipelines, Data Ingestion Frameworks, Data Transformation, Data Validation, Data Cleansing, Data Quality Frameworks, CDC (Change Data Capture)Metadata Driven Pipelines, Schema Evolution, Data Lineage
Big Data / Lakehouse: Apache Spark, Spark SQL, PySpark, Hadoop, Hive, HDFS, Kafka, Apache Sqoop, Delta Lake, Delta Live Tables(DLT), Databricks, Unity Catalog
Cloud Platforms: Amazon S3,EMR,Glue,Lambda,Redshift,Athena,EC2,IAM,Step Functions
Microsoft Azure: Azure Data Factory(ADF),Azure Databricks,Azure Synapse Analytics,Azure SQL Database,ADLS Gen2,Blob Storage, Key Vault
Google Cloud Platform: BigQuery, Dataproc, Cloud Composer, GCS,Pub/Sub, Dataflow, Cloud Logging, Data Catalog
Databases & Data Warehouses: Snowflake, Amazon Redshift, Google BigQuery, Azure SQL,SQL Server,Oracle,MySQL,DB2,Cassandra,MongoDB,Hbase
Data Modeling / Semantic Layer: Dimensional Data Modeling, Star Schema, Snowflake Schema, Medallion Architecture, Semantic Layer Design, Data Governance, Row-Level Security, Master Data Management, Metadata Management
Orchestration / DevOps: Apache Airflow, Cloud Composer, Azure Data Factory, Databricks workflows, Terraform,
Azure DevOps, Jenkins, Git, GitHub, Bitbucket, MavenCI/CD Pipelines
AI / Machine Learning: LLM Data Platforms, Prompt Engineering, Retrieval-Augmented Generation(RAG),Feature Engineering ,Natural Language Query Interfaces, AI Data Preparation, Model-Ready Dataset Development
API Development: FastAPI, REST APIs, OAuth, API Integration, JSON Processing
Development Methodologies: Agile, Scrum, SAFe Agile, SDLC, Test-Driven Development(TDD),Code Review, Production Support
Reporting & Visualization: Power BI,Tableau
Containers & Monitoring: Docker, Kubernetes, Cloud Logging, Monitoring, Performance Tuning, Cluster Optimization, LA Monitoring
About
• Senior Data Engineer with 6+ years of experience designing, developing, and supporting scalable cloud-based data engineering solutions using Python, SQL, PySpark, Apache Spark, Azure, AWS, and modern data platforms to deliver reliable data pipelines and analytical solutions for enterprise reporting and business intelligence requirements.
• Experienced in developing end-to-end ETL/ELT pipelines using Azure Data Factory, Azure Databricks, Azure Data Lake Storage, AWS Glue, and cloud storage technologies to ingest, transform, validate, and integrate structured and unstructured data from multiple sources into centralized analytical environments.
• Strong understanding of modern lakehouse architecture, including Medallion Architecture with Bronze, Silver, and Gold layers, Delta Lake, reusable data processing frameworks, and incremental loading strategies to improve data quality, traceability, scalability, and maintainability across enterprise data platforms.
• Proficient in Python, advanced SQL, PySpark, and Apache Spark for developing complex data transformations, implementing business rules, processing large datasets, and building scalable data ingestion frameworks, with experience applying metadata-driven patterns to standardize processing logic and reduce repetitive development effort.
• Experienced in data modeling and analytical dataset development, including source-to-target mappings, dimensional modeling, data cleansing, aggregations, schema validation, reconciliation, and structured transformation workflows to support efficient querying, consistent reporting, and reliable downstream data consumption.
• Skilled in data pipeline optimization and production support, applying Spark performance tuning, partitioning, caching, shuffle optimization, efficient DataFrame transformations, monitoring, root-cause analysis, and exception handling to improve execution efficiency, maintain data quality, and support reliable pipeline operations.
• Experienced in cloud engineering and software delivery practices using Git, Jenkins, CI/CD pipelines, Apache Airflow, Azure Data Factory, and cloud-native orchestration services, supporting version control, automated deployments, change management, technical documentation, and collaboration across engineering and infrastructure teams.
• Collaborative data engineering professional with experience working with developers, architects, analysts, infrastructure teams, and business stakeholders to translate requirements into technical solutions, troubleshoot operational issues, document architecture and implementation decisions, and continuously improve data platform performance, security, cost efficiency, and service reliability.
Profile Photo
