Divy Limbachiya

About

Data Engineer with experience across AWS, Azure, Snowflake, and Databricks. Built ELT and streaming pipelines, implemented data governance with row level security and IAM roles, and enabled analytics with dbt and BI tools. Interested in ML applications in production data systems.

Skills

Tech Toolbox

• Airflow
• Docker
• Kubernetes
• dbt
• AWS
• Azure
• GCP
• Databricks
• Power BI
• Tableau
• Snowflake
• Prefect
• Lambda
• S3
• ADF

Projects

Yahoo Finance to Snowflake with Airflow

Automated ELT pipeline that downloads AAPL price history from Yahoo Finance, saves CSV to S3, loads Snowflake, and triggers Metabase to rescan dashboards. Orchestrated with Apache Airflow.

• Stack: Airflow, AWS S3, Snowflake, Python, Metabase
• Outcome: Daily refreshed dashboards

GitHub Repo

Data Solutions Architecture on AWS

Integrated S3 bronze and silver layers with Redshift Serverless for gold analytics. Automated near real time pipelines with Glue, Lambda, SQS, and SNS. Centralized metadata with Glue and enabled predictive modeling with SageMaker.

• Stack: AWS S3, Glue, Redshift, Lambda, SageMaker
• Outcome: Automated ingestion and accelerated ML insights

GitHub Repo

Streaming ELT to Snowflake

Deployed real time streaming pipelines with Confluent Kafka and Databricks. Implemented Snowpipe ingestion and built dbt models with SCD Type 2 snapshots and materialized views to support ad hoc analysis.

• Stack: Kafka, Databricks, Snowflake, dbt
• Outcome: Near real time analytics

Performance Tuning for Spark Workloads

Engineered lakehouse architecture with ACID guarantees and tuned Spark to reduce runtime. Applied partitioning and bucketing strategies to improve processing efficiency.

• Stack: Delta Lake, Spark, Databricks
• Outcome: 25 percent lower runtime

Experience

• Data Engineer, Upwork

  Remote, Sep 2024 to Present

• Data Engineering Apprenticeship, TrendyTech

  Remote, Apr 2024 to Dec 2024

• Data Analytics Engineering Intern, Adani Ports

  Ahmedabad, Sep 2022 to Nov 2022

• Data Analyst, Visat IT Solutions

  Ahmedabad, Jan 2021 to Aug 2022

Education

• Cloud Computing Technologies: George Brown College, Toronto (Jan 2024 to Dec 2024)
• Data Analytics: Durham College, Oshawa (Jan 2023 to Sep 2023)
• BSc in Mathematics, Statistics, Physics: M.S. University of Baroda, India (Jun 2018 to May 2021)

Certifications

• AWS Certified Data Engineer
• Microsoft Certified Analytics Engineer, DP 600
• Databricks Lakehouse Fundamentals

Achievements

• Alleviated data anomalies by 40 percent and resolved 80 percent of integrity discrepancies.
• Mitigated storage costs by 30 percent through archival and lifecycle policies.

Contact