Experience: 6–8 Years
Location: Hyderabad
Employment Type: Contract
Open Positions: 1
Budget : Open
Duration: 1 year
Job Summary
We are seeking a highly skilled Lead Data Engineer with 6–8 years of experience in designing and building
scalable, cloud-native data platforms and high-performance data pipelines. The ideal candidate will have
expertise in developing batch and real-time data processing systems that support analytics, AI/ML
initiatives, and enterprise data solutions across domains such as Banking, FinTech, Consulting, and SaaS.
This role requires strong technical leadership, hands-on development capabilities, and the ability to
collaborate with cross-functional teams to deliver reliable and scalable data solutions.
Key Responsibilities
Data Engineering & Platform Development
Design, develop, and maintain end-to-end data pipelines for batch and real-time processing.
Build scalable ETL/ELT frameworks using Python and SQL.
Implement and manage workflow orchestration using tools such as Apache Airflow.
Design efficient data models, schemas, and transformation layers to support analytics and
downstream applications.
Develop data ingestion pipelines from various sources, including:
o APIs
o Relational and NoSQL Databases
o Event Streams
o External Systems
Streaming & Performance Optimization
Design and optimize real-time data streaming solutions using platforms such as Kafka.
Enhance pipeline performance for:
o High Throughput
o Low Latency
o Cost Efficiency
o Reliability
Implement monitoring, alerting, and logging mechanisms to ensure platform stability and SLA
compliance.
Cloud, DevOps & Engineering Best Practices
Develop and manage data solutions primarily on Microsoft Azure.
Leverage AWS or GCP experience where applicable.
Utilize Docker and CI/CD pipelines for deployment automation and version control.
Follow software engineering best practices, including:
o Code Reviews
o Automated Testing
o Documentation Standards
o Version Control Management
Ensure data quality, governance, validation, and compliance across all data pipelines.
AI & Advanced Data Engineering
Support the creation of AI/ML-ready datasets and feature engineering pipelines.
Build and maintain Feature Stores and MLOps workflows.
Develop or integrate AI-enabled solutions, including:
o LLM-based Data Pipelines
o Document Processing Workflows
o ETL Automation
o Retrieval-Augmented Generation (RAG) Architectures
Collaborate closely with Data Scientists, ML Engineers, and Analytics teams to enable AI-driven
business outcomes.
Leadership & Collaboration
Translate business requirements into scalable technical solutions.
Partner with Product, Analytics, Engineering, and Business stakeholders.
Mentor junior engineers and provide technical guidance.
Participate in architecture discussions and contribute to platform design decisions.
Drive best practices and continuous improvement initiatives across the data engineering team.
Required Skills & Experience
Must-Have Skills
5–8+ years of experience in Data Engineering.
Strong programming expertise in Python.
Advanced SQL skills and experience with complex data transformations.
Hands-on experience building ETL/ELT pipelines.
Experience with cloud platforms:
o Microsoft Azure (Preferred)
o AWS
o GCP
Experience with real-time data streaming technologies such as Kafka.
Strong experience with Apache Airflow or similar orchestration tools.
Solid understanding of:
o Data Modeling
o Data Warehousing Concepts
o Data Architecture Principles
Preferred Skills
Data Platforms & Technologies
Snowflake
BigQuery
Amazon Redshift
Delta Lake
dbt
Apache Spark / PySpark
AI & Machine Learning
MLOps
Feature Stores
MLflow
Large Language Models (LLMs)
Generative AI
Retrieval-Augmented Generation (RAG)
Domain Experience
Banking
FinTech
Consulting
Enterprise SaaS
Leadership
Prior experience in technical leadership roles.
Client-facing stakeholder management experience.
Experience driving architecture and solution design discussions.