Job Description

READ CAREFULLY BEFORE APPLYING | URGENT HIRING

ROLE: Lead Data Engineer – Hands-On

Client: MetLife

Location: Cary, NC — On-site / Hybrid

Experience: 12–18 Years

Employment Type: Full-Time

Eligibility: US CITIZENS ONLY

ABOUT THE ENGAGEMENT

The engagement involves building a centralized, AI-first enterprise Data Hub for a global insurance and financial services organization using Azure Databricks .

The platform ingests 150+ inbound data feeds and distributes data to 35+ downstream systems , leveraging a Medallion Architecture — Bronze / Silver / Gold .

AI is embedded across ingestion, canonical mapping, data quality, reconciliation, and business-user access.

This is a senior hands-on technical leadership role . The selected candidate will own the end-to-end technical design of the data and AI layers , build reference implementations, mentor engineering teams, and continue writing/reviewing production-grade Python, Scala, and PySpark code every week .

⚠️ Candidates who have NOT written or reviewed production code within the past year will NOT be considered.

WHAT YOU WILL OWN

Data Platform Architecture & Engineering

• Design and implement enterprise-scale Lakehouse architecture.
• Define Bronze / Silver / Gold data contracts.
• Design ADLS Gen2 zone layouts.
• Delta Lake table design, partitioning, schema evolution and retention.
• Build scalable and reusable data platform components.

Metadata-Driven Ingestion

• Design parameterized ingestion frameworks for:
• Batch files
• Database extracts
• CDC feeds
• Streaming data
• Implement streaming pipelines using Azure Event Hubs / Kafka / Spark Structured Streaming .
• Build schema inference, data profiling, metadata and lineage capabilities.

Spark & Data Engineering

• Develop production-grade Python, Scala and PySpark solutions.
• Establish coding and testing standards.
• Conduct PR/code reviews.
• Debug production incidents.
• Optimize Spark workloads, clusters and pipelines.
• Implement performance and cost optimization strategies.

Databricks & Azure

• Azure Databricks
• Delta Lake
• Unity Catalog
• Databricks Jobs & Workflows
• Cluster and pool management
• Model Serving
• ADLS Gen2
• Azure Data Factory
• Azure Event Hubs
• Azure Monitor
• Log Analytics

CI/CD & Infrastructure as Code

• Implement CI/CD for Databricks and ADF using Azure DevOps .
• Work with Databricks Asset Bundles .
• Implement Infrastructure as Code using Terraform .
• Automate data pipeline testing and deployments.
• Establish observability and operational monitoring.

AI / LLM ENGINEERING

A major component of this role is designing and delivering production-grade AI/LLM solutions .

You will work on:

• AI-augmented ingestion.
• AI-assisted canonical mapping.
• Auto-generated bridge documentation.
• DML and canonical table generation.
• Source-to-canonical mapping with human review gates.
• AI-driven data quality.
• Data drift and schema drift detection.
• Volume anomaly detection.
• Automated reconciliation.
• Synthetic privacy-preserving test data generation.
• Enterprise semantic layers.
• Knowledge graphs.
• GPT-powered conversational interfaces.
• Text-to-SQL.
• Semantic-layer retrieval.
• Row-level and column-level security.

LLM / GENAI MUST-HAVES

Candidates must have 3+ years of hands-on experience designing and shipping LLM-based systems in production , including:

• RAG pipelines
• Agentic AI workflows
• Tool-calling workflows
• Chunking strategies
• Embedding strategies
• Vector retrieval
• Hybrid retrieval
• Prompt engineering
• LLM evaluation
• Golden datasets
• Regression test suites
• Accuracy tracking
• Hallucination monitoring
• Human-in-the-loop feedback

Hands-on experience with at least one of:

• LangChain
• LlamaIndex
• LangGraph

Provider/platform experience:

• Azure OpenAI
• OpenAI
• Databricks Model Serving

SECURITY, GOVERNANCE & DATA PROTECTION

Strong enterprise security and governance experience is required across:

• Unity Catalog
• Data lineage
• Access control
• PII standards
• Entra ID
• Managed Identities
• RBAC
• POSIX ACLs
• Azure Key Vault
• Private Endpoints
• Data privacy and protection
• Enterprise data governance

The candidate will also participate in:

• Architecture Review Boards
• AI Governance Forums
• Technical design reviews
• Enterprise governance discussions

MUST-HAVE TECHNICAL SKILLS

✅ 12–18 years of total experience in Data Engineering / Data Platform delivery

✅ Expert-level Python

✅ Expert-level Scala

✅ Expert-level PySpark / Spark

✅ Strong SQL

✅ Data modelling — dimensional & normalized

✅ Data contracts & schema design

✅ Azure Databricks

✅ Delta Lake

✅ Unity Catalog

✅ Azure ADLS Gen2

✅ Azure Data Factory

✅ Azure Event Hubs

✅ Kafka / Spark Structured Streaming

✅ Databricks Jobs & Workflows

✅ Spark performance tuning

✅ Azure DevOps

✅ Terraform

✅ Databricks Asset Bundles

✅ CI/CD

✅ Metadata-driven frameworks

✅ Data profiling & lineage

✅ Enterprise Lakehouse / Medallion Architecture

✅ Azure security & governance

✅ 3+ years production LLM/GenAI engineering

✅ RAG / Agentic AI / Tool Calling

✅ LangChain / LlamaIndex / LangGraph

✅ Azure OpenAI / OpenAI / Databricks Model Serving

⭐ STRONGLY PREFERRED

• Knowledge graphs & ontologies
• RDF / SPARQL
• Neo4j
• Graph modelling over Lakehouse
• Text-to-SQL
• Enterprise semantic layers
• ML-based anomaly detection
• Time-series / transactional financial data
• Financial Services / Insurance domain
• Finance Close
• GL / General Ledger
• Subledger
• Reconciliation
• Actuarial data
• LLMOps / MLOps
• Model & prompt versioning
• LLM cost governance
• AI observability
• dbt
• Great Expectations
• Workday
• Workday Prism
• Accounting Center

CERTIFICATIONS — PLUS

• Databricks Data Engineer Professional
• Azure DP-203
• Azure DP-700
• Azure AZ-305

IDEAL CANDIDATE PROFILE

We are looking for a hands-on Lead Data Engineer / Data Architect / Principal Data Engineer who can operate at both architecture and engineering levels .

You should be comfortable designing the architecture, making technical decisions, writing production code, reviewing code, troubleshooting Spark workloads, mentoring engineers, and presenting architecture decisions to senior technical and business stakeholders.

⚠️ This is NOT a pure Data Architect, Solution Architect, Engineering Manager, or AI Strategy role.

⚠️ Recent hands-on production coding experience is mandatory.

⚠️ US CITIZENS ONLY.

#Nazztec #NazztecHiring #JoinNazztec #Hiring #UrgentHiring #LeadDataEngineer #DataEngineer #SeniorDataEngineer #PrincipalDataEngineer #DataArchitect #DataEngineering #DataPlatform #DataPlatformEngineer #BigData #BigDataEngineering #Azure #MicrosoftAzure #AzureDatabricks #Databricks #DeltaLake #UnityCatalog #AzureDataFactory #ADF #ADLS #ADLSGen2 #AzureEventHubs #Kafka #Spark #ApacheSpark #PySpark #Scala #Python #SQL #DataLake #Lakehouse #LakehouseArchitecture #MedallionArchitecture #BronzeSilverGold #DataArchitecture #DataModeling #DataEngineeringJobs #AzureJobs #DatabricksJobs #AI #ArtificialIntelligence #GenerativeAI #GenAI #LLM #LLMEngineering #LLMEngineer #LLMJobs #RAG #RAGPipeline #RetrievalAugmentedGeneration #AgenticAI #AgenticWorkflows #ToolCalling #LangChain #LlamaIndex #LangGraph #AzureOpenAI #OpenAI #DatabricksModelServing #PromptEngineering #VectorDatabase #VectorSearch #HybridSearch #Embeddings #SemanticSearch #TextToSQL #SemanticLayer #KnowledgeGraph #Neo4j #RDF #SPARQL #MLOps #LLMOps #AIEngineering #AIEngineer #MachineLearning #AnomalyDetection #DataQuality #DataGovernance #DataSecurity #DataLineage #DataCatalog #Metadata #MetadataDriven #DataContracts #SchemaEvolution #DataProfiling #DataObservability #DataOps #DevOps #AzureDevOps #Terraform #InfrastructureAsCode #IaC #CICD #ContinuousIntegration #ContinuousDeployment #SparkTuning #PerformanceTuning #CloudDataEngineering #CloudArchitecture #EnterpriseData #EnterpriseArchitecture #FinancialServices #Insurance #InsuranceTechnology #FinTech #MetLife #CaryNC #NorthCarolinaJobs #USJobs #USAJobs #USCitizen #USCitizenJobs #TechJobs #ITJobs #TechnologyJobs #DataScience #CloudJobs #DataProfessionals #TechHiring #ITHiring #DataHiring #AIJobs #RemoteJobs #HybridJobs #OnsiteJobs

Job Tags

Similar Jobs