Position Overview:
Data Analytics & AI department at RCG is looking for a highly skilled and experienced Data Platform Lead to design, build, and manage our data platform on Azure and Databricks. The successful candidate will be responsible for managing the end-to-end data platform development process, including data ingestion, processing, storage, and analysis. This role requires a strong technical background in data platform architecture, cloud computing, and big data technologies.
This role carries primary responsibilities as a Platform Lead / Platform Engineer and Azure Platform Admin, while also contributing to ML Ops.
Responsibilities:
Design, build and configure a highly scalable and robust data platform on Azure and Databricks.
Data platform shuld have capabilities of seamless integration & extensibility of existing cloud services to any new program/project initiative.
Develop and implement data ingestion and processing pipelines using Azure services such as Azure Data Factory, Azure Stream Analytics, and Databricks
Collaborate with cross-functional teams to ensure data platform meets business requirements and industry best practices
Manage cloud infrastructure on Azure platform and maintain Azure environment to ensure maxium efficiency, scalability and security.
Develop & implement cloud automation and orchestration using tools auch as Azure resource manager, Azure DevOps etc.
Collaborate with offshore team members with development & deployment activities, per project requirements.
Required Skills:
Platform Engineering:
Proficiency in Python, Scala, or SQL.
Expertise in Apache Spark, Hadoop, or Apache Kafka.
Cloud experience in Azure (preferred) or AWS.
Deployment and management of Databricks clusters.
Mastery of Azure Data Factory (ADF) for integration.
Expertise with Databricks workspace, runtime, and Databricks Connect.
Understanding of Databricks Unity Catalog.
Knowledge of Delta Live Tables and Delta Engine.
Familiarity with Delta Lake architecture.
Strong SQL skills.
Azure Platform Administration:
Management of Azure Cloud Services, including VMs, Storage Accounts, and networks.
Azure Security Management for policies and access controls.
Knowledge of Azure Active Directory and network security.
Performance monitoring, resource optimization, backup, and recovery.
High Availability (HA), Disaster Recovery (DR), and failover expertise.
Azure Services/Resources cost optimization and monitoring.
Azure DevOps and CI/CD proficiency.
Configure Azure resources, native services, security, and monitoring.
Design platform architecture with Azure Cloud Services and Databricks Delta Lake.
Create and publish guidelines for Data Integration services.
Participate in platform solution architecture, including data architecture.
Implement Databricks workspace, runtime, and Databricks Connect.
Configure Kafka integration services for real-time streaming and batch processing.
Enable Databricks Unity Catalog (UC) for lineage and governance.
Design and implement ML Ops for model deployments and inference pipelines.
Configure Azure DevOps, CI/CD for seamless deployment.
Implement monitoring and logging for Azure Service Groups and system metrics.
Ensure platform security, compliance, and data privacy.
ML Ops:
Proficient in Python, R, or Java for ML algorithms.
Experience with TensorFlow, PyTorch, and other ML frameworks.
Efficient code for data processing, model training, and deployment.
Cloud platform experience, preferably Azure.
Familiarity with Azure ML Services and Azure DevOps.
Strong data analytics skills for model training.
Qualifications:
Bachelor's degree in Computer Science, Information Technology, or related field.
7+ years of experience in cloud infrastructure management, Azure services, and infrastructure architecture
Strong understanding of Azure services such as Virtual Machines, Virtual Networks, Storage Accounts, and App Services
Experience in monitoring and optimizing Azure infrastructure for performance and efficiency
Proven experience and expertise in the respective role's technical skillsets.
Strong problem-solving skills and ability to work collaboratively.
Excellent communication skills for effective cross-team collaboration.
If you're passionate about leveraging data for informed decisions and thrive in a dynamic, collaborative environment, we invite you to join our team.
#LI-DW1
#J-18808-Ljbffr
