Description
- Lead a team of data platform engineers in the design, development, and implementation of data engineering solutions.
- Design, implement, and manage data ingestion pipelines to collect and ingest data from various sources into the cloud data platform. This might involve using tools like Apache Kafka, AWS Kinesis, or Azure Event Hubs.
- Develop and maintain data transformation processes to clean, enrich, and structure data for analytics and reporting purposes. This often involves using ETL (Extract, Transform, Load) tools or scripting languages like Python.
- Implement and enforce data security measures, including encryption, access controls, and compliance with data privacy regulations such as GDPR or HIPAA.
- Monitor and optimize the performance of the cloud data platform to ensure efficient data processing and query performance. This may involve adjusting configurations, indexing, and caching.
- Create and maintain automation scripts and workflows to manage data pipelines, backups, and routine tasks. Infrastructure as Code (IAC) tools like Terraform or AWS CloudFormation may be used.
- Set up monitoring and alerting systems to proactively identify and address issues with data pipelines, platform health, and data quality. Tools like Prometheus, Grafana, or cloud-native monitoring solutions can be employed.
- Document data platform architecture, processes, and best practices. Collaborate with data engineers, data scientists, and other stakeholders to understand their requirements and ensure the data platform meets their needs.
- Gather requirements from stakeholders and translate them into technical specifications.
- Design and implement data pipelines to collect, transform, and load data into data warehouses and data lakes.
- Maintain and optimize data systems to ensure high availability and performance.
- Work with other engineering teams to integrate data engineering solutions with other systems.
- Stay up to date on the latest data engineering technologies and trends.
- Assists with the data platform blueprint and design, encompassing the relevant data platform components.
- Collaborates with the Integration Architects and Data Architects to ensure cohesive integration between systems and data models.
Requirements
- Bachelor Degree in Computer Science, Software Engineering or related major from a reputable university.
- Minimum 10 years of relevant experience.
- High degree of (hands-on) technical guidance & vision with broad subject matter expertise in cloud, data and security technologies, specifically AWS, Azure, GCP, Oracle, and SIEM platforms (Splunk ES and Sentinel).
- Experience with Terraform
- Extensive experience and expertise across the Splunk platform, its architecture and distributed components including premium apps (ITSI, Enterprise Security, Phantom, etc…)
- Experience with data streaming technologies
- Experience of agile methodologies and associated toolsets and code repositories
- Coding experience using languages such as Python and/or Bash
- Experience working with CI/CD, Infrastructure-as-Code, and other automation tools
- Excellent analytical and problem-solving skills and Strong interpersonal skills
| Created on | 25 Oct 2023 |
| Last updated on | 29 Oct 2023 |
| You applied on | |
Apply Now 