Cloud Data Engineer

Introduction

Our partner is an industrial giant whose aim is to drive the world towards a sustainable future by building businesses and partnerships across various specializations ranging from energy to batteries through the help of a unique blend of knowledge and competence. They are committed to leading the world towards a more innovative, balanced and self-sustaining ecosystem through the optimal use of natural resources and products.

Tasks

The Cloud Data Engineer works on wide variety of business-critical, high-profile projects for internal customers and manufacturing locations, and helps them solve their data engineering, ML engineering problems, and leverage the ever-evolving cloud platform based on MS Azure.

• Design and build robust, scalable, secure batch and streaming analytics pipelines, data platforms, and enterprise data lake solutions 
• Cover the whole product development lifecycle starting from performing PoCs to integrate and deploy modern BI & analytics solutions into production
• Work as a technical expert alongside business process owners, product owners, DevOps, ML and infrastructure engineers to figure out how to reach business objective targets
• Represent the voice for data product developers internally and influence our service offerings and data product strategy by working with cloud platform team and other cross-functional teams
• Manage and orchestrate complex data pipelines using Azure Data Factory (ADF)
• Design, implement, and optimize data transformation processes using Python libraries (Pandas, Numpy, PySpark)
• Develop, maintain, and troubleshoot scalable data workflows on Databricks
• Utilize Azure Data Explorer (ADX) for real-time data analysis and complex queries
• Create and optimize SQL queries to extract, transform, and load data
• Automate routine data tasks with Automation Runbooks and Azure Functions
• Design and manage scalable data storage solutions using Azure Data Lake
• Write and execute unit tests to ensure the integrity and quality of data
• Utilize CI/CD pipelines using Azure DevOps for seamless integration and deployment
• Optimize Azure usage to manage and reduce costs effectively
• Ensure to work in compliance with the partner's Quality system, regulations, policies and standardized operating procedures
• Perform all other tasks upon the instructions of the superior in charge which may be necessitated by the operations of the related unit and which do not conflict with any applicable laws, statutory provisions and company rules
• Compliance with area-specific customer requirements

Expectations

• 3+ years of experience in data engineering or a similar role
• Microsoft Certified: Azure Data Engineer Associate or similar certification is a plus
• Strong problem-solving skills and attention to detail
• Excellent communication and teamwork abilities
• Master/Bachelor’s degree in Computer Science, Information Technology, or a related field
• Fluent in English is mandatory. Any other language is a plus
• Proficiency in using Azure Data Factory to create and manage data pipelines
• Proven experience in data transformation using Python (Pandas, Numpy, PySpark)
• Advanced knowledge of SQL and query optimization
• Understanding of Azure Data Explorer (ADX) for data analysis
• Hands-on experience with Databricks for big data processing and collaborative data science workflows
• Expertise in designing and managing data storage with Azure Data Lake
• Experience in automation with Runbooks and Azure Functions
• Familiarity with CI/CD pipelines using Azure DevOps
• Experience in writing and executing unit tests for data quality assurance
• Experience using GitHub for version control, collaboration, and code review

Advantageous

• Another language knowledge besides English
• Knowledge of Infrastructure as Code (IaC) tools (e.g., Terraform, ARM templates)
• Awareness of Azure pricing models and cost management practices

Employer's offer

• International environment
• Exciting projects