Experienced Data Engineer

Job Title: Experienced Data Engineer

About the Role: We are seeking a skilled data engineer to join our team. The ideal candidate should have experience in designing and implementing data pipelines using AWS services such as Glue, EMR, Lambda, and DynamoDB.

Main Responsibilities:

• Data Engineering Solutions: Design, develop, and implement end-to-end data engineering solutions from scratch on the AWS platform.
• DynamoDB Database Design: Design and implement scalable DynamoDB databases, optimizing performance and ensuring data consistency, throughput, and storage efficiency.
• Data Requirements Gathering: Collaborate with cross-functional teams and stakeholders to gather data requirements and understand business objectives.
• Data Modelling: Architect and implement data models and schemas that align with best practices and optimize query performance.
• Data Pipelines: Build robust and scalable data pipelines to extract, transform, and load (ETL) data from diverse sources into target systems.
• Third-Party API Integration: Integrate third-party APIs to retrieve and ingest data from external systems, ensuring data accuracy and reliability.
• Data Profiling and Cleansing: Perform data profiling, data cleansing, and data quality checks to ensure accuracy and consistency.
• Database Performance Optimization: Optimize SQL queries and database performance, leveraging SQL tuning techniques and indexing strategies.
• Data Governance: Implement data governance and security measures to protect sensitive data throughout the pipeline.
• Documentation: Develop and maintain documentation for data pipelines, data flows, and data transformations.
• Tech Knowledge Update: Stay up-to-date with industry trends and emerging technologies in data engineering, AWS services, and third-party API integrations.
• Mentorship: Mentor junior team members and provide technical guidance and support as needed.
• Geospatial Data Management: Manage and analyze geospatial data, implementing solutions for storing, retrieving, and processing location-based information while integrating geospatial indexing and query optimization techniques.

Requirements:

To be considered for this role, you must have a Bachelor's degree in Computer Science or a related field, along with a minimum of 2+ years of experience in data engineering, specifically in building and maintaining data pipelines from scratch. You should also possess strong proficiency in AWS services such as Glue, EMR, Lambda, Dynamo DB, Postgres, and expertise in data modelling, schema design, relational and non-relational database concepts, SQL, SQL tuning, and data modeling best practices.

Key Skills:

The ideal candidate will have experience in designing and implementing single-table architecture in DynamoDB, optimizing data models for scalability, efficient query patterns, and minimal read/write costs. Additionally, they should be proficient in using PartiQL to query and manipulate data in DynamoDB, with a focus on optimizing queries for performance and integrating complex business logic. Other essential skills include developing and optimizing SQL queries for performance, proficiency in Python programming for data manipulation, transformation, and scripting tasks, familiarity with data warehouse concepts, OLAP, and Big Data technologies, and experience in integrating third-party APIs to retrieve and ingest data from external systems.

Personal Attributes:

The successful candidate should be a team player with good interpersonal skills and problem-solving abilities. They should also be a self-motivator and a collaborative team player, possessing excellent time management skills and the ability to assume leadership roles when necessary. Additionally, they should be agile, action-oriented, and a quick learner, with an out-of-the-box thinking approach.