Capco - Senior Data Engineer - Python/Snowflake

Job Summary :

We are seeking a highly experienced Senior Data Engineer with 9 years of experience, specializing in building advanced Data Engineering solutions on the Snowflake/AI Cloud platform utilizing AWS. The role demands a minimum of 7 years of expertise in Python development and over 1 year of experience in building Snowflake applications using the Snowflake AI/Cortex platform (Cortex Agents, Search, and LLM). The engineer will be responsible for end-to-end data pipeline development, sophisticated data governance, cloud deployment, and implementing context enrichment techniques like RAG for LLM integration.

Job Description :

Python Development and Engineering Excellence :

- Apply 7 years of experience in developing Data Engineering and data science projects using advanced Python development techniques.

- Demonstrate a deep understanding of implementing Object-Oriented Programming (OOP) in Python, leveraging advanced data structures like Pandas data frames to write clean and maintainable engineering code.

- Implement Object-Relational Mapping (ORM) in Python using frameworks like SQLAlchemy or equivalent for robust database interactions.

- Possess a technical understanding of multi-threading concepts and concurrency implementation using Python server-side custom modules to enhance application performance.

- Utilize modern development tools, including VS Code or Anaconda, for development and proficiently use Git or Bitbucket for version control and utilize standard Python unit testing frameworks.

Snowflake AI/Cortex and Data Modeling :

- Possess mandatory 1 years of experience in building Snowflake applications utilizing the Snowflake AI/Cortex platform, specifically focusing on Cortex Agents, Cortex Search, and Cortex LLM.

- Demonstrate a strong understanding of context enrichment techniques for LLMs within Snowflake, specifically using Prompts or Retrieval-Augmented-Generation (RAG) methods.

- Good understanding of input query enrichment using Snowflake YAMLs and integrating them with LLMs within Snowflake.

- Apply required experience with different data modeling techniques necessary for designing efficient and scalable data warehouses within Snowflake.

- Demonstrate a good understanding of Relevance search principles and the process of building custom interaction applications with LLMs.

Snowflake Governance and Data Management :

- Possess a good understanding of Snowflake Account hierarchy models and best practices for defining a robust Account-role-permissions strategy.

- Demonstrate proficiency in Data Governance/Security concepts within Snowflake, including implementing Row/Column level dynamic data masking using Snowflake Tags.

- Possess good technical knowledge of Data sharing, preferably utilizing the Internal Data Marketplace and Data Exchanges for various data listings.

- Demonstrate good skills in developing applications using Snowpipe and Snowpark, effectively moving and processing data from Cloud sources like AWS S3 and handling unstructured data from data lakes.

Cloud Deployment and CI/CD :

- Exhibit proficiency in developing and deploying Python applications like Lambda on the AWS Cloud platform.

- Demonstrate strong skills in deploying web applications on AWS Cloud using Docker containers or Kubernetes, including practical experience utilizing CI/CD pipelines.

- Work effectively using Agile and SCRUM methodologies, preferably leveraging JIRA for task management.

- Be dedicated to continuously improving functionality through experimentation, performance tuning, and incorporating customer feedback.

Required Skills & Qualifications :

- Experience : Mandatory 9 years total experience, with 7 years in Python development/Data Engineering.

- Snowflake AI : 1 years of experience in building applications using Snowflake AI/Cortex platform (Agents, Search, LLM, RAG).

- Core Python : Deep understanding of OOP in Python, data structures (Pandas), concurrency, and SQLAlchemy/ORM.

- Cloud/DevOps : Good experience deploying Python apps/web apps on AWS Cloud using Docker/Kubernetes and utilizing CI/CD pipelines.

- Data Governance : Good knowledge of Snowflake Account-role-permissions strategy, data sharing, and dynamic data masking/Snowflake Tags.

- Methodology : Experience using Agile and SCRUM methodologies and Git or Bitbucket for version control.

- Soft Skills : Good analytical skills, strong problem solving ability, and capability to communicate technical concepts clearly.

Preferred Skills :

- Experience with Snowpark.

- Experience in building Snowflake native applications using Streamlit and deploying them onto AWS Cloud instances.

- Experience in implementing CI/CD pipelines within Snowflake applications.

- Experience with any application Cache implementation within Python web applications.

- Experience with duckdb and Apache arrow.