Senior Data Engineer

Job Description:

We are seeking a highly skilled and experienced Data Engineer to join our team. The ideal candidate will have a strong background in data engineering technologies, including Vector DB, Graph DB, Airflow, Spark, PySpark, Python, LangChain, AWS Functions, Redshift, and SSIS.

The successful candidate will be responsible for designing and developing data pipelines for Generative AI projects, ensuring seamless, high-performance data flows that efficiently support the data requirements of our cutting-edge AI initiatives.

Key Responsibilities:

• ">

• Design and develop data pipelines using a combination of technologies such as Vector DB, Graph DB, Airflow, Spark, PySpark, Python, LangChain, AWS Functions, Redshift, and SSIS.

  ">">

• Collaborate with data scientists, AI researchers, and other stakeholders to understand data requirements and translate them into effective data engineering solutions.

  ">">

• Manage movement, organization, and quality assessments of large sets of data to facilitate the creation of knowledge bases for RAG systems and model training.

  ">">

• Demonstrate familiarity with data integration services such as AWS Glue and Azure Data Factory, showcasing the ability to effectively utilize these platforms for seamless data ingestion, transformation, and orchestration across various sources and destinations.

  ">">

• Construct data warehouses and data lakes, demonstrating a strong foundation in organizing and consolidating large volumes of structured and unstructured data for efficient storage, retrieval, and analysis.

  ">">

• Optimize and maintain data pipelines to ensure high-performance, reliable, and scalable data processing.

  ">">

• Develop and implement data validation and quality assurance procedures to ensure the accuracy and consistency of the data used in Generative AI projects.

  ">">

• Monitor and troubleshoot data pipeline performance, identify bottlenecks, and implement improvements as necessary.

  ">">

• Stay current with emerging trends and technologies in the fields of data engineering, Generative AI, and related areas to ensure the continued success of our projects.

  ">">

• Collaborate with team members on documentation, knowledge sharing, and best practices for data engineering within a Generative AI context.

  ">">

• Ensure data privacy and security compliance in accordance with industry standards and regulations.

  ">">

Required Skills and Qualifications:

• ">

• Bachelors or Masters degree in Computer Science, Engineering, or a related field.

  ">">

• Strong experience with data engineering technologies, including Vector DB, Graph DB, Airflow, Spark, PySpark, Python, LangChain, AWS Functions, Redshift, and SSIS.

  ">">

• Strong understanding of data warehousing concepts, ETL processes, and data modeling.

  ">">

• Strong understanding of S3 and code-based scripting to move large volumes of data across application storage layers.

  ">">

• Familiarity with Generative AI concepts and technologies, such as GPT-4, Transformers, and other natural language processing techniques.

  ">">

• Excellent problem-solving, analytical, and critical thinking skills.

  ">">

• Strong communication and collaboration skills, with the ability to work effectively with cross-functional teams.

  ">">

Preferred Qualifications/ Skills:

• ">

• Knowledge of cloud computing platforms, such as AWS, Azure, or Google Cloud Platform.

  ">">

• Experience with big data technologies, such as Hadoop, Hive, or Presto.

  ">">

• Familiarity with machine learning frameworks, such as TensorFlow or PyTorch.

  ">">

• A continuous learning mindset and a passion for staying up-to-date with the latest advancements in data engineering and Generative AI.

  ">">

About This Role:

This is an exciting opportunity to join our team and contribute to the development of innovative Generative AI solutions. If you have a passion for data engineering and are looking for a challenging role that offers opportunities for growth and professional development, we encourage you to apply.

Contact Information:

Please note that we do not accept applications via email. To apply, please submit your resume and cover letter through our website.