Senior Data Engineer

3 weeks ago

Hyderabad, India Circuitry.ai Full time

Job Description:

Location: Hyderabad, India - Hybrid Remote (3 days a week in office preferred)

Position Overview:

As a Data Engineer focused on AI development, you will play a pivotal role in the design, development, and deployment of data infrastructure and pipelines that support artificial intelligence and machine learning initiatives within our organization. You will work closely with data scientists, machine learning engineers, and software developers to ensure that data is readily accessible, high-quality, and efficiently processed to fuel AI algorithms and models.

Responsibilities:

  • Data Pipeline Development: Design, implement, and maintain data pipelines that collect, transform, and store data from various sources, ensuring data quality and consistency. These pipelines will serve as the foundation for AI model training and deployment.
  • Data Integration: Collaborate with cross-functional teams to integrate data from diverse sources, including structured and unstructured data, into the data ecosystem. Ensure data is accessible and ready for AI development.
  • Data Cleansing and Preprocessing: Develop data preprocessing routines to clean and preprocess raw data, handling missing values, outliers, and data inconsistencies, making it suitable for machine learning algorithms. Ensures data consistency and integrity during the integration process, performing data validation and cleaning as needed.
  • Data Modelling Support: Work closely with data scientists and machine learning engineers to understand data requirements for AI model training. Assist in feature engineering and dataset creation for model development.
  • Meta store Management and Data Governance: Configure and maintain databricks meta stores to for efficient data discovery. Implement data governance practices, including data catalogs, version control, and access controls to maintain data lineage and ensure data security and compliance.
  • Performance Optimization: Continuously monitor and optimize data pipelines and infrastructure for efficiency and scalability, ensuring timely data processing for AI applications. Monitor and tune data systems, identifies and resolves performance bottlenecks, and implements caching and indexing strategies to enhance query performance.
  • Collaboration: Collaborate with cross-functional teams to understand AI project requirements and translate them into data engineering solutions. Collaborate with data scientists and analysts to optimize models and algorithms for data quality, security, and governance.
  • Documentation: Maintain comprehensive documentation of data pipelines, transformations, and processes to facilitate knowledge sharing and troubleshooting.
  • Quality Assurance: Implement and maintain data quality checks and validation processes to identify and address data anomalies or issues that may impact AI model performance. Implement data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data.
  • Learning and Development: Stay up-to-date with industry trends, emerging technologies, and best practices in data engineering and AI development. Propose and evaluate new tools and technologies that can enhance the AI development process.

Job Requirements:

Education

  • A bachelor’s degree in computer science, data science, software engineering, information systems, or related quantitative field; master’s degree preferred.

Experience

  • Minimum 4 years in data management, focusing on data integration, modelling, optimization, data quality, and related data engineering activities preferably on data bricks platform.
  • Demonstrated experience in developing and managing data warehouses and ETL/ELT pipelines for AI/ML applications.

Certifications

  • Must be Certified Data Engineering Professional/Associate from Data bricks.

Skills

  • Expertise in Databricks, including its architecture, security features, and monitoring/logging capabilities.
  • Expert-level knowledge and hands-on experience with configuring and orchestrating complex workflows.
  • Proficient in big data languages such as PySpark, SQL, and/or DBT.
  • Skilled in database technologies including SQL, NoSQL, Deltalake, and MongoDB.
  • Experience with cloud platforms like AWS or Azure, and container technologies like Docker or Kubernetes.
  • Strong understanding of ETL and ELT processes, preferably with Databricks.
  • Knowledgeable in AI/ML principles, models, tools, and deployment strategies.
  • Advanced expertise in Apache technologies like Kafka, Airflow, and Spark for creating effective data pipelines.
  • Capable of designing and implementing data solutions for AI, ML, and BI applications.
  • Proficient in programming languages like Python, and data science tools such as Scala or SQL.
  • Collaborative skills to work with diverse technical teams, guiding and educating on data products.
  • Exceptional problem-solving abilities, including troubleshooting and resolving issues in unfamiliar code or systems.
  • Excellent business insight and interpersonal skills, capable of working with senior stakeholders to drive change.
  • Ability to articulate business cases, data sources, management concepts, and analytical methods.
  • Skilled in translating technical language for executive, business, IT, and quantitative audiences.

  • Senior Azure Data Engineer

    3 weeks ago

    Hyderabad, India Enable Data Incorporated Full time

    Enable Data Incorporated is seeking a highly skilled Senior Spark Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem. As a Senior Spark Engineer, you will be responsible for overseeing all aspects...

  • Senior Azure Data Engineer

    4 weeks ago

    Hyderabad, India Enable Data Incorporated Full time

    Enable Data Incorporated is seeking a highly skilled Senior Spark Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem. As a Senior Spark Engineer, you will be responsible for overseeing all aspects...

  • Senior Azure Data Engineer

    4 weeks ago

    hyderabad, India Enable Data Incorporated Full time

    Enable Data Incorporated is seeking a highly skilled Senior Spark Engineer to join our dynamic team. As a leading provider of advanced application, data, and cloud engineering services, we leverage modern solutions to drive increased value across our clients' business ecosystem. As a Senior Spark Engineer, you will be responsible for overseeing all aspects...

  • Senior Data Engineer

    3 weeks ago

    Hyderabad, India LTIMindtree Full time

    Senior Data EngineerJob Req Id: 1248440 Bachelor’s Degree in Computer Science or a related discipline • 5+ years of applicable engineering experience• Experience in GCP, Big Query• Strong proficiency in Python with an emphasis in building data pipelines• Ability to write complex SQL to perform common types of analysis and aggregations• Experience...

  • Senior Data Engineer

    3 weeks ago

    Hyderabad, India Epam Full time

    DescriptionEPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects that deliver...

  • Senior Data Engineer

    4 weeks ago

    Hyderabad, India Larsen & Toubro Infotech Ltd Full time

    Senior Data Engineer Job Req Id: 1248440 Bachelor’s Degree in Computer Science or a related discipline• 5+ years of applicable engineering experience• Experience in GCP, Big Query• Strong proficiency in Python with an emphasis in building data pipelines• Ability to write complex SQL to perform common types of analysis and aggregations •...

  • Senior Data Engineer

    2 weeks ago

    hyderabad, India Zenwork, Inc Full time

    Job Title: Senior Data Engineer - AWS, Redshift, QuickSightJob Description:We seek an experienced Data Engineer to join our team and help optimize our data processing and visualization capabilities. The ideal candidate will have a strong background in data engineering, particularly leveraging AWS products.Roles and Responsibilities:·      Design,...

  • Senior Data Engineer

    3 weeks ago

    Hyderabad, India Swiss Re Full time

    About the RoleAs a Senior Data Engineer, you will contribute to and support data management activities using most advanced Microsoft Azure technologies. The data management team enables data scientists and data analysts to provide risk knowledge advice and insights as well as data analytics SaaS solutions to Swiss Re clients.Key responsibilities...

  • Senior Data Engineer

    4 weeks ago

    hyderabad, India LTIMindtree Full time

    Senior Data Engineer Job Req Id: 1248440 Bachelor’s Degree in Computer Science or a related discipline• 5+ years of applicable engineering experience • Experience in GCP, Big Query • Strong proficiency in Python with an emphasis in building data pipelines • Ability to write complex SQL to perform common types of analysis and aggregations •...

  • Senior Data Engineer

    4 weeks ago

    hyderabad, India Epam Full time

    Description EPAM is a leading global provider of digital platform engineering and development services. We are committed to having a positive impact on our customers, our employees, and our communities. We embrace a dynamic and inclusive culture. Here you will collaborate with multi-national teams, contribute to a myriad of innovative projects...

  • Senior Data Engineer

    3 weeks ago

    Hyderabad, India Apptad Inc. Full time

    Role - Senior Data EngineerExp - 7+ yrsLocation - Bangalore/Hyderabad* Proficient in Python to design, develop, and maintain scalable data pipelines and ETL processes. * Has knowledge in pulling data using R via API's.* Extensive experience in building and optimizing data architectures, as well as strong skills in data modeling and SQL. * Capable enough to...

  • Senior Data Engineer

    3 weeks ago

    Hyderabad, India Apptad Inc. Full time

    Role - Senior Data EngineerExp - 7+ yrsLocation - Bangalore/Hyderabad* Proficient in Python to design, develop, and maintain scalable data pipelines and ETL processes. * Has knowledge in pulling data using R via API's.* Extensive experience in building and optimizing data architectures, as well as strong skills in data modeling and SQL. * Capable enough to...

  • Senior Data Engineer

    3 weeks ago

    Hyderabad, India Apptad Inc. Full time

    Role - Senior Data Engineer Exp - 7+ yrs Location - Bangalore/Hyderabad * Proficient in Python to design, develop, and maintain scalable data pipelines and ETL processes. * Has knowledge in pulling data using R via API's. * Extensive experience in building and optimizing data architectures, as well as strong skills in data modeling and SQL. * Capable...

  • Senior Data Engineer

    3 weeks ago

    Hyderabad, India Crunchyroll Full time

    About CrunchyrollWE HELP EVERYONE BELONG. IT’S OUR PURPOSE.Founded by fans, Crunchyroll delivers the art and culture of anime to a passionate community. We super-serve over 100 million anime and manga fans across 200+ countries and territories, and help them connect with the stories and characters they crave. Whether that experience is online or in-person,...

  • Senior Data Engineer

    1 week ago

    Hyderabad, India Zenwork, Inc Full time

    Job Title: Senior Data Engineer - AWS, Redshift, QuickSightJob Description:We seek an experienced Data Engineer to join our team and help optimize our data processing and visualization capabilities. The ideal candidate will have a strong background in data engineering, particularly leveraging AWS products.Roles and Responsibilities:·      Design,...

  • Senior Data Engineer

    2 weeks ago

    Hyderabad, India Zenwork, Inc Full time

    Job Title: Senior Data Engineer - AWS, Redshift, QuickSightJob Description:We seek an experienced Data Engineer to join our team and help optimize our data processing and visualization capabilities. The ideal candidate will have a strong background in data engineering, particularly leveraging AWS products.Roles and Responsibilities:·      Design,...

  • Senior Data Engineer

    2 weeks ago

    hyderabad, India Zenwork, Inc Full time

    Job Title: Senior Data Engineer - AWS, Redshift, QuickSight Job Description: We seek an experienced Data Engineer to join our team and help optimize our data processing and visualization capabilities. The ideal candidate will have a strong background in data engineering, particularly leveraging AWS products. Roles and Responsibilities: ...

  • Senior Data Engineer

    4 weeks ago

    Hyderabad, India Swiss Re Full time

    About the Role As a Senior Data Engineer, you will contribute to and support data management activities using most advanced Microsoft Azure technologies. The data management team enables data scientists and data analysts to provide risk knowledge advice and insights as well as data analytics SaaS solutions to Swiss Re clients.  Key responsibilities...

  • Senior Data Engineer

    3 weeks ago

    hyderabad, India Swiss Re Full time

    About the Role As a Senior Data Engineer, you will contribute to and support data management activities using most advanced Microsoft Azure technologies. The data management team enables data scientists and data analysts to provide risk knowledge advice and insights as well as data analytics SaaS solutions to Swiss Re clients.  Key...

  • Senior Data Engineer

    4 days ago

    Hyderabad, India V3 Staffing Solutions India P Limited Full time

    We have this opening with the global company that is shaping the future of gaming through groundbreaking technology and unparalleled creativity.Senior Data Engineer (AWS): Crafting Efficient Data Pipelines for Cloud PlatformsWe're seeking a seasoned Senior Data Engineer with over 9 years of hands-on experience in crafting and implementing highly efficient...