Data Engineer

3 months ago

Greater Hyderabad Area, India Circuitry.ai Full time

Data Engineer


Location: Hyderabad, India - Hybrid Remote (3 days a week in office preferred)


Position Overview:

As a Data Engineer focused on AI development, you will play a pivotal role in the design, development, and deployment of data infrastructure and pipelines that support artificial intelligence and machine learning initiatives within our organization. You will work closely with data scientists, machine learning engineers, and software developers to ensure that data is readily accessible, high-quality, and efficiently processed to fuel AI algorithms and models.


Responsibilities:

  • Data Pipeline Development: Design, implement, and maintain data pipelines that collect, transform, and store data from various sources, ensuring data quality and consistency. These pipelines will serve as the foundation for AI model training and deployment.
  • Data Integration: Collaborate with cross-functional teams to integrate data from diverse sources, including structured and unstructured data, into the data ecosystem. Ensure data is accessible and ready for AI development.
  • Data Cleansing and Preprocessing: Develop data preprocessing routines to clean and preprocess raw data, handling missing values, outliers, and data inconsistencies, making it suitable for machine learning algorithms. Ensures data consistency and integrity during the integration process, performing data validation and cleaning as needed.
  • Data Modeling Support: Work closely with data scientists and machine learning engineers to understand data requirements for AI model training. Assist in feature engineering and dataset creation for model development.
  • Data Governance: Implement data governance practices, including data cataloging, version control, and access controls to maintain data lineage and ensure data security and compliance.
  • Performance Optimization: Continuously monitor and optimize data pipelines and infrastructure for efficiency and scalability, ensuring timely data processing for AI applications. Monitor and tune data systems, identifies and resolves performance bottlenecks, and implements caching and indexing strategies to enhance query performance
  • Collaboration: Collaborate with cross-functional teams to understand AI project requirements and translate them into data engineering solutions. Collaborate with data scientists and analysts to optimize models and algorithms for data quality, security, and governance.
  • Documentation: Maintain comprehensive documentation of data pipelines, transformations, and processes to facilitate knowledge sharing and troubleshooting.
  • Quality Assurance: Implement and maintain data quality checks and validation processes to identify and address data anomalies or issues that may impact AI model performance. Implement data quality checks and validations within data pipelines to ensure the accuracy, consistency, and completeness of data
  • Learning and Development: Stay up-to-date with industry trends, emerging technologies, and best practices in data engineering and AI development. Propose and evaluate new tools and technologies that can enhance the AI development process.


Job Requirements:


Education

  • A bachelor’s degree in computer science, data science, software engineering, information systems, or related quantitative field; master’s degree preferred

Experience

  • Minimum 3 years in data management, focusing on data integration, modeling, optimization, data quality, and related data engineering activities.
  • Demonstrated experience in developing and managing data warehouses for AI/ML applications.

Skills

  • Proficient in big data languages such as PySpark, SQL, and/or DBT.
  • Skilled in database technologies including SQL, NoSQL, Deltalake, and MongoDB.
  • Experience with cloud platforms like AWS or Azure, and container technologies like Docker or Kubernetes.
  • Strong understanding of ETL and ELT processes, preferably with Databricks.
  • Knowledgeable in AI/ML principles, models, tools, and deployment strategies.
  • Advanced expertise in Apache technologies like Kafka, Airflow, and Spark for creating effective data pipelines.
  • Capable of designing and implementing data solutions for AI, ML, and BI applications.
  • Proficient in programming languages like Python, and data science tools such as Scala or SQL.
  • Collaborative skills to work with diverse technical teams, guiding and educating on data products.
  • Exceptional problem-solving abilities, including troubleshooting and resolving issues in unfamiliar code or systems.
  • Excellent business insight and interpersonal skills, capable of working with senior stakeholders to drive change.
  • Ability to articulate business cases, data sources, management concepts, and analytical methods.
  • Skilled in translating technical language for executive, business, IT, and quantitative audiences.

  • Data Engineer

    3 months ago

    Greater Hyderabad Area, India Circuitry.ai Full time

    Data Engineer Location: Hyderabad, India - Hybrid Remote (3 days a week in office preferred) Position Overview: As a Data Engineer focused on AI development, you will play a pivotal role in the design, development, and deployment of data infrastructure and pipelines that support artificial intelligence and machine learning initiatives within our...

  • Big Data Engineer

    4 weeks ago

    Hyderabad, Telangana, India data Full time

    Overview: We're seeking a skilled Big Data Engineer to join our team at data.ai. As a key member of our engineering team, you will be responsible for designing and implementing innovative data solutions that drive business growth. If you're passionate about working with large data sets, have experience with big data technologies, and are eager to...

  • Data Engineer

    4 weeks ago

    Hyderabad, India NTT DATA Full time

    Job Description Role Definition:Data engineers build and test scalable Big Data ecosystems for the businesses and help build data systems that are stable and highly optimized. Data engineers also update the existing systems with newer or upgraded versions of the current technologies to improve the efficiency of the databases, data storage layers and process...

  • Big Data Engineer, Data Modeling Expert

    1 month ago

    Hyderabad, Telangana, India data Full time

    Unlock the Power of Data with UsWe're seeking an experienced Big Data Engineer to join our team and contribute to the development of innovative analytics and data products. As a key member of our engineering team, you will work closely with Data Scientists, Data Engineers, Product Managers, and Software Architects to design and implement complex data product...

  • Principal Data Engineer

    4 months ago

    Greater Hyderabad Area, India Blackbaud India Full time

    About the roleWe are seeking a Principal Data Engineer who has a passion for data and an obsession for delivering high quality solutions and value to customers. You will be part of a team focused on data ingestion, semantic modeling, access, and security. Your efforts will support activities in our data governance, informatics, and science pillars and...

  • Principal Data Engineer

    4 months ago

    Greater Hyderabad Area, India Blackbaud India Full time

    About the role We are seeking a Principal Data Engineer who has a passion for data and an obsession for delivering high quality solutions and value to customers. You will be part of a team focused on data ingestion, semantic modeling, access, and security. Your efforts will support activities in our data governance, informatics, and science pillars and...

  • Data Engineer

    2 months ago

    Hyderabad, India NTT DATA Services Full time

    Role Definition:Data engineers build and test scalable Big Data ecosystems for the businesses and help build data systems that are stable and highly optimized. Data engineers also update the existing systems with newer or upgraded versions of the current technologies to improve the efficiency of the databases, data storage layers and process the data...

  • Innovative Big Data Solutions Engineer

    7 hours ago

    Hyderabad, Telangana, India data Full time

    About the RoleWe are seeking an experienced Big Data Solutions Engineer to join our team in Hyderabad, India. As a key member of our engineering team, you will be responsible for designing and implementing complex data products components based on requirements with possible technical solutions.You will work closely with Data Scientists, Data Engineers,...

  • Big Data Engineer

    3 weeks ago

    Hyderabad, Telangana, India NTT DATA Services Full time

    Data Engineer RoleData engineers at NTT DATA Services are responsible for building and testing scalable Big Data ecosystems for our clients. They work closely with data scientists to ensure the source data is aggregated and cleansed, and with product managers to understand the business objectives. Data engineers must have expertise in all phases of data...

  • Data Engineer

    4 weeks ago

    Greater Bengaluru Area, India Kratos Gamer Network Full time

    About KGeN: KGeN (Kratos Gamer Network) is building a multi-chain gamer network that aims to become the home of global gamer data, owned by gamers. At the heart of it is the revolutionary Proof of Gamer (P.O.G) Engine that allows gamers to build, own & monetise their reputation. This data framework allows game publishers to engage with precise gamer...

  • Data Engineer

    4 weeks ago

    Greater Bengaluru Area, India Kratos Gamer Network Full time

    About KGeN:KGeN (Kratos Gamer Network) is building a multi-chain gamer network that aims to become the home of global gamer data, owned by gamers. At the heart of it is the revolutionary Proof of Gamer (P.O.G) Engine that allows gamers to build, own & monetise their reputation. This data framework allows game publishers to engage with precise gamer cohorts,...

  • Big Data Systems Engineer

    3 weeks ago

    Hyderabad, Telangana, India NTT DATA Full time

    Job Title: Data EngineerJob Summary:We are seeking a skilled Data Engineer to design, build, and maintain large-scale data systems. The ideal candidate will have expertise in big data ecosystems, data processing, and data storage.Key Responsibilities:Design and develop scalable data systems for business applications.Work closely with data scientists to...

  • Advanced Data Analytics Engineer with Cloud Expertise

    6 days ago

    Hyderabad, Telangana, India data Full time

    Unlock the Power of Big DataWe are seeking an experienced Advanced Data Analytics Engineer to join our team in Hyderabad, India. As a key member of our engineering team, you will play a crucial role in designing and implementing innovative data products using agile methodologies and open-source technologies.About the RoleThis is an exciting opportunity to...

  • Data Intelligence Specialist

    1 day ago

    Hyderabad, Telangana, India data Full time

    About the RoleWe are seeking a highly skilled Data Intelligence Specialist to join our engineering team in Hyderabad, India.The ideal candidate will have a strong passion for data analysis and quality monitoring, with a deep understanding of complex datasets and statistical measurements.Key ResponsibilitiesTake full ownership of data.ai core intelligence...

  • Data Engineer

    3 weeks ago

    Greater Bengaluru Area, India Kunato Full time

    We are seeking a dynamic and driven Data Engineer with a strong blend of technical expertise and business acumen. In this role, you will collaborate with cross-functional teams, including Software Engineers, Data Scientists, Business Analysts, Product Managers, and other key stakeholders, to develop innovative data solutions that fuel business insights and...

  • Data Engineer

    3 weeks ago

    Greater Bengaluru Area, India Kunato Full time

    We are seeking a dynamic and driven Data Engineer with a strong blend of technical expertise and business acumen. In this role, you will collaborate with cross-functional teams, including Software Engineers, Data Scientists, Business Analysts, Product Managers, and other key stakeholders, to develop innovative data solutions that fuel business insights and...

  • Senior Data Engineer for Digital Transformation

    6 days ago

    Hyderabad, Telangana, India NTT DATA Full time

    Job Description:We are seeking a highly skilled Senior Data Engineer to join our team in Remote, Telangana (IN). This is a full-time position with 40 hours per week.About the Role:The successful candidate will design, develop, and maintain scalable data pipelines in the Python ecosystem. They will also participate in regular team ceremonies and contribute to...

  • Senior Data Engineer

    3 weeks ago

    Greater Kolkata Area, India FinArb Consulting Full time

    Job Title: Senior Data EngineerResponsibilities:We are seeking a self-driven Senior Data Engineer to join our dynamic team of data and software engineers. Your primary responsibilities will include:Designing, building, and maintaining scalable data pipelines and workflows.Developing and optimizing data ingestion, transformation, and integration...

  • Data Analyst Position

    4 weeks ago

    Hyderabad, Telangana, India data Full time

    A Career Defining Role in Data IntelligenceAs a Data Analyst, you will play a pivotal role in helping us uncover hidden insights and trends within our vast dataset. Your expertise will be instrumental in ensuring the accuracy and quality of our data, enabling us to make informed decisions that drive business growth.ResponsibilitiesTake ownership of data.ai's...

  • Data Engineering Leader

    4 weeks ago

    Greater Bengaluru Area, India CodeVyasa Full time

    Job DescriptionWe are seeking an experienced Senior Data Engineer to join our team at CodeVyasa.The ideal candidate will have a strong background in designing, implementing, and maintaining large-scale data processing pipelines using PySpark for both streaming and batch processing.Key Responsibilities:Data Pipeline Development: Design, implement, and...