Data Engineer

2 months ago

Chandigarh, India Spark Brains Pvt. Ltd. Full time

1. Data Acquisition

- Candidate should manage the existing Data pipelines built for data ingestion.

- Create and manage new data pipelines following the best practices for the new ingestion of data.

- Continuously monitor the data ingestion through Change Data Capture for the incremental load.

- Any failed batch job schedule to be analyzed and fixed to capture the data.

- Maintaining and continuously updating the technical documentation of the ingested data and maintaining the centralized data dictionary, with necessary data classifications.


2. Data Extraction and Cleaning

- Extraction of data from the data sources to be cleaned and ingested into a big data platform.

- Automation of data cleaning has to be defined before ingestions.

- Data cleaning to handle the missing data and remove any outliers and resolve any inconsistencies.

- Data quality check has to be performed in terms of accuracy, completeness, consistency, timeliness, believability, and interpretability.


3. Data Integration, Aggregation and Representation

- Exposing Data views or Data models to Reporting and source systems using Hive or Impala, or similar tools.

- Exposing cleansed data to the Artificial Intelligence team for building data science models.


4. Informatica Data Catalog

- Implement and configure the Informatica Enterprise Data Catalog (EDC) solution to discover and catalog data assets across the organization.

- Develop and maintain custom metadata scanners, resource configurations, and lineage extraction processes.

- Integrate EDC with other Informatica tools, such as Data Quality (IDQ), Master Data Management (MDM), and Axon Data Governance.

- Define and implement data classification, data profiling, and data quality rules to improve data visibility, accuracy, and trustworthiness.

- Collaborate with data stewards, data owners, and data governance teams to identify, document, and maintain business glossaries, data dictionaries, and data lineage information.

- Establish and maintain data governance policies, standards, and procedures within the EDC environment.

- Monitor and troubleshoot EDC performance issues, ensuring optimal performance and data availability.

- Train and support end-users in effectively utilizing the data catalog for data discovery and analysis.

- Keep up to date with industry best practices and trends, continuously improving the organization's data catalog implementation.

- Collaborate with cross-functional teams to drive data catalog adoption and ensure data governance compliance across the organization.


Skill Set:

- Certified Big Data Engineer from Cloudera/AWS/Azure

- Expertise with Big data products Cloudera stack.

- Expertise in Big Data querying tools, such as Hive, Hbase, and Impala.

- Expertise in SQL, writing complex queries/views, partitions, and bucketing.

- Strong Experience in Spark using Python/Scala.

- Expertise in messaging systems, such as Kafka or RabbitMQ.

- Hands-on experience in the Management of the Hadoop cluster with all included services.

- Implementing ETL process using Sqoop/Spark.

- Implementation including loading from disparate data sets, Pre-processing using Hive.

- Ability to design solutions independently based on high-level architecture.

- Collaborate with other development teams.

- Expertise in building stream-processing systems, using solutions such as Spark-Streaming, Apache NIFI, and KAFKA.

- Expertise with NoSQL databases such as HBase.

- Experience with Informatica Enterprise Data Catalog (EDC) implementation and administration.

- Strong knowledge of data management, data governance, and metadata management concepts.

- Proficiency in SQL and experience with various databases (e.g., Oracle, SQL Server, PostgreSQL) and data formats (e.g., XML, JSON, CSV).

- Experience with data integration, ETL/ELT processes, and Informatica Data Integration.


Location: Chandigarh

Salary: No bars for the right candidate.

Working: 5 days (WFO)

  • Data Engineer

    2 months ago

    Chandigarh, India Oceaneering Full time

    Position Summary and Location Assist with building, maintaining, and optimizing data pipelines, ensuring data flows efficiently across systems. You will work closely with senior data engineers and data analysts to support data integration, ETL (Extract, Transform, Load) processes, and overall data infrastructure. Duties & Responsibilities ...

  • Data Engineering Specialist

    4 weeks ago

    Chandigarh, Chandigarh, India Oceaneering Full time

    Job OverviewWe are seeking a highly skilled Data Engineer to join our team. This role is responsible for designing, building, and maintaining scalable data pipelines to move data from various sources to the data warehouse or data lake.

  • Senior Cloud Data Engineer

    1 month ago

    Chandigarh, Chandigarh, India Think Right Technologies Pvt Ltd Full time

    We are seeking a highly skilled Senior Cloud Data Engineer to join our team at Think Right Technologies Pvt Ltd. As a Senior Cloud Data Engineer, you will be responsible for designing, developing, and deploying large-scale data processing systems in cloud environments.Key Responsibilities:Develop and maintain scalable data processing systems using Python and...

  • Data Engineering and Science Lead

    4 weeks ago

    Chandigarh, Chandigarh, India Think Right Technologies Pvt. Ltd. Full time

    Think Right Technologies Pvt. Ltd. is a leading technology firm that delivers innovative solutions to clients worldwide. We are seeking an exceptional Data Engineering and Science Lead to join our team.The estimated salary for this role is ₹1,500,000 - ₹2,500,000 per annum, based on the location in India.Job Description:We are looking for a highly...

  • Data Engineering Specialist

    1 week ago

    Chandigarh, Chandigarh, India Edifecs Full time

    About the RoleWe are seeking a highly skilled Data Engineering Specialist to join our innovative software teams at Edifecs.In this position, you will be responsible for onboarding customers to our Risk Adjustment workflow applications. You will work closely with platform engineering, product, and implementation teams to design and develop scalable data...

  • Senior Data Engineer

    7 months ago

    Chandigarh, India Basware Full time

    Job Description In this role, you will be responsible for processing data, developing and maintaining big data pipelines, and enhancing data quality and efficiency. As a Senior Data Engineer , you are a key contributor in our Product Development team focused on Data and Analytics. This position involves combining data from various sources, designing...

  • Data Scientist: ML Engineer

    2 weeks ago

    Chandigarh, India Flahy Full time

    Who We AreFlahy is a biotechnology company that develops artificial intelligence driven detection & diagnostics products for Precision Oncology and collaborates with institutions for the development of targeted therapies. Flahy has developed FlahyBase, a computational platform that leverages sophisticated statistical & computational capabilities to analyze...

  • Data scientist: ml engineer

    2 weeks ago

    Chandigarh, India Flahy Full time

    Who We Are Flahy is a biotechnology company that develops artificial intelligence driven detection & diagnostics products for Precision Oncology and collaborates with institutions for the development of targeted therapies. Flahy has developed Flahy Base, a computational platform that leverages sophisticated statistical & computational capabilities to...

  • Sr. data integration engineer

    2 weeks ago

    Chandigarh, India Edifecs Full time

    Sr. Data Integration Engineer Overview Edifecs is seeking a Sr. Data Integration Engineer to join our innovative software teams. In this position, you will be responsible for onboarding customers to the Risk Adjustment workflow applications. As part of this role, you will work with platform engineering, product, and implementation teams. The ideal...

  • Data scientist: ml engineer

    2 weeks ago

    Chandigarh, India Flahy Full time

    Who We AreFlahy is a biotechnology company that develops artificial intelligence driven detection & diagnostics products for Precision Oncology and collaborates with institutions for the development of targeted therapies. Flahy has developed Flahy Base, a computational platform that leverages sophisticated statistical & computational capabilities to analyze...

  • Data Scientist

    4 months ago

    Chandigarh, Chandigarh, India Xorosoft Inc Full time

    **Job Title**:Data Scientist/Engineer **Vacancies**: 1 **Location**:Mohali, Punjab **Experience**:4-6 Years Are you passionate about using data to drive business decisions and optimize processes? We are looking for an experienced Data Scientist/Engineer to join our team and lead efforts in building AI-driven data models. This role offers a unique...

  • Data Scientist: ML Engineer

    1 week ago

    Chandigarh, India Flahy Full time

    Who We AreFlahy is a biotechnology company that develops artificial intelligence driven detection & diagnostics products for Precision Oncology and collaborates with institutions for the development of targeted therapies. Flahy has developed FlahyBase, a computational platform that leverages sophisticated statistical & computational capabilities to analyze...

  • Data Scientist

    2 months ago

    Chandigarh, Chandigarh, India GrayCell Technologies Full time

    **We're Hiring: Data scientist** **Location**: Chandigarh (onsite) **Experience**:4-5 years **Employment Type**: Full-time, Immediate Joiner **Working Days**: 5 days a week We are looking for a skilled professional with 5+ years of experience to join our dynamic team. If you're based in Chandigarh or Punjab and ready for an exciting remote opportunity,...

  • Data Scientist: ML Engineer

    2 weeks ago

    Chandigarh, India Flahy Full time

    Who We AreFlahy is a biotechnology company that develops artificial intelligence driven detection & diagnostics products for Precision Oncology and collaborates with institutions for the development of targeted therapies. Flahy has developed FlahyBase, a computational platform that leverages sophisticated statistical & computational capabilities to analyze...

  • Data Scientist: ML Engineer

    2 weeks ago

    Chandigarh, India Flahy Full time

    Who We AreFlahy is a biotechnology company that develops artificial intelligence driven detection & diagnostics products for Precision Oncology and collaborates with institutions for the development of targeted therapies. Flahy has developed FlahyBase, a computational platform that leverages sophisticated statistical & computational capabilities to analyze...

  • Data Scientist: ML Engineer

    2 weeks ago

    Chandigarh, India Flahy Full time

    Who We Are Flahy is a biotechnology company that develops artificial intelligence driven detection & diagnostics products for Precision Oncology and collaborates with institutions for the development of targeted therapies. Flahy has developed FlahyBase, a computational platform that leverages sophisticated statistical & computational capabilities to...

  • Data Scientist: ML Engineer

    1 week ago

    Chandigarh, India Flahy Full time

    Who We AreFlahy is a biotechnology company that develops artificial intelligence driven detection & diagnostics products for Precision Oncology and collaborates with institutions for the development of targeted therapies. Flahy has developed FlahyBase, a computational platform that leverages sophisticated statistical & computational capabilities to analyze...

  • Data Scientist: ML Engineer

    1 week ago

    Chandigarh, India Flahy Full time

    Who We AreFlahy is a biotechnology company that develops artificial intelligence driven detection & diagnostics products for Precision Oncology and collaborates with institutions for the development of targeted therapies. Flahy has developed FlahyBase, a computational platform that leverages sophisticated statistical & computational capabilities to analyze...

  • Data Scientist: ML Engineer

    2 weeks ago

    Chandigarh, India Flahy Full time

    Who We Are Flahy is a biotechnology company that develops artificial intelligence driven detection & diagnostics products for Precision Oncology and collaborates with institutions for the development of targeted therapies. Flahy has developed FlahyBase, a computational platform that leverages sophisticated statistical & computational capabilities to...

  • Data Scientist

    2 months ago

    Chandigarh, Chandigarh, India GrayCell Technologies Full time

    We're Hiring: Data scientist **Location**: Chandigarh (onsite) **Experience**: 4-5 years **Employment Type**: Full-time, Immediate Joiner **Working Days**: 5 days a week We are looking for a skilled professional with 5+ years of experience to join our dynamic team. If you're based in Chandigarh or Punjab and ready for an exciting remote opportunity,...