Big Data Engineer

20 hours ago

Kanpur, Uttar Pradesh, India Growel Softech Pvt. Ltd. Full time ₹ 6,00,000 - ₹ 18,00,000 per year

Skill : Big Data Need to Have : Python, Pyspark, Trino, Hive Good to Have : Snowflake, SQL, Airflow, Openshift, Kubernentes Location : Hyderabad, Pune, Bangalore Job Description : Develop and maintain data pipelines, ELT processes, and workflow orchestration using Apache Airflow, Python and PySpark to ensure the efficient and reliable delivery of data. Design and implement custom connectors to facilitate the ingestion of diverse data sources into our platform, including structured and unstructured data from various document formats .

Collaborate closely with cross functional teams to gather requirements, understand data needs, and translate them into technical solutions. Implement DataOps principles and best practices to ensure robust data operations and efficient data delivery. Design and implement data CI/CD pipelines to enable automated and efficient data integration, transformation, and deployment processes.

Monitor and troubleshoot data pipelines, proactively identifying and resolving issues related to data ingestion, transformation, and loading. Conduct data validation and testing to ensure the accuracy, consistency, and compliance of data. Document data workflows, processes, and technical specifications to facilitate knowledge sharing and ensure data governance.

Responsibilities: Proficiency in using Apache Airflow and Spark for data transformation, data integration, and data management.

Experience implementing workflow orchestration using tools like Apache Airflow or similar platforms. Demonstrated experience in developing custom connectors for data ingestion from various sources.

Strong understanding of SQL and database concepts, with the ability to write efficient queries and optimize performance. Strong programming skills, particularly in Python, with experience in web scraping and implementing OCR based data extraction. Experience implementing DataOps principles and practices, including data CI/CD pipelines. Understanding of NLP techniques to analyze text data and derive valuable insights for compliance and business intelligence purposes. Familiarity with NLP techniques and libraries for text data analysis.

Familiarity with data visualization tools Apache SuperSet and dashboard development. Knowledge of data streaming and real time data processing technologies (e.g., Apache Kafka). Strong understanding of software development principles and practices, including version control (e.g., Git) and code review processes. Required Skills Python, Pyspark, SQL, Airflow, Trino, Hive, Snowflake, Agile Scrum Optional Skill Linux, Openshift, Kubernentes, Superset

  • Senior Data Engineer

    1 week ago

    Kanpur, Uttar Pradesh, India appsierra Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Data Engineering and Infrastructure: 6+ years of experience in a Data Engineering role, designing, building, and managingscalable and reliable data systems.Proficient with big data and stream-processing technologies such as Spark and Kafka.Hands-on experience with cloud platforms, particularly AWS services like EC2 and RDS.Skilled in building and...

  • Azure Data Engineer

    21 hours ago

    Kanpur, Uttar Pradesh, India Growel Softech Pvt. Ltd. Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Experienced Azure Data Engineer Tech Lead ( Tech Lead ) with 7 to 10 years of expertise Design, develop, and maintain robust data pipelines using Azure Databricks, Azure Data Factory, Pyspark / Python, Azure Data Lake Gen2, and Azure SQL Database.Integrate and manage data from various source systems, ensuring data quality and consistency.Mandatory Skills :...

  • Data Analyst

    1 week ago

    Kanpur, Uttar Pradesh, India C3iHub, IIT Kanpur Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    DescriptionWe are looking for a data engineer and scientist with a deep understanding of blockchain network protocols, infrastructure, systems engineering, distributed systems, and architectural design. The candidate will lead research and ideation to scale data infrastructure and feasibility in production. The candidate will have the opportunity to expand...

  • GCP + Terraform Expert

    18 hours ago

    Kanpur, Uttar Pradesh, India Growel Softech Pvt. Ltd. Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    We are looking for Terraform expert who can automate GCP environment.  Job Overview: We are seeking a skilled Senior Cloud Infrastructure Engineer to automate infrastructure creation on Google Cloud Platform (GCP) using Terraform . The candidate will be responsible for provisioning and setting up GCP projects and environments for the entire GCP data...

  • Security Analysis Advisor

    7 days ago

    Kanpur, Uttar Pradesh, India NTT DATA North America Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Req ID:337539NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now.We are currently seeking a Security Analysis Advisor to join our team in Nodia, Uttar Pradesh (IN-UP), India (IN).Role OverviewThe Cloud Security...

  • Sr. Project Engineer and Project Engineer

    2 weeks ago

    Kanpur, Uttar Pradesh, India Department of Management Sciences, IIT Kanpur Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Open Position for Sr. Project Engineer (02) and Project Engineer (02) at Centre for Energy Regulation (CER), IIT Kanpur. Last Date for submitting online applications is 8 Oct., 2025.Qualifications include (Check weblink for details)Sr. Project Engineer (02)M. E. / M. Tech. (Power Systems, Energy Systems, Renewable Energy Technologies, or related fields) + 3...

  • Azure Devops Engineer

    15 hours ago

    Kanpur, Uttar Pradesh, India Growel Softech Pvt. Ltd. Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Skill NameAzure Devops EngineerJob Location Pan India JD - Proven experience as an Azure DevOps Engineer or similar role.- Strong experience with Azure Cloud services and architecture.- Extensive experience with Terraform for Infrastructure as Code (IaC).- Hands-on experience with Azure Data Factory for data integration.- Proficiency in API development and...

  • Lead Engineer

    24 hours ago

    Kanpur, Uttar Pradesh, India Technip Energies Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Job DescriptionBe part of the solution at Technip Energies and embark on a one-of-a-kind journey. You will be helping to develop cutting-edge solutions to solve real-world energy problems.We are currently seeking aLead Engineer – Process & Technologyto join our Process & Technology team based inNoida & Gurugram, India.About us:Technip Energies is a global...

  • Machine learning Engineer

    2 weeks ago

    Kanpur, Uttar Pradesh, India Weekday AI (YC W21) Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    This role is for one of the Weekday's clientsMin Experience: 3 yearsLocation: Gurugram, Delhi, Uttar Pradesh, NCR, NOIDA, Kanpur, UttarakhandJobType: full-timeWe are seeking a highly skilled and motivatedMachine Learning Engineerwith 3-5 years of hands-on experience to join our growing team. The ideal candidate will have strong expertise in designing,...

  • Sr. System Engineer

    1 day ago

    Kanpur, Uttar Pradesh, India Growel Softech Pvt. Ltd. Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Location: Hyd/Chennai/Bglore Job Description: Informatica Dev: Having 7+yrs of relevant experience with strong in DW fundaments and Data Modelling concepts and Informatica suit Experience in Informatica PowerCenter, Workflow Manager and Workflow Designer Good knowledge on Data modelling concepts Good knowledge and experience in IICS Good knowledge and...