Data Engineer

2 weeks ago

BangalorePune, India Gethyr Full time

We are seeking a talented and motivated Data Engineer stream processing, particularly utilizing Flink and Kafka, alongside expertise in Spark and AWS technologies.

As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining robust stream processing solutions to handle large volumes of data in real-time, ensuring high performance, scalability, and reliability.

Responsibilities :

  • Stream Processing Development: Design, develop, and optimize stream processing pipelines using Apache Flink and Kafka to process real-time data streams efficiently.
  • Data Ingestion: Implement robust data ingestion pipelines to collect, process, and distribute streaming data from various sources into the Flink and Kafka ecosystems.
  • Data Transformation: Perform data transformation and enrichment operations on streaming data using Spark Streaming and other relevant technologies to derive actionable insights.
  • Performance Optimization: Continuously optimize stream processing pipelines for performance, scalability, and reliability, ensuring low-latency and high-throughput data processing.
  • Monitoring and Troubleshooting: Monitor stream processing jobs, troubleshoot issues, and implement necessary optimizations to ensure smooth operation and minimal downtime.
  • Integration with AWS Services: Leverage AWS technologies such as Amazon Kinesis, AWS Lambda, Amazon EMR, and others to build end-to-end stream processing solutions in the cloud environment.
  • Data Governance and Security: Implement data governance and security measures to ensure compliance with regulatory requirements and protect sensitive data in streaming pipelines.
  • Collaboration: Collaborate closely with cross-functional teams including data scientists, software engineers, and business stakeholders to understand requirements and deliver impactful solutions.
  • Documentation: Create and maintain comprehensive documentation for stream processing pipelines, including design specifications, deployment instructions, and operational procedures.
  • Continuous Learning: Stay updated with the latest advancements in stream processing technologies, tools, and best practices, and incorporate them into the development process as or Master's degree in Computer Science, Engineering, or a related field.
  • 2-5 years of professional experience in data engineering roles with a focus on stream processing.
  • Strong proficiency in Apache Flink and Kafka for building real-time stream processing applications.
  • Hands-on experience with Spark and Spark Streaming for batch and stream processing.
  • Solid understanding of cloud computing platforms, particularly AWS services such as Amazon, Kinesis, AWS Lambda, Amazon EMR, etc.
  • Proficiency in programming languages such as Java, Scala, or Python.
  • Experience with containerization and orchestration tools like Docker and Kubernetes is a plus.
  • Excellent problem-solving skills and the ability to troubleshoot complex distributed systems.
  • Strong communication skills and the ability to work effectively in a collaborative team (ref:hirist.tech)

  • Senior Data Engineer

    3 days ago

    Pune, India Data Axle Full time

    Job DescriptionAbout Data Axle:Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and...

  • Senior Data Engineer

    2 weeks ago

    pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Senior Data Engineer

    1 month ago

    Pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Senior Data Scientist

    2 weeks ago

    pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the USA. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based...

  • Senior Data Scientist

    1 month ago

    Pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the USA. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Aptus Data Labs

    1 month ago

    bangalore, India Aptus Data Labs Full time

    Aptus Data Labs is an innovative Data Engineering and Advanced Analytics company offering analytical solutions and consulting services.Role - Machine Learning EngineerExp - 3-5 YrsLocation - RemoteSkill - AWS Sagemaker , ML Pipeline, Machine Learning, AI, Python, NLP, , CI/CD , AWS Services - Lambda, Step functions and other AWS ServicesThe ideal...

  • Aptus Data Labs

    2 weeks ago

    Bangalore, India Aptus Data Labs Full time

    Aptus Data Labs is an innovative Data Engineering and Advanced Analytics company offering analytical solutions and consulting services.Role - Machine Learning EngineerExp - 3-5 YrsLocation - RemoteSkill - AWS Sagemaker , ML Pipeline, Machine Learning, AI, Python, NLP, , CI/CD , AWS Services - Lambda, Step functions and other AWS ServicesThe ideal...

  • Aptus Data Labs

    4 weeks ago

    Bangalore, Karnataka, India Aptus Data Labs Full time

    Aptus Data Labs is an innovative Data Engineering and Advanced Analytics company offering analytical solutions and consulting services.Role - Machine Learning EngineerExp- 3-5 YrsLocation - RemoteSkill - AWS Sagemaker , ML Pipeline, Machine Learning, AI, Python, NLP, , CI/CD , AWS Services - Lambda, Step functions and other AWS ServicesThe ideal...

  • Software Engineer

    4 weeks ago

    Pune, India Data Axle Full time

    Job DescriptionAbout Data Axle:Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and...

  • Software Engineer

    4 weeks ago

    pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Software Engineer

    4 weeks ago

    Pune, India Data Axle Full time

    Job Description About Data Axle: Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set up a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology...

  • Data Engineer

    1 week ago

    bangalore, India ANZ Full time

    About the role Data Engineer – Role Type – FTE Job Grade: Group 4.3 At ANZ our purpose is to shape a world where people and communities thrive and to achieve this, we need a talented Data Engineer to join our Australia Commercial tribe. The Australia Data tribe sits within our Australia Commercial division and our mission is...

  • Data Engineer

    4 days ago

    bangalore, India SMARTBORN SOFTWARE CONSULTING PVT.LTD. Full time

    Role Description This is a full-time hybrid role as a Data Engineer at SMARTBORN SOFTWARE CONSULTING PVT.LTD. The Data Engineer will be responsible for day-to-day tasks related to data engineering, including data modeling, Extract Transform Load (ETL) development, data warehousing, and data analytics. The role is based in Bengaluru, with flexibility for...

  • Data Engineer, IN Data Engineering

    1 month ago

    bangalore, India ADCI - Karnataka Full time

    IN Data Engineering & Analytics(IDEA) Team is looking to hire a rock star Data Engineer to build and manage the largest petabyte-scale data infrastructure in India for Amazon India businesses.IN Data Engineering & Analytics (IDEA) team is the central Data engineering and Analytics team for all A.in businesses. The team's charter includes 1) Providing Unified...

  • Data Engineer

    1 week ago

    bangalore, India Australia and New Zealand Banking Group Limited (ANZ) Full time

    Data Engineer Data Engineer Req ID: Department: Commercial Data & Analytics Division: Australia Commercial Location: Bengaluru About the role Data Engineer – Role Type – FTE Job Grade: Group 4.3 At ANZ our purpose is to shape a world where people and communities thrive and to achieve this, we need a talented Data Engineer to join...

  • Data Engineer

    1 month ago

    bangalore, India Analog Devices Full time

    Gather business and functional requirements and translate into robust scalable, solutions aligned to the Data Architecture guidelines Hands on development work on all aspects of data analysis, data provisioning, Data modeling, performance tuning and optimization Design, implement and operate large scale, high volume, high performance data structures for...

  • Data Engineer

    1 day ago

    bangalore, India Joveo Full time

    Role: Data Engineer Location: Hybrid (Bangalore/Hyderabad) About Joveo Joveo was founded in 2017 by a group of visionary entrepreneurs and enthusiastic problem solvers who foresaw the need for greater transparency and predictability in recruitment marketing. The focus was on building a cutting-edge technology platform that uses data and...

  • Data Engineer

    1 week ago

    bangalore, India Aryng Full time

    Welcome! You made it to the job description page! Aryng is looking for a Senior Data Engineer with experience in developing enterprise-class distributed data engineering solutions on the cloud. We are seeking an entrepreneurial and technology-proficient Data Engineer who is an expert in theimplementation of a large-scale, highly efficient data platform,...

  • Data Engineer

    1 month ago

    bangalore, India Yara Full time

    Responsibilities Participate in designing and building data pipelines to collect, manage and store data across different solutions and tools where the data resides for the YAA teams. Monitor, manage and maintain data pipelines as part of the day-to-day data operations. Build data mart tables for various use cases by working with the different data...

  • Data Engineer

    1 month ago

    Bangalore, India Magna Hire Full time

    Responsibilities : Support the data engineering team in implementing analytics solutions, ensuring adherence to project timelines and quality standards. Participate in at least one full implementation lifecycle of dbt (data build tool), contributing to design, development, and deployment phases. Assist senior team members in optimizing dbt models and...