Data Engineer

Found in: Adzuna IN C2 - 2 weeks ago

BangalorePune, IN Gethyr Full time

We are seeking a talented and motivated Data Engineer stream processing, particularly utilizing Flink and Kafka, alongside expertise in Spark and AWS technologies.

As a Data Engineer, you will play a crucial role in designing, implementing, and maintaining robust stream processing solutions to handle large volumes of data in real-time, ensuring high performance, scalability, and reliability.

Responsibilities :

  • Stream Processing Development: Design, develop, and optimize stream processing pipelines using Apache Flink and Kafka to process real-time data streams efficiently.
  • Data Ingestion: Implement robust data ingestion pipelines to collect, process, and distribute streaming data from various sources into the Flink and Kafka ecosystems.
  • Data Transformation: Perform data transformation and enrichment operations on streaming data using Spark Streaming and other relevant technologies to derive actionable insights.
  • Performance Optimization: Continuously optimize stream processing pipelines for performance, scalability, and reliability, ensuring low-latency and high-throughput data processing.
  • Monitoring and Troubleshooting: Monitor stream processing jobs, troubleshoot issues, and implement necessary optimizations to ensure smooth operation and minimal downtime.
  • Integration with AWS Services: Leverage AWS technologies such as Amazon Kinesis, AWS Lambda, Amazon EMR, and others to build end-to-end stream processing solutions in the cloud environment.
  • Data Governance and Security: Implement data governance and security measures to ensure compliance with regulatory requirements and protect sensitive data in streaming pipelines.
  • Collaboration: Collaborate closely with cross-functional teams including data scientists, software engineers, and business stakeholders to understand requirements and deliver impactful solutions.
  • Documentation: Create and maintain comprehensive documentation for stream processing pipelines, including design specifications, deployment instructions, and operational procedures.
  • Continuous Learning: Stay updated with the latest advancements in stream processing technologies, tools, and best practices, and incorporate them into the development process as or Master's degree in Computer Science, Engineering, or a related field.
  • 2-5 years of professional experience in data engineering roles with a focus on stream processing.
  • Strong proficiency in Apache Flink and Kafka for building real-time stream processing applications.
  • Hands-on experience with Spark and Spark Streaming for batch and stream processing.
  • Solid understanding of cloud computing platforms, particularly AWS services such as Amazon, Kinesis, AWS Lambda, Amazon EMR, etc.
  • Proficiency in programming languages such as Java, Scala, or Python.
  • Experience with containerization and orchestration tools like Docker and Kubernetes is a plus.
  • Excellent problem-solving skills and the ability to troubleshoot complex distributed systems.
  • Strong communication skills and the ability to work effectively in a collaborative team (ref:hirist.tech)

  • Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Pune, IN vipsa solutions Full time

    SALARY : 15LPA - Engineer:Location: Pune, BangaloreExperience: 6-9 yearsNP: Immediate- 15 daysKey accountabilities:As a Data Engineer you will be accountable for:- Understanding operational needs by collaborating with specialized teams- Supporting key, involves architecture designing, building and deploying data systems, pipelines etc- Build agile, scalable,...

  • Senior Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Hyderabad/Pune, IN LAGNUS PRIVATE LIMITED Full time

    About the job : ROLE : Snowflake and Matillion Data Engineer JD : Experience : 4 to 6 Years Overview :- Design, implement, and optimize data pipelines and workflows using Snowflake and Matillion to support our data integration, transformation, and analytics initiatives.- Collaborate with cross-functional teams to gather requirements, define data models, and...

  • Senior Big Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Pune, IN Recrud Full time

    Senior Big Data Engineer Position. Experience Required: 6 -10 Years. Employment Type: Full-Time. About Our Client:. Our client is a distinguished product engineering consulting firm based in Bangalore, dedicated to providing innovative offshore development solutions. With over half a century of combined expertise in product engineering and consulting...

  • Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Pune, IN Pylon Management Consulting Full time

    Requirements:- Overall 5+ years of IT experience with 3+ years in data-related technologies.- Minimum 2.5 years of experience in Big Data technologies and working exposure in at least one cloud platform on related data services (AWS / Azure / GCP).- Hands-on experience with the Hadoop stack - HDFS, sqoop, Kafka, Pulsar, NiFi, Spark, Spark Streaming, Flink,...

  • eInfochips - AWS Data Engineer - Data Pipeline

    Found in: Adzuna IN C2 - 2 weeks ago

    Pune/Ahmedabad/Bangalore/Indore, IN eInfochips Full time

    Role : AWS Data EngineerExperience : 5+ yearsLocation : Pune/Ahmedabad/Indore/ bangaloreCompany Overview :eInfochips, an Arrow company, is a global leader in product engineering and semiconductor design services. Renowned for our technological innovations, we've developed over 500 products, achieving 40M deployments across 140 countries. Our expertise...

  • Azure Data Engineer

    Found in: Adzuna IN C2 - 1 day ago

    Bangalore/Hyderabad/Pune, IN Kezan Consulting Full time

    Work Timings : 10 AM - 7 PMNumber of Openings : 5Joining Date : Within a weekJob Description :We are seeking experienced Azure Data Engineers to join our team on a contractual basis. As an Azure Data Engineer, you will be responsible for designing, implementing, and maintaining data solutions on the Azure platform. The ideal candidate will have a strong...

  • Python Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Hyderabad/Pune/Mumbai/Chennai, IN Swift Strategic Staff Solutions INC Full time

    Job Description :We are seeking a talented and experienced Python Data Engineer to join our data team and play a crucial role in building, maintaining, and optimizing data pipelines for our cloud-based data warehouse.Responsibilities :- Design, develop, and implement data pipelines for ingesting, transforming, and loading data into Snowflake using Python...

  • Databricks Administrator/Support Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Pune/Bangalore, IN Tekfortune IT India Pvt Ltd Full time

    Title : Databricks EngineerExperience : 4 to 10 yearsWork Location : Pune and Bangalore LocationNotice Period : Immediate to 45 DaysJob Description : We are seeking a highly skilled and experienced Databricks Engineer to join our growing team and play a crucial role in enhancing our data processing and analytics capabilities.As a Databricks Engineer you will...

  • Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Noida, IN Xperthunt Full time

    About the Role :We're seeking a highly skilled Data Engineer to join our team and play a pivotal role in building and maintaining our data infrastructure.You'll collaborate closely with data scientists, analysts, and stakeholders to understand their data needs and design, develop, and implement data solutions that empower effective analysis and...

  • Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Any Location, IN INSIGHT GLOBAL SOLUTIONS Full time

    Data Engineer Must Haves :- Bachelors Degree in science, technology, or another related field.- 3+ years of data engineering experience.- 3+ years of experience with Cloud platforms like GCP, Azure, Airflow.- 4+ years of data warehouse, relational database, and SQL experience.- 4+ years of supporting ETL process and job scheduling experience.- 3+ years of...

  • CGI - Data Engineer - Data Pipeline

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Hyderabad/Chennai/Pune, IN CGI Information Systems and Management Consultants Full time

    Required Skills :- 5+ years of experience in database administration duties- Bachelor's degree (or equivalent) in computer science, information technology, engineering, or related discipline- Experience in building or maintaining ETL processes.- Experienced in working with big data analytics tools.- Need to be proficient in various database types and...

  • Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Pune/Mumbai/Hyderabad/Chennai/Bangalore, IN RapidBraiins Full time

    Position Overview :As a Talend - Data Engineering Specialist, you will play a key role in designing, developing, and maintaining data pipelines using Talend, AWS, and SQL. You will work closely with cross-functional teams to gather requirements, design data solutions, and implement scalable and efficient ETL processes. Additionally, you will leverage your...

  • Lead Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Hyderabad/Pune, IN iMind Your Business Solutions Private Limited Full time

    Job Description :We are hiring Lead Data Engineers for the leaders In Analytics & Cloud Applications, headquartered in San Francisco and have offices in India (Bangalore, Hyderabad and Pune)Our client provides strategic guidance and technology systems for clients wishing to solve their most complex and interesting business challenges involving cloud...

  • Senior Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Hyderabad/Pune/Bangalore, IN iMind Your Business Solutions Private Limited Full time

    Job Description :We are hiring Data Engineers for the leaders In Analytics & Cloud Applications, headquartered in San Francisco and have offices in India (Bangalore, Hyderabad and Pune).Our client provides strategic guidance and technology systems for clients wishing to solve their most complex and interesting business challenges involving cloud...

  • AWS Data Engineer

    Found in: Adzuna IN C2 - 1 day ago

    Bangalore/Pune/Hyderabad, IN Kezan Consulting Full time

    Company Overview : Kezan Consulting is a leading provider of Human Resources Services in the industry. We specialize in providing comprehensive HR solutions to our clients, helping them achieve their goals and drive business success. With a team of highly skilled professionals and a commitment to delivering excellence, we have established ourselves as a...

  • Data Solution Architect

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Pune, IN TELL Jobs Full time

    Job Description : The person appointed will be an integral member of the Architecture Team and will be responsible for the design and implementation of the enterprise-wide data strategy, ensuring the strategy supports the current and future business needs.The role will involve collaborating with Business and IT stakeholders at all levels to ensure the...

  • Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Pune/Gurgaon/Gurugram/Bangalore/Noida, IN HuQuo Consulting Pvt. Ltd. Full time

    Data Engineer Role Mandate skills : SSIS + Datawarehouse + Azure Data Factory + SQLJob Description :- Designing and building optimized data pipelines using cutting-edge technologies in a cloud environment to drive analytical insights.- Constructing infrastructure for efficient ETL processes from various sources and storage systems.- Collaborating closely...

  • Senior Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Kolkata/Pune, IN Nilasu consulting Full time

    Summary :- We are seeking a highly skilled and experienced Senior Data Engineer to join our team.- You will play a key role in designing, developing, and maintaining large-scale data processing pipelines using Python and Spark/PySpark.- Your expertise in distributed computing frameworks and DevOps tools will be instrumental in building efficient and scalable...

  • Data Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Mumbai/Chennai/Pune/Bangalore, IN Coders Brain Pvt Ltd Full time

    JD - Python and Snowflake- Experienced Data engineer(6+) responsible for developing, overseeing, organizing, storing, and analyzing data and data systems.- Responsible for Cloud migrations for data pipelines using AWS and Snowflake- Participate in all aspects of the software development lifecycle for AWS and Snowflake solutions, including planning,...

  • AI/ML Engineer

    Found in: Adzuna IN C2 - 2 weeks ago

    Bangalore/Pune/Hyderabad/Chennai, IN Nilasu consulting Full time

    Summary :We are seeking a talented and passionate AI/Machine Learning Engineer to join our team. You will play a key role in the entire lifecycle of machine learning projects, from data engineering and classical machine learning to potentially exploring deep learning and natural language processing techniques. Your strong foundation in Python coding,...