Big Data Engineer

3 weeks ago

Pune, Maharashtra, India PRONEXUS CONSULTING PRIVATE LIMITED Full time

About the Role :

We are seeking a highly skilled Senior Data Engineer with expertise in designing, building, and optimizing large-scale data pipelines and platforms.

The ideal candidate will have strong hands-on experience with big data technologies, AWS cloud services, and modern CI/CD automation frameworks.

You will play a pivotal role in architecting robust data solutions, ensuring scalability, performance, and reliability, while collaborating closely with cross-functional teams across engineering, product, and operations.

Key Responsibilities :

Data Platform Engineering :

- Design, develop, and enhance data ingestion, transformation, and orchestration pipelines using open-source frameworks, AWS cloud services, and GitLab automation.

- Implement best practices in distributed data processing using PySpark, Python, and SQL.

Collaboration & Solutioning :

- Partner with product managers, data scientists, and technology stakeholders to design and validate scalable data platform capabilities.

- Translate business requirements into technical specifications and implement data-driven solutions.

Optimization & Automation :

- Identify, design, and implement process improvements including automation of manual processes, pipeline optimization, and system scalability enhancements.

- Drive adoption of infrastructure-as-code and automated CI/CD pipelines for data workloads.

Monitoring & Reliability :

- Define, implement, and maintain robust monitoring, logging, and alerting mechanisms for data pipelines and services.

- Ensure data quality, availability, and reliability across the production environment.

Technical Enablement :

- Provide platform usage guidance, technical support, and best practices to teams consuming the data platform.

- Contribute to internal knowledge bases, playbooks, and engineering documentation.

Required Qualifications :

- Proven experience in building, maintaining, and optimizing large-scale data pipelines in distributed computing environments.

- Strong programming experience in Python and PySpark, with advanced working knowledge of SQL (4+ years).

- Expertise in working within Linux environments for data development and operations.

- Strong knowledge and experience with AWS services such as S3, EMR, Glue, Redshift, Lambda, and Step Functions.

- Hands-on experience with DevOps/CI/CD tools such as Git, Bitbucket, Jenkins, AWS CodeBuild, and CodePipeline.

- Familiarity with monitoring and alerting platforms (CloudWatch, Prometheus, Grafana, or equivalent).

- Knowledge of Palantir is a strong plus.

- Experience collaborating with cross-functional teams (engineering, product, operations) in a fast-paced environment.

Preferred Skills :

- Experience with containerized environments (Docker, Kubernetes).

- Exposure to data governance, lineage, and metadata management tools.

- Working knowledge of infrastructure-as-code tools (Terraform, CloudFormation).

- Familiarity with streaming technologies such as Kafka or Kinesis.

(ref:hirist.tech)

  • Big Data Engineer

    4 weeks ago

    Pune, Maharashtra, India Nice Software Solutions Pvt. Ltd. Full time

    Big Data Engineer (PySpark)Location: Pune/Nagpur (WFO)Experience: 8 - 12 YearsEmployment Type: Full-timeJob OverviewWe are looking for an experienced Big Data Engineer with strong expertise in PySpark and Big Data ecosystems. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines while ensuring high...

  • Big Data Engineer

    5 days ago

    Pune, Maharashtra, India Talent Sketchers Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Designation: Big Data EngineerExperience: 4+ YearsWork Mode: Remote OpportunityNotice Period: Immediate Joiners/ Serving Notice PeriodJob Description:Role: Big Data EngineerThis Data Engineer will be engaged in data science-related research and software application development and engineering duties related to our enterprise-grade Wi-Fi technology to provide...

  • Big Data Engineer

    1 week ago

    Pune, Maharashtra, India Nice Software Solutions Pvt. Ltd. Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Big Data Engineer (PySpark)Location:Pune/Nagpur (WFO)Experience:8 - 12 YearsEmployment Type:Full-timeJob OverviewWe are looking for an experiencedBig Data Engineerwith strong expertise in PySpark and Big Data ecosystems. The ideal candidate will be responsible for designing, developing, and optimizing scalable data pipelines while ensuring high performance...

  • Big Data Engineer

    2 weeks ago

    Pune, Maharashtra, India RiskInsight Consulting Pvt Ltd. Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    ResponsibilitiesDesign, develop, and implement robust Big Data solutions using technologies such as Hadoop, Spark, and NoSQL databasesBuild and maintain scalable data pipelines for effective data ingestion, transformation, and analysisCollaborate with data scientists, analysts, and cross-functional teams to understand business requirements and translate them...

  • Big Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Information Technology Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Big Data EngineerKey Responsibilities : - Build and manage highly scalable Big Data pipelines for large structured datasets. - Collaborate with Product Management & Engineering leadership to design and implement optimal solutions. - Participate in design discussions and contribute to selecting, integrating, and maintaining Big Data frameworks. -...

  • Big Data Engineer

    5 days ago

    Pune, Maharashtra, India Capco Technologies Pvt Ltd Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Big Data Engineer Job Title: Big Data Engineer About Us "Capco, a Wipro company, is a global technology and management consulting firm. Awarded with Consultancy of the year in the British Bank Award and has been ranked Top 100 Best Companies for Women in India 2022 by Avtar & Seramount. With our presence across 32 cities across globe, we support 100...

  • Big Data Engineer

    2 weeks ago

    Pune, Maharashtra, India Information Technology Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Key ResponsibilitiesBuild and manage highly scalable Big Data pipelines for large structured datasets.Collaborate with Product Management & Engineering leadership to design and implement optimal solutions.Participate in design discussions and contribute to selecting, integrating, and maintaining Big Data frameworks.Develop distributed data processing systems...

  • Data Engineer

    5 days ago

    Pune, Maharashtra, India Data Axle Full time ₹ 15,00,000 - ₹ 28,00,000 per year

    About Data Axle:Data Axle Inc. has been an industry leader in data, marketing solutions, sales and research for 50 years in the US. Data Axle has set a strategic global centre of excellence in Pune. This centre delivers mission critical data services to its global customers powered by its proprietary cloud-based technology platform and leveraging proprietary...

  • Big Data Lead

    3 days ago

    Pune, Maharashtra, India Populace World Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Designation: Big Data Lead/ArchitectLocation: PuneExperience: 8-10 yearsNP - immediate joiner/15-30 days' noticeJob OverviewWe are looking to hire a talented big data engineer to develop and manage our company's Big Data solutions. In this role, you will be required to design and implement Big Data tools and frameworks, implement ELT processes, collaborate...

  • Big Data Engineer

    7 days ago

    Pune, Maharashtra, India Talent Worx Full time ₹ 10,00,000 - ₹ 25,00,000 per year

    Location – Pune/ChennaiExp - 6 to 9Technology: - Big Data (Hadoop, Hive, Spark, Scala, PLSQL/ No SQL Database)Knowledge/Experience (Must):· 6-9 years of experience in Hadoop/big data technologies.· Experience with Spark/Storm/Kafka or equivalent streaming/batch processing and eventbased messaging.· Relational and NoSQL database integration and data...