Senior Big Data Developer

3 days ago

Delhi Kolkata Mumbai, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000
Job Summary:

We are seeking an experienced PySpark Data Engineer to drive the development of robust data processing and transformation solutions within our data platform. You will be responsible for designing, implementing, and maintaining PySpark-based applications to handle complex data processing tasks, ensure data quality, and integrate with diverse data sources.

Key Responsibilities:

- Design, develop, and test PySpark-based applications to process, transform, and analyze large-scale datasets from various sources, including relational databases, NoSQL databases, batch files, and real-time data streams.
- Implement efficient data transformation and aggregation using PySpark and relevant big data frameworks.
- Develop robust error handling and exception management mechanisms to ensure data integrity and system resilience within Spark jobs.
- Optimize PySpark jobs for performance, including partitioning, caching, and tuning of Spark configurations.

Data Analysis and Transformation:

- Collaborate with data analysts, data scientists, and data architects to understand data processing requirements and deliver high-quality data solutions.
- Analyze and interpret data structures, formats, and relationships to implement effective data transformations using PySpark.
- Work with distributed datasets in Spark, ensuring optimal performance for large-scale data processing and analytics.

Data Integration and ETL:

- Design and implement ETL (Extract, Transform, Load) processes to ingest and integrate data from various sources, ensuring consistency, accuracy, and performance.
- Integrate PySpark applications with data sources such as SQL databases, NoSQL databases, data lakes, and streaming platforms

Qualifications and Skills:

- Bachelor's degree in Computer Science, Information Technology, or a related field.
- 5+ years of hands-on experience in big data development, preferably with exposure to data-intensive applications.
- Strong understanding of data processing principles, techniques, and best practices in a big data environment.
- Proficiency in PySpark, Apache Spark, and related big data technologies for data processing, analysis, and integration.
- Experience with ETL development and data pipeline orchestration tools (e.g., Apache Airflow, Luigi).
- Strong analytical and problem-solving skills, with the ability to translate business requirements into technical solutions.
- Excellent communication and collaboration skills to work effectively with data analysts, data architects, and other team members.

  • Senior Big Data Engineer

    2 days ago

    Kolkata, Delhi, Mumbai, India beBeeDataEngineer Full time ₹ 12,00,000 - ₹ 30,00,000

    We are currently seeking a highly skilled Data Engineer to lead our data platform development team and drive the creation of robust, scalable data processing solutions.Job Description:Data Engineering Development:Design, develop, and test complex data processing applications using Apache Spark to handle large-scale datasets from various sources, including...

  • Senior Big Data Developer

    4 days ago

    Kolkata, Delhi, Mumbai, India beBeeData Full time US$ 1,50,000 - US$ 2,00,000

    Software Engineer OpportunityJob Description:We are seeking a highly skilled Software Engineer to join our team.Develop, test, and deploy data processing applications using Apache Spark and Scala.Optimize and tune Spark applications for better performance on large-scale data sets.Work with the Cloudera Hadoop ecosystem (e.g., HDFS, Hive, Impala, HBase,...

  • Big Data Engineer

    5 days ago

    Delhi, Mumbai, Kolkata, India beBeeDataEngineer Full time US$ 90,000 - US$ 1,20,000

    Job Title: Data Engineer for Big Data ProcessingWe are seeking an experienced data engineer to drive the development of robust data processing and transformation solutions within our data platform. You will be responsible for designing, implementing, and maintaining PySpark-based applications to handle complex data processing tasks, ensure data quality, and...

  • Big Data Development Lead

    1 day ago

    Mumbai, Maharashtra, India beBeeSpark Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Job Title : Senior Big Data EngineerWe are looking for experienced professionals to lead our big data initiatives. The ideal candidate will have hands-on expertise in big data processing using Apache Spark with Scala, along with proficiency in workflow management tools like Oozie and data flow platforms such as Apache NiFi.Key Responsibilities :Design,...

  • Big Data Developer

    4 weeks ago

    Mumbai, Maharashtra, India HirePower Staffing Solution Full time

    Position Overview: We are seeking a skilled Big Data Developer to join our growing delivery team, with a dual focus on hands-on project support and mentoring junior engineers. This role is ideal for a developer who not only thrives in a technical, fast-paced environment but is also passionate about coaching and developing the next generation of talent. You...

  • Big Data Developer

    6 days ago

    Navi Mumbai, Mumbai City, Mumbai, India Future Focus Infotech Full time

    Job DescriptionOverview:We are seeking a highly motivated and detail-oriented individual to join our team as a Big Data Developer. This role requires a dynamic professional who can adapt to evolving business needs and drive value through their expertise.Key Responsibilities:- Provide support and expertise in the domain of Big Data Developer.- Collaborate...

  • Big Data Developer

    10 hours ago

    Navi Mumbai, Maharashtra, India Smartavya Analytica Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Job Title: Big Data DeveloperLocation: Navi Mumbai, IndiaExp: 5+ YearsDepartment: Big Data and CloudJob Summary: Smartavya Analytica Private Limited is seeking a skilled Hadoop Developer to join our team and contribute to the development and maintenance of large-scale Big Data solutions. The ideal candidate will have extensive experience in Hadoop ecosystem...

  • Senior Big Data Engineer

    19 hours ago

    Mumbai, Maharashtra, India beBeeBigDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Senior Big Data EngineerJob Summary:Data Engineers design, develop and maintain large-scale data processing systems. In this role, you will work with our team to build scalable data pipelines using PySpark and ensure the smooth operation of our data infrastructure.About the Role:The ideal candidate will have strong experience in building data...

  • Senior Big Data Developer

    22 hours ago

    Delhi, Delhi, India beBeeDeveloper Full time ₹ 1,00,00,000 - ₹ 2,00,00,000

    Big Data Development ExpertThis position is open for a seasoned professional with a strong background in designing, developing and maintaining large-scale distributed data pipelines. The ideal candidate will have a deep understanding of Apache Spark and Scala, as well as experience working with UNIX shell scripting, Hive, Sqoop, and Impala.Key...

  • Big Data Developer

    3 weeks ago

    Mumbai, Maharashtra, India Tech Consulting Full time

    Role Overview:We are seeking a highly skilled Big Data & Cloud Technologies Training & Mentorship Specialist to design, deliver, and mentor teams on cutting-edge data engineering, analytics, and cloud practices. The ideal candidate will have deep technical expertise in Azure, AWS, CI/CD pipelines, ETL processes, and modern data platforms like Databricks,...