Mastering Big Data

5 days ago

Nellore, Andhra Pradesh, India beBeeDataEngineering Full time ₹ 15,00,000 - ₹ 20,10,000

Job Title: Data Engineer

About the Role

We are seeking an experienced Data Engineer to join our team. As a key member of our data engineering group, you will design, develop and maintain large-scale data processing systems to extract, transform, and load data from various sources into a centralized repository.

Main Responsibilities
  • ETL Development:
    • Design and develop efficient ETL processes using Spark to extract, transform, and load data from heterogeneous sources into the data warehouse.
    • Optimize ETL workflows for performance and scalability, leveraging Spark and SQL to efficiently process large data volumes.
    • Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
    • Design and implement data solutions following best practices, organizing data into optimized layers.
    • Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.
  • Data Pipeline Management:
    • Develop and maintain data pipelines using Spark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
    • Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.
  • Data Analysis and Query Optimization:
    • Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Spark applications.
  • Project Coordination and Continuous Improvement:
    • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
    • Stay updated on the latest developments in Spark, recommending and implementing best practices and optimizations.
    • Document ETL processes, data lineage, and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.
    Requirements
    • Bachelor's degree in Computer Science, Engineering, or a related field.
    • Minimum of 3 years of experience as an ETL developer, with a strong focus on Spark development.
    • Proficiency in Python programming, with extensive experience in developing and debugging Spark applications.
    • In-depth understanding of Spark architecture and internals, with hands-on experience in Spark RDDs, DataFrames, and SQL.
    • Expertise in writing and optimizing complex SQL queries for data manipulation, aggregation, and analysis.
    • Proven experience in working with large-scale data warehousing and ETL frameworks.
    • Strong problem-solving skills and the ability to troubleshoot and resolve ETL process issues.
    • Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
    Preferred Qualifications
    • Experience with cloud platforms with a preference for AWS.
    • Experience with data platform tools such as Snowflake and Tableau.
    • Demonstrated ability to implement best practices for ETL processes and data management.
    • Strong understanding of data governance and data quality principles.
    • Relevant certifications in Spark or related technologies.

  • Big Data Developer

    12 hours ago

    Nellore, Andhra Pradesh, India beBeeBigDataSpecialist Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    We are seeking a highly skilled professional with expertise in Big Data technologies.The ideal candidate will have over 5 years of experience in Big Data projects, with a strong understanding of data structures and algorithms.They should be proficient in PySpark/Scala Spark/Java Spark SQL and have a proven track record of implementing data pipelines,...

  • Big Data Professional

    5 days ago

    Nellore, Andhra Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Big Data ProfessionalWe are seeking a highly skilled and experienced Big Data professional to join our team.Job Description:Skill: Big DataYears of Experience: 6 - 12 yearsLocation: Pune/ Mumbai/ ChennaiMust-Have Skills:Pyspark DevelopmentHive AdministrationGood-to-Have Skills:Spark ProgrammingHBase ImplementationData Quality AssuranceAgile...

  • Big Data Solutions Architect

    12 hours ago

    Nellore, Andhra Pradesh, India beBeeData Full time US$ 80,000 - US$ 1,50,000

    Job Title: Data ArchitectRole Summary:We are seeking a skilled Data Architect to design and implement robust Big Data solutions.Key Responsibilities:Design and develop scalable Big Data architectures using cloud-based technologies.Collaborate with cross-functional teams to ensure seamless integration of data systems.Develop and maintain data governance...

  • Highly Skilled Big Data Specialist

    6 hours ago

    Nellore, Andhra Pradesh, India beBeeBigData Full time ₹ 15,00,000 - ₹ 20,00,000

    Big Data Developer Job DescriptionWe are seeking an experienced Big Data Engineer to join our team. The ideal candidate will have a strong background in designing, developing, and maintaining large-scale big data systems.Key Responsibilities:Design and develop scalable, fault-tolerant, and efficient big data solutions using NoSQL databases like Cassandra or...

  • Senior Big Data Solutions Developer

    4 days ago

    Nellore, Andhra Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,10,00,000

    Big Data Engineer Job ">This is a challenging role that involves handling large datasets to drive business decisions. The primary responsibilities include developing, implementing, and maintaining complex data pipelines using various technologies such as PySpark and Hive.">The ideal candidate will have experience in big data engineering with a focus on...

  • Senior Big Data Solutions Architect

    12 hours ago

    Nellore, Andhra Pradesh, India beBeeData Full time ₹ 12,00,000 - ₹ 18,00,000

    Big Data Engineer OpportunityTata Consultancy Services is seeking a skilled professional to fill the role of Big Data Engineer.Job Description:As a Big Data Engineer, you will be responsible for designing, developing, and maintaining large-scale data processing systems using technologies such as Scala, Spark, Hive, and Hadoop.Our ideal candidate will have...

  • Senior Big Data Professional

    7 days ago

    Nellore, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Data Engineer PositionWe are seeking a seasoned Big Data professional to join our team as a Data Engineer. The successful candidate will be responsible for designing, building, and maintaining large-scale data processing systems using Pyspark and Python.Required Skills and QualificationsMandatory Requirements:

  • Big Data Developer

    5 days ago

    Nellore, Andhra Pradesh, India beBeeDataEngineer Full time ₹ 9,00,000 - ₹ 12,00,000

    Job Title: Data ArchitectDescription:Data architects design, build, and maintain the systems that store, process, and retrieve data. They ensure data quality, security, and integrity throughout its lifecycle.Responsibilities:Developing and implementing data architectures to meet business requirementsDesigning and building data pipelines to extract,...

  • Lead Big Data Architect

    12 hours ago

    Nellore, Andhra Pradesh, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    We are looking for an experienced Big Data Engineer to join our team. The ideal candidate will have 5+ years of experience in designing and implementing data pipelines using Azure/AWS Databricks, Python/Scala/Spark/Pyspark, HIVE/HBase/Impala/Parquet.A strong background in SQL and RDBMS is a must, as well as experience with Git/GitHub and continuous...

  • Big Data Engineer

    1 week ago

    Nellore, Andhra Pradesh, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    We're seeking a seasoned Big Data Engineer to design and build large-scale data platforms on Google Cloud.Key Responsibilities:Design robust data pipelines for ingestion, transformation, and analyticsMaintain expertise in GCP ecosystem (mandatory)Work with Big Data tools: Hadoop, Spark, KafkaHands-on experience in Databricks (Unity Catalog, Workflow,...