Senior Big Data Architect

2 weeks ago

Bareilly, Uttar Pradesh, India beBeeDataEngineer Full time ₹ 1,00,00,000 - ₹ 1,50,00,000

Job Title: Data Engineer

Key Responsibilities:

As a skilled Databricks PySpark Developer, you will design, develop and maintain ETL processes to extract, transform and load data from various sources into our data lake and data warehouse.

You will work closely with data engineers, data scientists and business intelligence teams to build and optimize data workflows that support the project's analytics and reporting needs.

  • Data Engineering:
    • Design and develop ETL processes using Databricks PySpark to extract, transform and load data from heterogeneous sources into our data lake and data warehouse.
    • Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
    • Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
    • Design and implement data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver and Gold layers.
    • Ensure data is appropriately cleansed, enriched and optimized at each stage to support robust analytics and reporting.
  • Data Pipeline Management:
    • Hands-on experience in creating advanced data pipelines using databricks workflows.
    • Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity and reliability throughout the ETL lifecycle.
    • Collaborate with data engineering, data science and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.
  • Data Analysis and Query Optimization:
    • Write and optimize complex SQL queries for data manipulation, aggregation and analysis within Databricks PySpark applications.
  • Project Coordination and Continuous Improvement:
    • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
    • Stay updated on the latest developments in Databricks PySpark, Spark SQL and related technologies, recommending and implementing best practices and optimizations.
    • Document ETL processes, data lineage and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.

Required Qualifications:

To be successful as an ETL developer, you will need:

  • Education: A Bachelor's degree in Computer Science, Engineering or a related field.
  • Experience: Minimum of 3 years of experience as an ETL developer, with a strong focus on Databricks PySpark development.
  • Skills: Proficiency in Python programming, extensive experience in developing and debugging Databricks PySpark applications, in-depth understanding of Spark architecture and internals, hands-on experience in Spark RDDs, DataFrames and Spark SQL, expertise in writing and optimizing complex SQL queries for data manipulation, aggregation and analysis.
  • Attributes: Strong problem-solving skills, ability to troubleshoot and resolve ETL process issues, excellent communication and collaboration skills, ability to work effectively in a team environment.

  • Senior Big Data Architect

    2 weeks ago

    Bareilly, Uttar Pradesh, India beBeeData Full time ₹ 18,00,000 - ₹ 25,00,000

    Big Data Solutions ArchitectWe are seeking a highly skilled big data solutions architect with expertise in designing and implementing scalable, high-performance data pipelines.The ideal candidate will have strong knowledge of AWS cloud services, Databricks, and big data frameworks.Main Responsibilities:Design, develop, and maintain complex data pipelines...

  • Big Data Solutions Architect

    6 days ago

    Bareilly, Uttar Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Opening:We are seeking a highly skilled Big Data Solutions Architect to join our team, with a focus on hands-on project delivery and mentoring junior engineers.This role is ideal for a developer who not only thrives in a technical, fast-paced environment but is also passionate about coaching and developing the next generation of talent.You will work on...

  • Chief Big Data Engineer

    7 days ago

    Bareilly, Uttar Pradesh, India beBeeCloudDataEngineer Full time ₹ 10,00,000 - ₹ 15,00,000

    Job Description:">">We are seeking a seasoned Big Data Architect with expertise in designing, developing and implementing large-scale data engineering solutions leveraging AWS services.">">Key Responsibilities:">">Design and develop scalable Big Data architectures using AWS services for complex data processing.">Build and maintain data pipelines using Spark...

  • Senior Data Architect

    1 week ago

    Bareilly, Uttar Pradesh, India Alight Solutions Full time

    Our storyAt Alight, we believe a company's success starts with its people. At our core, we Champion People, help our colleagues Grow with Purpose and true to our name we encourage colleagues to "Be Alight."Our Values:Champion People – be empathetic and help create a place where everyone belongs.Grow with purpose – Be inspired by our higher calling of...

  • Senior Big Data Engineer

    1 week ago

    Bareilly, Uttar Pradesh, India beBeeBigData Full time ₹ 19,44,000 - ₹ 25,20,000

    Job Title:Hadoop Big Data Ecosystem ProfessionalWe are seeking an experienced Hadoop Big Data Ecosystem Professional to take on a challenging role in designing and developing scalable data processing systems using the latest technologies in the Hadoop ecosystem.Key Responsibilities:• Hands-on experience of Scala, Spark, Hive, and other tools within the...

  • Big Data Specialist

    1 week ago

    Bareilly, Uttar Pradesh, India beBeeData Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Opportunity: Big Data SpecialistWe are seeking a highly skilled big data specialist to design, build, and maintain robust data pipelines for processing channel activity data from multiple sources.The ideal candidate will possess strong knowledge of cloud computing, programming languages like Python, and experience in handling data from various sources...

  • Big Data Administrator

    6 days ago

    Bareilly, Uttar Pradesh, India beBeeDataAdministrator Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Big Data Platform AdministratorWe are seeking a highly skilled professional to manage our big data platform.Administer and monitor Big Data Platform components including Hadoop, Hive, HBase, and BigSQL.Support technical team requests for data copies across environments, data cleanup, query tuning, etc.Troubleshoot and resolve issues related to user queries,...

  • Big Data Specialist

    1 week ago

    Bareilly, Uttar Pradesh, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000

    Job Title: Big Data DeveloperWe are seeking an accomplished big data developer to lead our team in delivering scalable data-driven solutions.The ideal candidate will have hands-on experience with Spark, Scala and PySpark, as well as a strong understanding of database systems.Key Responsibilities:Developing efficient big data solutions using data analytics...

  • Chief Data Architect

    6 days ago

    Bareilly, Uttar Pradesh, India beBeeBigData Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    Are you passionate about building scalable and secure data platforms?We are seeking a Lead Big Data Engineer to join our team and contribute to the development of our next-generation Enterprise TruRisk Platform.Job Description:We are looking for a talented engineer to provide technical leadership and expertise in designing, developing, and supporting a...

  • Chief Data Architect

    1 week ago

    Bareilly, Uttar Pradesh, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    About Senior Data Engineer Role:Job Title: Senior Data EngineerWe are seeking an experienced professional to design, develop, and maintain large-scale data pipelines using big data technologies. The successful candidate will be responsible for creating efficient ETL pipelines utilizing big data technologies such as pySpark, Hadoop, and Hive.Key...