Chief Data Architect

4 days ago

Chennai, Tamil Nadu, India beBeeData Full time ₹ 20,00,000 - ₹ 25,00,000
Job Title: Data Engineer

This role requires a skilled professional to design and develop data processing systems, utilizing Databricks PySpark to extract, transform, and load data from various sources into our data lake and data warehouse.

As a member of our team, you will work closely with data engineers, data scientists, and business intelligence teams to build and optimize data workflows that support project analytics and reporting needs.

Key Responsibilities:
  • ETL Development:
    • Design and develop ETL processes using Databricks PySpark to extract, transform, and load data from heterogeneous sources into our data lake and data warehouse.
    • Optimize ETL workflows for performance and scalability, leveraging Databricks PySpark and Spark SQL to efficiently process large data volumes.
    • Implement robust error handling and monitoring mechanisms to proactively detect and resolve issues within ETL processes.
    • Design and implement data solutions following the Medallion Architecture principles, organizing data into Bronze, Silver, and Gold layers.
    • Ensure data is appropriately cleansed, enriched, and optimized at each stage to support robust analytics and reporting.
  • Data Pipeline Management:
    • Hands-on experience in creating advanced data pipelines using databricks workflows.
    • Develop and maintain data pipelines using Databricks PySpark, ensuring data quality, integrity, and reliability throughout the ETL lifecycle.
    • Collaborate with data engineering, data science, and business intelligence teams to translate data requirements into efficient ETL workflows and pipelines.
  • Data Analysis and Query Optimization:
    • Write and optimize complex SQL queries for data manipulation, aggregation, and analysis within Databricks PySpark applications.
  • Project Coordination and Continuous Improvement:
    • Participate in project planning and coordination activities to ensure timely delivery of ETL solutions.
    • Stay updated on the latest developments in Databricks PySpark, Spark SQL, and related technologies, recommending and implementing best practices and optimizations.
    • Document ETL processes, data lineage, and metadata to facilitate knowledge sharing and ensure compliance with data governance standards.
Requirements:
  • Bachelor's degree in Computer Science, Engineering, or a related field.
  • Minimum of 3 years of experience as an ETL developer, with a strong focus on Databricks PySpark development.
  • Proficiency in Python programming, with extensive experience in developing and debugging Databricks PySpark applications.
  • In-depth understanding of Spark architecture and internals, with hands-on experience in Spark RDDs, DataFrames, and Spark SQL.
  • Expertise in writing and optimizing complex SQL queries for data manipulation, aggregation, and analysis.
  • Proven experience in working with large-scale data warehousing and ETL frameworks.
  • Strong problem-solving skills and the ability to troubleshoot and resolve ETL process issues.
  • Excellent communication and collaboration skills, with the ability to work effectively in a team environment.
PREFERRED QUALIFICATIONS:
  • Experience with cloud platforms with a preference for AWS.
  • Experience with data platform tools such as DataBricks, Snowflake, and Tableau.
  • Demonstrated ability to implement best practices for ETL processes and data management.
  • Strong understanding of data governance and data quality principles.
  • Relevant certifications in Databricks PySpark, Spark SQL, or related technologies.

  • Chief Security Architect

    4 days ago

    Chennai, Tamil Nadu, India beBeeSecurity Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Chief Security ArchitectWe are seeking a highly skilled Chief Security Architect to join our organization. As a key member of the team, you will be responsible for designing and implementing robust security measures to protect our systems and data.The ideal candidate will have a strong background in cloud security, with experience working with AWS...

  • Chief Solutions Architect

    3 days ago

    Chennai, Tamil Nadu, India beBeeArchitect Full time ₹ 1,04,000 - ₹ 1,30,878

    Chief Solutions Architect PositionWe are seeking a highly skilled and experienced Chief Solutions Architect to join our team.Main Responsibilities:We require you to design and implement software solutions that enhance knowledge of Spring Boot, Java 8 features, and NoSQL databases like MongoDB.You will improve expertise in messaging queues and distributed...

  • System Architect for Data Innovation

    24 hours ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job Title: Chief Data ArchitectAt Siemens, we seek a highly skilled Chief Data Architect to join our Product Sustainability team. In this role, you will be at the forefront of designing and optimizing data use cases through the strategic use of advanced data technologies.Key Responsibilities:Design robust and scalable architecture for data use cases using...

  • Chief Data Architect

    2 days ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 8,00,000 - ₹ 12,00,000

    About this roleA comprehensive data engineering transformation is underway, and we're seeking a talented individual to spearhead the project.This is an opportunity to architect, build, and implement a complete data ecosystem, encompassing data architecture, ETL pipelines, governance frameworks, and quality assurance systems.You'll collaborate closely with...

  • Chief Technology Visionary

    17 hours ago

    Chennai, Tamil Nadu, India beBeeVisionary Full time ₹ 35,00,000 - ₹ 45,00,000

    Job Title: Chief Technology VisionaryA highly accomplished technology professional with 15+ years of experience, we are seeking a seasoned Chief Technology Visionary to spearhead the design and implementation of comprehensive solutions.About the Role: As our ideal candidate, you will be responsible for architecting and implementing end-to-end solutions that...

  • Chief Software Architect

    19 hours ago

    Chennai, Tamil Nadu, India Banyan Software Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    About Loan Vision Loan Vision, a leading provider of financial software for mortgage lenders, empowers over 250 institutions with advanced loan-level accounting and business intelligence tools. Backed by Banyan Software since 2022, Loan Vision is scaling rapidly with a focus on cloud transformation, AI innovation, and platform modernization. Our flagship...

  • Chief Data Architect

    4 days ago

    Chennai, Tamil Nadu, India beBeeDataModeling Full time ₹ 1,04,000 - ₹ 1,30,878

    Senior Data Modeler Job Description We are seeking a highly skilled Senior Data Modeler to join our team. The ideal candidate will have extensive experience in data modeling, SQL, metadata management, and data governance. The successful candidate will have expertise in relational, dimensional, and analytic data modeling with hands-on experience in databases...

  • Chief Data Systems Architect

    21 hours ago

    Chennai, Tamil Nadu, India beBeeFinancial Full time US$ 2,00,000 - US$ 2,50,000

    Immersive Financial ArchitectThis role calls for a visionary and experienced software architect who can design and develop cutting-edge platforms for financial data.Key responsibilities will include conceptualizing and developing data-driven systems, providing technical guidance to teams, and mentoring junior engineers. The ideal candidate will have 8+ years...

  • Salesforce Technical Architect

    21 hours ago

    Chennai, Tamil Nadu, India NTT DATA Full time US$ 1,50,000 - US$ 2,00,000 per year

    NTT DATA strives to hire exceptional, innovative and passionate individuals who want to grow with us. If you want to be part of an inclusive, adaptable, and forward-thinking organization, apply now. We are currently seeking a Salesforce Technical Architect to join our team in Noida\Bengaluru\Chennai\Hyderabad, Karnataka, India. Job Summary: We are seeking...

  • Chief Data Architect

    4 days ago

    Chennai, Tamil Nadu, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Data Engineer Job DescriptionWe are seeking a skilled Data Engineer to play a crucial role in helping us deliver a better future for everyone.The ideal candidate will have excellent knowledge on implementing full life cycle of data management principles such as data governance, architecture, modelling, storage, security, master data, and quality.Key...