PySpark Data Solutions Architect

4 days ago

Bellary, Karnataka, India beBeeDataEngineer Full time ₹ 25,00,000 - ₹ 30,00,000
Job Description

We are seeking an experienced Senior Data Engineer to join our team. The ideal candidate will bring hands-on expertise in PySpark, Python, SQL, and real-time data streaming to contribute to the design and implementation of scalable data pipelines in cloud environments.

The selected candidate will be responsible for designing, developing, and optimizing large-scale data processing solutions using PySpark, Spark, and Python. They will also implement ETL/ELT solutions to integrate structured and unstructured data from multiple sources, develop and maintain Hive queries, and write and optimize SQL queries for large datasets.

Additionally, the successful candidate will build, monitor, and support real-time streaming solutions using Apache Kafka, work on Databricks to create, manage, and optimize data workflows, and deploy and manage solutions in Azure cloud services (e.g., Data Lake, Synapse, Azure Databricks).

Strong problem-solving and debugging skills are essential for this role, as well as excellent written and verbal communication skills. Ability to work independently in a fast-paced environment and adapt to new technologies quickly is also required.

">

Key Responsibilities:

  1. Design, develop, and optimize data pipelines using PySpark, Spark, and Python.
  2. Implement ETL/ELT solutions to integrate structured and unstructured data from multiple sources.
  3. Develop and maintain Hive queries and ensure seamless data transformations.
  4. Write and optimize SQL queries for large datasets.
  5. Build, monitor, and support real-time streaming solutions using Apache Kafka.
  6. Work on Databricks to create, manage, and optimize data workflows.
  7. Deploy and manage solutions in Azure cloud services (e.g., Data Lake, Synapse, Azure Databricks).
  8. Debug, troubleshoot, and optimize code for performance and scalability.
  9. Collaborate with cross-functional teams to deliver data-driven solutions.
  10. Adhere to best practices for coding, testing, and deployment.
">Required Skills & Qualifications

Experience: 3–4 years of experience in data engineering with a focus on PySpark and Python.

Knowledge: Strong knowledge of Spark, Hive, SQL, and Unix shell scripting.

Databricks Experience: Experience with Databricks in a production environment.

Azure Cloud Services: Hands-on experience with Azure cloud services (preferred: Data Lake, Synapse, Databricks).

Real-Time Streaming: Experience working with Apache Kafka or other real-time streaming technologies.

Problem-Solving: Strong problem-solving and debugging skills.

Communication: Excellent written and verbal communication skills.

Adaptability: Ability to work independently in a fast-paced environment and adapt to new technologies quickly.

  • Data Architect

    5 days ago

    Bellary, Karnataka, India beBeeDataArchitect Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    Job Title: Data ArchitectRole Overview:We are seeking an experienced Data Architect to lead our team in designing and implementing scalable data systems.The ideal candidate will have a strong background in data engineering, with expertise in PySpark, Python, and SQL.Key Responsibilities:Design and develop scalable data pipelines and systems,Work closely with...

  • Lead Data Solutions Architect

    6 days ago

    Bellary, Karnataka, India beBeeDataArchitect Full time ₹ 25,00,000 - ₹ 35,00,000

    **Job Overview:**We are seeking a highly skilled Lead Data Solutions Architect to design and implement scalable data pipelines using Microsoft Fabric. As a key member of our team, you will be responsible for developing ETL/ELT solutions, managing data storage and compute resources, and integrating data from diverse sources.Responsibilities:Design and...

  • Data Architect Expertise

    4 days ago

    Bellary, Karnataka, India beBeeDataArchitect Full time ₹ 18,00,000 - ₹ 21,00,000

    Job TitleA seasoned data architect is needed to design and implement scalable data solutions on AWS.Key Responsibilities:Design, develop and maintain large-scale data architectures on AWS.Build and optimize data pipelines, warehouses and data lakes using AWS services.Work on data modeling, ETL/ELT processes and performance tuning.Collaborate with...

  • Senior Data Architect Position

    4 days ago

    Bellary, Karnataka, India beBeeDataArchitecture Full time ₹ 15,00,000 - ₹ 20,00,000

    Job Role: Data ArchitectDescription:We are seeking an accomplished Data Architect to design, implement, and manage data pipelines and systems. The ideal candidate will have a strong background in cloud-based infrastructure, data warehousing, and big data technologies.Tech Stack:AWSPythonPySparkSQLKey Requirements:Expertise in cloud-based architecture, data...

  • Chief Data Architect

    2 weeks ago

    Bellary, Karnataka, India beBeeDataScientist Full time ₹ 1,50,00,000 - ₹ 2,00,00,000

    We are seeking a highly skilled Data Architect to join our team. As a key member of the data platform implementation team, you will be instrumental in designing, developing and maintaining ETL processes to ensure efficient extraction, transformation and loading of data from various sources into our data lake and data warehouse.Key Responsibilities:Design and...

  • GCP Data Solutions Architect

    3 days ago

    Bellary, Karnataka, India beBeeDataEngineer Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Data Engineer Position">We are seeking a skilled Data Engineer to design and deploy data pipelines, orchestrate workflows, and integrate with APIs and CI/CD pipelines.">Responsibilities:Designing and implementing data pipelines using GCP technologiesOrchestrating workflows using Apache AirflowProcessing and transforming data using PySpark, Dataflows, and...

  • Chief Data Architect

    4 days ago

    Bellary, Karnataka, India beBeeData Full time ₹ 2,00,00,000 - ₹ 2,50,00,000

    Job TitleWe are seeking a visionary Data Engineer to lead the design and implementation of scalable, high-performance data solutions that support Sales, Marketing, Manufacturing, Finance, and Engineering across a world-class manufacturing organization.This is an exciting opportunity to work with a talented team of professionals who share your passion for...

  • Data Platform Specialist

    5 days ago

    Bellary, Karnataka, India beBeeDataEngineer Full time ₹ 20,00,000 - ₹ 25,00,000

    Senior Data EngineerWe are seeking a seasoned data engineer with expertise in designing and implementing scalable data pipelines using Azure Synapse, Databricks, and Apache Spark (PySpark). The ideal candidate will work closely with data architects, analysts, and product owners to deliver high-quality solutions that meet the needs of our business.Design,...

  • Enterprise Big Data Solutions Architect

    1 week ago

    Bellary, Karnataka, India beBeeData Full time ₹ 1,20,00,000 - ₹ 2,00,00,000

    Big Data Engineer JobGreetings from a leading technology firmWe are seeking an experienced Big Data Engineer to join our team. The ideal candidate will have a minimum of 6 years and maximum of 12 years of experience in big data engineering.Key Responsibilities:Designing, developing, and deploying big data solutions using PySpark and Hive.Working with...

  • Data Governance Solutions Architect

    6 days ago

    Bellary, Karnataka, India beBeeSolutions Full time ₹ 1,50,00,000 - ₹ 2,50,00,000

    ">Solutions Architect">We are seeking a skilled Solutions Architect to design, deploy, and maintain big data-based solutions. As a Solutions Architect, your work will be a combination of hands-on contribution, customer engagement, and technical team management.You will be responsible for:Technical management across the full life cycle of big data-based...