PySpark Developer

3 weeks ago

Hyderabad Bengaluru Chennai, India Coders Brain Technology Private Limited Full time
Job Description

ROLE RESPONSIBILITIES

Data Engineering and Processing:

Develop and manage data pipelines using PySpark on Databricks.

Implement ETL/ELT processes to process structured and unstructured data at scale.

Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.

Databricks Platform Expertise:

Experience in Perform Design, Development & Deployment using Azure Services (Data Factory,

Databricks, PySpark, SQL)

Develop and maintain scalable data pipelines and build new Data Source integrations to support

increasing data volume and complexity.

Leverage the Databricks Lakehouse architecture for advanced analytics and machine learning

workflows.

Manage Delta Lake for ACID transactions and data versioning.

Develop notebooks and workflows for end-to-end data solutions.

Cloud Platforms and Deployment:

Deploy and manage Databricks on Azure (e.g., Azure Databricks).

Use Databricks Jobs, Clusters, and Workflows to orchestrate data pipelines.

Optimize resource utilization and troubleshoot performance issues on the Databricks platform.

CI/CD and Testing:

Build and maintain CI/CD pipelines for Databricks workflows using tools like Azure

DevOps, GitHub Actions, or Jenkins.

Write unit and integration tests for PySpark code using frameworks like Pytest or unittest.

Collaboration and Documentation:

Work closely with data scientists, data analysts, and IT teams to deliver robust data solutions.

Document Databricks workflows, configurations, and best practices for internal use.

TECHNICAL QUALIFICATIONS

Experience:

7+ years of experience in data engineering or distributed systems development.

Strong programming skills in Python and PySpark.(7+ years)

Hands-on experience with Databricks and its ecosystem, including Delta Lake and Databricks

SQL.

Knowledge of big data frameworks like Hadoop, Spark, and Kafka.

Databricks Expertise:

Proficiency in setting up and managing Databricks Workspaces, Clusters, and Jobs.

Familiarity with Databricks MLflow for machine learning workflows is a plus.

Cloud Platforms:

Expertise in deploying Databricks solutions Azure (e.g., Data Lake, Synapse).

Knowledge of Kubernetes for managing containerized workloads is advantageous.

Database Knowledge:

Experience with both SQL (e.g., PostgreSQL, SQL Server) and NoSQL databases

(e.g., MongoDB, Cosmos DB).

GENERAL QUALIFICATIONS

Strong analytical and problem-solving skills.

Ability to manage multiple tasks in a high-intensity, deadline-driven environment.

Excellent communication and organizational skills.

Experience in regulated industries like insurance is a plus.

EDUCATION REQUIREMENTS

A Bachelor's Degree in Computer Science, Data Engineering, or a related field is preferred.

Relevant certifications in Databricks, PySpark, or cloud platforms are highly desirable.

  • PySpark Developer

    4 days ago

    Hyderabad, Bengaluru, Chennai, India Coders Brain Technology Private Limited Full time

    Job Description ROLE RESPONSIBILITIES Data Engineering and Processing: Develop and manage data pipelines using PySpark on Databricks. Implement ETL/ELT processes to process structured and unstructured data at scale. Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks. Databricks Platform Expertise: Experience in...

  • Pyspark Developer

    2 weeks ago

    Hyderabad, Telangana, India Tata Consultancy Services Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Job Title : PySpark SMEJob Location – HyderabadExperience: 4-8 Years of experience in IT industry with exposure to Python, Python , PySpark , Delta Lake , DataStage , Terada ( Bteq ,)Strong in Pyspark and SQL Requirement:Strong knowledge of Pyspark and SQL2 + years of experience in PySpark.4+Years of experience in DataStage, Terada.6+ Years of experience...

  • Pyspark Developer

    6 days ago

    Chennai, Tamil Nadu, India MP DOMINIC AND CO Full time

    Job Summary - Design develop and implement scalable data pipelines and streaming use cases using PySpark and Spark on a distributed computing platform - Possess strong programming skills in Spark streaming - Have familiarity with cloud platforms like GCP - Gain experience in big data technologies such as Hadoop Hive and HDFS - Perform ETL operations...

  • Pyspark Developer

    3 days ago

    Chennai, Hyderabad, Pune, India Tata Consultancy Services Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    TCS is Hiring Location :Bangalore, Chennai, Kolkata, Hyderabad, PuneExp : 4-8 YrsFunctional Skills: Experience in Credit Risk/Regulatory risk domainTechnical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scriptingGood to Have Skills: Exposure to Machine Learning TechniquesJob Description:4+ Years of experience with Developing/Fine tuning...

  • PySpark Developer

    2 weeks ago

    Hyderabad, Telangana, India Tata Consultancy Services Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Job Title: PySpark DeveloperExperience: 6 to 8 YearsLocation: Hyderabad ( Onsite )Job DescriptionMust-HaveStrong knowledge of Pyspark and SQL2 + years of experience in PySpark.4+Years of experience in DataStage, Terada.6+ Years of experience in Python.Experience in GCP cloud ServicesDelta LakeGood-to-HavePysparkGCPGoogle Cloud / Google...

  • PySpark Developer

    2 days ago

    Hyderabad, India Tata Consultancy Services Full time

    Job Title: PySpark Developer Experience: 6 to 8 Years Location: Hyderabad ( Onsite ) Job Description Must-Have Strong knowledge of Pyspark and SQL 2 + years of experience in PySpark. 4+Years of experience in DataStage, Terada. 6+ Years of experience in Python. Experience in GCP cloud Services Delta Lake Good-to-Have Pyspark GCP Google Cloud / Google...

  • Pyspark Developer

    2 weeks ago

    Hyderabad, Telangana, India KloudPortal Technology Solutions PVT. LTD Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Job DescriptionJob Summary We are hiring a Senior PySpark Developer with 4-7 years of experience in building and optimising data pipelines using PySpark on Databricks, within AWS cloud environments. This role involves modernising legacy systems, integrating with Kafka, and collaborating across cross-functional teams.Key ResponsibilitiesDevelop and optimise...

  • PySpark Developer

    2 weeks ago

    Hyderabad, Telangana, India Algoleap Technologies Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    SUMMARY Job SummaryWe are looking for a Senior PySpark Developer with 3 to 6 years of experience in building and optimizing data pipelines using PySpark on Databricks, within AWS cloud environments. This role focuses on the modernization of legacy domains, involving integration with systems like Kafka and collaboration across cross-functional teams.Key...

  • PySpark Developer

    2 days ago

    Hyderabad, India Algoleap Technologies Full time

    SUMMARY Job Summary We are looking for a Senior PySpark Developer with 3 to 6 years of experience in building and optimizing data pipelines using PySpark on Databricks, within AWS cloud environments. This role focuses on the modernization of legacy domains, involving integration with systems like Kafka and collaboration across cross-functional teams. Key...

  • Pyspark Developer

    2 days ago

    Hyderabad, India KloudPortal Technology Solutions PVT. LTD Full time

    Job Description Job SummaryWe are hiring a Senior PySpark Developer with 4-7 years of experience in building and optimising data pipelines using PySpark on Databricks, within AWS cloud environments. This role involves modernising legacy systems, integrating with Kafka, and collaborating across cross-functional teams. Key Responsibilities Develop and optimise...