PySpark Developer

4 days ago

Bengaluru Bangalore Chennai Pune, India beBeeData Full time US$ 1,04,000 - US$ 1,30,878

Job Summary

We are seeking a highly skilled PySpark expert with hands-on experience in Databricks to join our data engineering team. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on the Databricks Unified Analytics Platform.

Key Responsibilities:
  • Develop and manage data pipelines using PySpark on Databricks.
  • Implement ETL/ELT processes to process structured and unstructured data at scale.
  • Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.
Databricks Expertise:
  • Experience in designing, developing, and deploying cloud-based data solutions using Azure Services (Data Factory, Databricks, PySpark, SQL).
  • Develop and maintain scalable data pipelines and build new Data Source integrations to support increasing data volume and complexity.
  • Leverage the Databricks Lakehouse architecture for advanced analytics and machine learning workflows.
Cloud Platforms and Deployment:
  • Deploy and manage Databricks on Azure (e.g., Azure Databricks).
  • Use Databricks Jobs, Clusters, and Workflows to orchestrate data pipelines.
  • Optimize resource utilization and troubleshoot performance issues on the Databricks platform.
Required Skills and Qualifications:
  • Strong programming skills in Python and PySpark.
  • Hands-on experience with Databricks and its ecosystem, including Delta Lake and Databricks SQL.
  • Knowledge of big data frameworks like Hadoop, Spark, and Kafka.
  • Proficiency in setting up and managing Databricks Workspaces, Clusters, and Jobs.
  • Familiarity with Databricks MLflow for machine learning workflows is a plus.
Education Requirements:
  • A Bachelor's Degree in Computer Science, Data Engineering, or a related field is preferred.
  • Relevant certifications in Databricks, PySpark, or cloud platforms are highly desirable.
Benefits:
  • Opportunity to work with cutting-edge technologies in a dynamic, fast-paced environment.
  • Collaboration with experienced professionals in the field of data engineering.
  • Professional growth and development opportunities.

  • Pyspark Developer

    1 day ago

    Bengaluru, Chennai, Pune, India Tekskills Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    JOB SUMMARY:Experience 6+years (relevant minimum 5+ years)Detailed job description - Skill Set:Candidate should have good hands on experience in Pyspark preferrable more than 7 years.Should have very good understanding of Agile development methodologies and excellent team handling experience.Mandatory Skills - Pyspark, Python, Agile

  • Pyspark Developer

    1 day ago

    Bengaluru, Chennai, Pune, India VHR Solutions Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Job Description:We are looking for a skilled PySpark Developer to join our data engineering team. The ideal candidate will have strong experience in building scalable data pipelines using Apache Spark (PySpark), integrating with big data platforms, and working with large datasets in distributed environments. You will work closely with data engineers,...

  • Pyspark Developer

    1 day ago

    Bengaluru, Chennai, Pune, India Quess Corp Limited Full time US$ 1,04,000 - US$ 1,30,878 per year

    ROLE SUMMARYWe are seeking a highly skilled PySpark Developer with hands-on experience in Databricks IT Systems Development unit in an offshore capacity. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on the Databricks Unified Analytics Platform. The ideal candidate will have expertise in big data...

  • Pyspark Developer

    1 day ago

    Pune, Maharashtra, India Tech Mahindra Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Pyspark DeveloperRequirements:Mandatory: Primary skill: Pyspark, Data Engineering, Azure Data BricksGood Experience of Hadoop, Hive, and Cloudera/ Azure/GCP 3+ years of experience in the design and implementation of Big Data systems using PySpark, database migration, transformation and integration solutions for any Data warehousing project.Must have...

  • Senior Software Developer

    4 days ago

    Bengaluru / Bangalore, Chennai, India beBeeSoftwareEngineer Full time US$ 1,20,000 - US$ 2,00,000

    Job Description:As a seasoned software engineer, you will be responsible for designing, developing, and implementing complex software solutions using PySpark.You will work collaboratively with cross-functional teams to deliver high-quality software products that meet business needs.Key responsibilities include producing scalable and efficient software,...

  • Pyspark developer

    5 days ago

    Bengaluru, Chennai, Pune, India MFX Infotech Private Limited Full time

    Job DescriptionROLE SUMMARYWe are seeking a highly skilledPySpark Developerwith hands-on experience inDatabricksto join Sompo's IT Systems Development unit in an offshore capacity. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on theDatabricks Unified Analytics Platform. The ideal candidate will...

  • PySpark Developer

    7 days ago

    Hyderabad, Bengaluru, Chennai, India Coders Brain Technology Private Limited Full time

    Job DescriptionROLE RESPONSIBILITIESData Engineering and Processing:Develop and manage data pipelines using PySpark on Databricks.Implement ETL/ELT processes to process structured and unstructured data at scale.Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.Databricks Platform Expertise:Experience in Perform Design,...

  • PySpark Developers

    1 day ago

    Chennai, Tamil Nadu, India LTIMindtree Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    Skill : PySpark DeveloperJob Locations : Chennai, PuneNotice Period : AnyExperience : 3-8 years Job Description :PySpark DeveloperMandatory Skills : (Apache Spark, Big Data Hadoop Ecosystem, SparkSQL, Python)A good professional experience in Bigdata PySpark HIVE Hadoop PLSQLGood knowledge of AWS and SnowflakeGood understanding of CICD and system design...

  • Lead Data Engineer

    1 week ago

    Chennai, Bengaluru / Bangalore, Pune, India beBeePyspark Full time ₹ 15,00,000 - ₹ 20,00,000

    Job OverviewWe are seeking an experienced Lead Data Engineer to join our team. The ideal candidate will have a strong background in Pyspark and Python, with at least 6 years of experience.Main Responsibilities:Lead the development of data engineering projects using Pyspark and PythonCollaborate with cross-functional teams to design and implement data...

  • Pyspark Developer

    1 day ago

    Bengaluru, Karnataka, India Sigma Allied Services Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Position - Pyspark DeveloperLocation - BangaloreExperience - 6 to 9 yrsRequired Skills -Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact.Must have skills : PySparkMinimum 5 year(s) of experience is requiredEducational Qualification : 15 years full time educationSummary:As an...