PySpark Developer

2 weeks ago

Coimbatore, Tamil Nadu, India Corpxcel Consulting Full time

Location : Chennai/Bangalore/Hyderabad/Coimbatore/ Pune

WFO : 3 days Mandatory from the above-mentioned locations.

Role Summary :

We are seeking a highly skilled PySpark Developer with hands-on experience in Databricks to join Companies IT Systems Development unit in an offshore capacity. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on the Databricks Unified Analytics Platform. The ideal candidate will have expertise in big data frameworks, distributed computing, and cloud platforms, with a deep understanding of Databricks architecture. This is an excellent opportunity to work with cutting-edge technologies in a dynamic, fast-paced environment.

Role Responsibilities :

Data Engineering and Processing :

- Develop and manage data pipelines using PySpark on Databricks.

- Implement ETL/ELT processes to process structured and unstructured data at scale.

- Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.

Databricks Platform Expertise :

- Experience in Perform Design, Development & Deployment using Azure Services (Data Factory, Databricks, PySpark, SQL)

- Develop and maintain scalable data pipelines and build new Data Source integrations to support increasing data volume and complexity.

- Leverage the Databricks Lakehouse architecture for advanced analytics and machine learning workflows.

- Manage Delta Lake for ACID transactions and data versioning.

- Develop notebooks and workflows for end-to-end data solutions.

Cloud Platforms and Deployment :

- Deploy and manage Databricks on Azure (e.g., Azure Databricks).

- Use Databricks Jobs, Clusters, and Workflows to orchestrate data pipelines.

- Optimize resource utilization and troubleshoot performance issues on the Databricks platform.

CI/CD and Testing :

- Build and maintain CI/CD pipelines for Databricks workflows using tools like Azure DevOps, GitHub Actions, or Jenkins.

- Write unit and integration tests for PySpark code using frameworks like Pytest or unittest.

Collaboration and Documentation :

- Work closely with data scientists, data analysts, and IT teams to deliver robust data solutions.

- Document Databricks workflows, configurations, and best practices for internal use.

Technical Qualifications :

Experience :

- 4+ years of experience in data engineering or distributed systems development.

- Strong programming skills in Python and PySpark.

- Hands-on experience with Databricks and its ecosystem, including Delta Lake and Databricks SQL.

- Knowledge of big data frameworks like Hadoop, Spark, and Kafka.

Databricks Expertise :

- Proficiency in setting up and managing Databricks Workspaces, Clusters, and Jobs.

- Familiarity with Databricks MLflow for machine learning workflows is a plus.

Cloud Platforms :

- Expertise in deploying Databricks solutions Azure (e.g., Data Lake, Synapse).

- Knowledge of Kubernetes for managing containerized workloads is advantageous.

Database Knowledge :

- Experience with both SQL (e.g., PostgreSQL, SQL Server) and NoSQL databases (e.g., MongoDB, Cosmos DB).

General Qualifications :

- Strong analytical and problem-solving skills.

- Ability to manage multiple tasks in a high-intensity, deadline-driven environment.

- Excellent communication and organizational skills.

- Experience in regulated industries like insurance is a plus.

Education Requirements :

- A Bachelors Degree in Computer Science, Data Engineering, or a related field is preferred.

- Relevant certifications in Databricks, PySpark, or cloud platforms are highly desirable.

(ref:hirist.tech)

  • Expert Cloud Systems Developer

    1 week ago

    Coimbatore, Tamil Nadu, India beBeeInfrastructure Full time ₹ 18,00,000 - ₹ 25,00,000

    Cloud Infrastructure Specialist Job DescriptionAs a Cloud Infrastructure Specialist, you will be responsible for designing, building, and maintaining scalable and secure cloud-based systems.Key Responsibilities:Design and implement cloud architectures using cloud services such as AWS Glue or DataBricksDevelop and deploy cloud-native applications using...

  • Software Developer/Solutions Architect

    1 week ago

    Coimbatore, Tamil Nadu, India beBeeSoftwareEngineer Full time ₹ 25,00,000 - ₹ 35,00,000

    Job Opportunity: Senior Software Engineer/ Technical SpecialistAbout the Role:We are seeking a seasoned professional to lead our data engineering initiatives.This is an exciting opportunity for someone to take on a challenging role, leveraging their expertise in Azure Data Engineering and Data Factory, Synapse Analytics, Databricks, SQL, T-SQL, Python, and...

  • AWS Cloud Data Solutions Specialist

    2 weeks ago

    Coimbatore, Tamil Nadu, India beBeeDataEngineering Full time ₹ 20,00,000 - ₹ 25,00,000

    Our team is seeking a seasoned Data Engineer with expertise in designing, developing, and supporting data pipelines and applications using various technologies such as Python, PySpark, SQL, and AWS services.Key Responsibilities:We are looking for someone to design and implement robust ETL pipelines using PySpark, focusing on performance, scalability, and...

  • Senior Business Intelligence Developer

    2 weeks ago

    Coimbatore, Tamil Nadu, India beBeeData Full time ₹ 15,00,000 - ₹ 25,00,000

    About Data Architecture RolesWe seek a skilled Data Architect to lead our data ecosystem, ensuring seamless integration, accessibility, and reliability across platforms.Key Responsibilities:Design end-to-end data architecture solutions aligned with business needs.Architect, build, and optimize data pipelines and ETL workflows using industry-leading...

  • Senior Cloud Data Engineer

    2 weeks ago

    Coimbatore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 15,00,000

    Role OverviewLead a challenging project to design, develop, and optimize large-scale data processing solutions using PySpark. We seek an expert in PySpark, Python, SQL, and real-time data streaming with hands-on experience in cloud environments.Location: India. Remote.Job Type: ContractKey Responsibilities:Develop, test, and deploy scalable data pipelines in...

  • Cloud Infrastructure Specialist

    2 weeks ago

    Coimbatore, Tamil Nadu, India beBeecloudinfrastructure Full time ₹ 15,00,000 - ₹ 25,00,000

    Cloud Infrastructure SpecialistWe are seeking a skilled Cloud Engineer to join our team. The ideal candidate will have experience with AWS – Glue or DataBricks and Python/Pyspark.Main Responsibilities:Design, build, and maintain scalable cloud-based systems for business growth.Collaborate with cross-functional teams to ensure seamless integration with...

  • Chief Data Architecture Specialist

    2 weeks ago

    Coimbatore, Tamil Nadu, India beBeeDataEngineering Full time ₹ 25,00,000 - ₹ 35,00,000

    Job Title: Data EngineerJob Overview:We are seeking a highly skilled Data Engineer to join our organization in Coimbatore, India.Key Responsibilities:Design and Develop Scalable Data Pipelines: Utilize PySpark, Python, and SQL to create efficient data pipelines that meet business needs.Collaborate with Cross-Functional Teams: Work closely with data...

  • Data Systems Architect

    1 week ago

    Coimbatore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 10,00,000 - ₹ 14,00,000

    Job OverviewWe're seeking a highly skilled Data Systems Architect to design, build, and maintain large-scale data systems on AWS.Key Responsibilities:Design and implement data pipelines using AWS services like S3, Glue, and Redshift.Develop and deploy cloud-based data applications using Python and PySpark.Collaborate with cross-functional teams to ensure...

  • Cloud Data Pipeline Specialist

    2 weeks ago

    Coimbatore, Tamil Nadu, India beBeeDataEngineer Full time US$ 1,50,000 - US$ 1,90,000

    We are seeking a seasoned professional to lead the design and implementation of large-scale data pipelines using GCP services.Job Responsibilities:Design and implement scalable data pipelinesOrchestrate workflows using Apache AirflowProcess and transform data using ETL tools like Pyspark and DataprocIntegrate with APIs and CI/CD pipelinesTroubleshoot and...

  • Unlock Your Potential as a Master Data Architect

    2 weeks ago

    Coimbatore, Tamil Nadu, India beBeeDataEngineer Full time ₹ 15,00,000 - ₹ 25,00,000

    Job Title: Data EngineerWe are seeking an experienced Data Engineer to join our team.This is a critical role that involves ensuring smooth data migrations, maintaining high-quality data, and enabling scalable operations across multiple workloads.The successful candidate will have strong knowledge of data engineering principles, experience with ETL...