Pyspark developer

5 days ago

Bengaluru Chennai Pune, India MFX Infotech Private Limited Full time

Job Description

ROLE SUMMARY

We are seeking a highly skilledPySpark Developerwith hands-on experience inDatabricksto join Sompo's IT Systems Development unit in an offshore capacity. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on theDatabricks Unified Analytics Platform. The ideal candidate will have expertise inbig data frameworks,distributed computing, and cloud platforms, with a deep understanding ofDatabricks architecture. This is an excellent opportunity to work with cutting-edge technologies in a dynamic, fast-paced environment.

ROLE RESPONSIBILITIES

Data Engineering and Processing:

Develop and manage data pipelines usingPySparkonDatabricks.

Implement ETL/ELT processes to process structured and unstructured data at scale.

Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.

Databricks Platform Expertise:

Experience in Perform Design, Development & Deployment using Azure Services (Data Factory, Databricks, PySpark, SQL)

Develop and maintain scalable data pipelines and build new Data Source integrations to support increasing data volume and complexity.

Leverage theDatabricks Lakehouse architecturefor advanced analytics and machine learning workflows.

ManageDelta Lakefor ACID transactions and data versioning.

Developnotebooksand workflows for end-to-end data solutions.

Cloud Platforms and Deployment:

Deploy and manage Databricks on Azure(e.g., Azure Databricks).

Use DatabricksJobs,Clusters, andWorkflowsto orchestrate data pipelines.

Optimize resource utilization and troubleshoot performance issues on the Databricks platform.

CI/CD and Testing:

Build and maintainCI/CD pipelinesfor Databricks workflows using tools likeAzure DevOps,GitHub Actions, orJenkins.

Write unit and integration tests for PySpark code using frameworks likePytestorunittest.

Collaboration and Documentation:

Work closely with data scientists, data analysts, and IT teams to deliver robust data solutions.

Document Databricks workflows, configurations, and best practices for internal use.

TECHNICAL QUALIFICATIONS

Experience:

4+ years of experience in data engineering or distributed systems development.

Strong programming skills inPythonandPySpark.

Hands-on experience withDatabricksand its ecosystem, includingDelta LakeandDatabricks SQL.

Knowledge of big data frameworks likeHadoop,Spark, andKafka.

Databricks Expertise:

Proficiency in setting up and managingDatabricks Workspaces,Clusters, andJobs.

Familiarity with DatabricksMLflowfor machine learning workflows is a plus.

Cloud Platforms:

Expertise in deploying Databricks solutions Azure(e.g., Data Lake, Synapse).

Knowledge ofKubernetesfor managing containerized workloads is advantageous.

Database Knowledge:

Experience with both SQL (e.g.,PostgreSQL,SQL Server) and NoSQL databases (e.g.,MongoDB,Cosmos DB).

GENERAL QUALIFICATIONS

Strong analytical and problem-solving skills.

Ability to manage multiple tasks in a high-intensity, deadline-driven environment.

Excellent communication and organizational skills.

Experience in regulated industries like insurance is a plus.

EDUCATION REQUIREMENTS

ABachelor's Degreein Computer Science, Data Engineering, or a related field is preferred.

Relevant certifications inDatabricks,PySpark, or cloud platforms are highly desirable.

  • Pyspark Developer

    6 days ago

    Chennai, Tamil Nadu, India MP DOMINIC AND CO Full time

    Job Summary - Design develop and implement scalable data pipelines and streaming use cases using PySpark and Spark on a distributed computing platform - Possess strong programming skills in Spark streaming - Have familiarity with cloud platforms like GCP - Gain experience in big data technologies such as Hadoop Hive and HDFS - Perform ETL operations...

  • Pyspark Developer

    4 days ago

    Chennai, Hyderabad, Pune, India Tata Consultancy Services Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    TCS is Hiring Location :Bangalore, Chennai, Kolkata, Hyderabad, PuneExp : 4-8 YrsFunctional Skills: Experience in Credit Risk/Regulatory risk domainTechnical Skills: Spark ,PySpark, Python, Hive, Scala, MapReduce, Unix shell scriptingGood to Have Skills: Exposure to Machine Learning TechniquesJob Description:4+ Years of experience with Developing/Fine tuning...

  • PySpark Developer

    5 days ago

    Hyderabad, Bengaluru, Chennai, India Coders Brain Technology Private Limited Full time

    Job Description ROLE RESPONSIBILITIES Data Engineering and Processing: Develop and manage data pipelines using PySpark on Databricks. Implement ETL/ELT processes to process structured and unstructured data at scale. Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks. Databricks Platform Expertise: Experience in...

  • Pyspark developer

    3 weeks ago

    Bengaluru, Chennai, Pune, India MFX Infotech Private Limited Full time

    Job DescriptionROLE SUMMARYWe are seeking a highly skilledPySpark Developerwith hands-on experience inDatabricksto join Sompo's IT Systems Development unit in an offshore capacity. This role focuses on designing, building, and optimizing large-scale data pipelines and processing solutions on theDatabricks Unified Analytics Platform. The ideal candidate will...

  • Pyspark Developer

    7 days ago

    Bengaluru, Chennai, Kolkata, India Tata Consultancy Services Full time ₹ 1,04,000 - ₹ 1,30,878 per year

    Job Description:5+ Years of experience with Developing/Fine tuning and implementing programs/applicationsUsing Python/PySpark/Scala on Big Data/Hadoop Platform.Roles and Responsibilities:a) Work with a Leading Banks Risk Management team on specific projects/requirements pertaining to risk Models inconsumer and wholesale bankingb) Enhance Machine Learning...

  • PySpark Developer

    3 weeks ago

    Hyderabad, Bengaluru, Chennai, India Coders Brain Technology Private Limited Full time

    Job DescriptionROLE RESPONSIBILITIESData Engineering and Processing:Develop and manage data pipelines using PySpark on Databricks.Implement ETL/ELT processes to process structured and unstructured data at scale.Optimize data pipelines for performance, scalability, and cost-efficiency in Databricks.Databricks Platform Expertise:Experience in Perform Design,...

  • Pyspark Developer

    1 week ago

    Bengaluru, Karnataka, India Synechron Technologies Pvt. Ltd. Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Day-to-Day Activities:Design, develop, and maintain ETL pipelines using PySpark on CDP.Implement and manage data ingestion processes from various sources. Process, cleanse, and transform large datasets using PySpark. Conduct performance tuning and optimization of ETL processes. Implement data quality checks and validation routines. Automate data workflows...

  • Pyspark Developer

    3 days ago

    Bengaluru, India Sigma Allied Services Full time

    Position - Pyspark Developer Location - Bangalore Experience - 6 to 9 yrs Required Skills -Project Role Description : Lead the effort to design, build and configure applications, acting as the primary point of contact. Must have skills : PySparkMinimum 5 year(s) of experience is requiredEducational Qualification : 15 years full time educationSummary:As an...

  • Pyspark

    2 weeks ago

    Bengaluru, India Riverforest Connections Full time

    Position Purpose - For the projects which are implemented on Data SSC platform, we need senior developer for python - pyspark development to migrate project from spark/scala to python-pyspark._ **Responsibilities** **Direct Responsibilities** Design high quality deliverables adhering to business requirement with defined standards and design principles,...

  • Officer-Pyspark Developer

    1 week ago

    Pune, Maharashtra, India Citi Full time ₹ 15,00,000 - ₹ 20,00,000 per year

    AtCitiwe're not just building technology, we're building the future of banking. Encompassing a broad range of specialties, roles, and cultures, our teams are creating innovations used across the globe. Citi is constantly growing and progressing through our technology, with laser focused on evolving the ways of doing things. As one of the world's most global...