Current jobs related to Databricks + Pyspark - Andhra Pradesh - Virtusa

  • Databricks + Pyspark

    2 weeks ago

    Indore, Madhya Pradesh, India Cognizant Full time

    **Job Summary** **Responsibilities** - Lead the development and implementation of clinical data systems using Scala and PySpark to ensure efficient data processing. - Oversee the setup and maintenance of CI/CD pipelines in Azure DevOps to streamline development workflows. - Provide expertise in Azure Repos for version control and collaboration on code...

  • Aws Databricks

    1 week ago

    Andhra Pradesh, India Virtusa Full time

    Must have 7plus years of experience total IT experience in Data engineering DW area Must have 3plus yrs of exposure on AWS ecosystem specifically DataBricks, Redshift Lambda Glue Starbust Minimum 5 yrs of technical expertise with data models data mining and segmentation techniques Expert on programming languages SQL Python Pyspark Must be good at...

  • Cloudera

    23 hours ago

    Andhra Pradesh, India Virtusa Full time

    Job Description: Design, develop, and maintain data pipelines and ETL processes using Databricks. Manage and optimize data solutions on cloud platforms such as Azure and AWS. Implement big data processing workflows using PySpark. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective...

  • Cloudera

    23 hours ago

    Andhra Pradesh, India Virtusa Full time

    Design, develop, and maintain data pipelines and ETL processes using Databricks. Manage and optimize data solutions on cloud platforms such as Azure and AWS. Implement big data processing workflows using PySpark. Collaborate with data scientists, analysts, and other stakeholders to understand data requirements and deliver effective solutions. Ensure data...

  • Dwh Testing

    23 hours ago

    Andhra Pradesh, India Virtusa Full time

    Role: QA Automation Engineer Data Engineering Team **Job Description**: As a QA Automation Engineer, you will be an integral part of the data engineering team, ensuring the quality and reliability of the lakehouse infrastructure and data pipelines. You will design, develop, and implement automated testing solutions to validate data ingestion,...

  • Infrastructure Engineering Lead Analyst

    2 weeks ago

    Andhra Pradesh, India The Cigna Group Full time ₹ 5,00,000 - ₹ 8,00,000 per year

    INTRODUCTION TO EVERNORTH:Evernorth Health Services India, established in Hyderabad in 2024, is an innovation hub for Evernorth Health Services, the pharmacy, care and benefits division of The Cigna Group. The innovation hub will support innovation-focused areas, such as generative AI, product development, process improvement, analytics, and software...

  • Infrastructure Engineering Advisor

    3 days ago

    Andhra Pradesh, India The Cigna Group Full time

    INTRODUCTION TO EVERNORTH: Evernorth Health Services India, established in Hyderabad in 2024, is an innovation hub for Evernorth Health Services, the pharmacy, care and benefits division of The Cigna Group. The innovation hub will support innovation-focused areas, such as generative AI, product development, process improvement, analytics, and software...

  • HIH - Software Engineering Senior Analyst

    1 week ago

    Andhra Pradesh, India The Cigna Group Full time ₹ 5,00,000 - ₹ 8,00,000 per year

    Position OverviewCigna,a leading Health Services company, is looking for data engineers/developers in our Data & Analytics organization. The Full Stack Engineer is responsible for the delivery of a business need end-to end starting from understanding the requirements to deploying the software into production. This role requires you to be fluent in some of...

  • HIH - Software Engineering Senior Analyst

    3 days ago

    Andhra Pradesh, India The Cigna Group Full time

    Position Overview Cigna,a leading Health Services company, is looking for data engineers/developers in our Data & Analytics organization. The Full Stack Engineer is responsible for the delivery of a business need end-to end starting from understanding the requirements to deploying the software into production. This role requires you to be fluent in some of...

  • Data Scientist

    2 weeks ago

    Noida, Uttar Pradesh, India Iris Software Full time

    **Posted On**:11 Oct 2024 **Location**: Noida, UP, India **Company**:Iris Software **Why Join Us?** Are you inspired to grow your career at one of **India’s Top 25 Best Workplaces in IT industry? **Do you want to do the best work of your life at one of the **fastest growing IT services companies**? Do you aspire to thrive in an award-winning work...

Databricks + Pyspark

3 weeks ago

Andhra Pradesh, India Virtusa Full time

**Detailed Job Description for**Databricks + PySpark Developer**:

- Data Pipeline Development: Design, implement, and maintain scalable and efficient data pipelines using PySpark and Databricks for ETL processing of large volumes of data.
- Cloud Integration: Develop solutions leveraging Databricks on cloud platforms (AWS/Azure/GCP) to process and analyze data in a distributed computing environment.
- Data Modeling: Build robust data models, ensuring high-quality data integration and consistency across multiple data sources.
- Optimization: Optimize PySpark jobs for performance, ensuring the efficient use of resources and cost-effective execution.
- Collaborative Development: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver actionable insights.
- Automation & Monitoring: Implement monitoring solutions for data pipeline health, performance, and failure detection.
- Documentation & Best Practices: Maintain comprehensive documentation of architecture, design, and code. Ensure adherence to best practices for data engineering, version control, and CI/CD processes.
- Mentorship: Provide guidance to junior data engineers and help with the design and implementation of new features and components.

**Required Skills & Qualifications**:

- Experience: 6+ years of experience in data engineering or software engineering roles, with a strong focus on PySpark and Databricks.

**Technical Skills**:

- Proficient in PySpark for distributed data processing and ETL pipelines.
- Experience working with Databricks for running Apache Spark workloads in a cloud environment.
- Solid knowledge of SQL, data wrangling, and data manipulation.
- Experience with cloud platforms (AWS, Azure, or GCP) and their respective data storage services (S3, ADLS, BigQuery, etc.).
- Familiarity with data lakes, data warehouses, and NoSQL databases (e.g., MongoDB, Cassandra, HBase).
- Experience with orchestration tools like Apache Airflow, Azure Data Factory, or DBT.
- Familiarity with containerization (Docker, Kubernetes) and DevOps practices.
- Problem Solving: Strong ability to troubleshoot and debug issues related to distributed computing, performance bottlenecks, and data quality.
- Version Control: Proficient in Git based workflows and version control.
- Communication Skills: Excellent written and verbal communication skills, with the ability to explain complex technical concepts to both technical and non-technical stakeholders.
- Education: Bachelor or Master’s degree in Computer Science, Engineering, or a related field (or equivalent practical experience).

**About Virtusa**

Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 27,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.

Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.

Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.