Databricks + Pyspark

3 weeks ago

Chennai Tamil Nadu, India Virtusa Full time

Data Pipeline Development: Design, implement, and maintain scalable and efficient data pipelines using PySpark and Databricks for ETL processing of large volumes of data.
Cloud Integration: Develop solutions leveraging Databricks on cloud platforms (AWS/Azure/GCP) to process and analyze data in a distributed computing environment.
Data Modeling: Build robust data models, ensuring high-quality data integration and consistency across multiple data sources.
Optimization: Optimize PySpark jobs for performance, ensuring the efficient use of resources and cost-effective execution.
Collaborative Development: Work closely with data scientists, analysts, and other stakeholders to understand data requirements and deliver actionable insights.
Automation & Monitoring: Implement monitoring solutions for data pipeline health, performance, and failure detection.
Documentation & Best Practices: Maintain comprehensive documentation of architecture, design, and code. Ensure adherence to best practices for data engineering, version control, and CI/CD processes.
Mentorship: Provide guidance to junior data engineers and help with the design and implementation of new features and components.
- ____________________
Required Skills & Qualifications:
Experience: 6+ years of experience in data engineering or software engineering roles, with a strong focus on PySpark and Databricks.
Technical Skills:
Proficient in PySpark for distributed data processing and ETL pipelines.
Experience working with Databricks for running Apache Spark workloads in a cloud environment.
Solid knowledge of SQL, data wrangling, and data manipulation.
Experience with cloud platforms (AWS, Azure, or GCP) and their respective data storage services (S3, ADLS, BigQuery, etc.).
Familiarity with data lakes, data warehouses, and NoSQL databases (e.g., MongoDB, Cassandra, HBase).
Experience with orchestration tools like Apache Airflow, Azure Data Factory, or DBT.
Familiarity with containerization (Docker, Kubernetes) and DevOps practices.
Problem Solving: Strong ability to troubleshoot and debug issues related to distributed computing, performance bottlenecks, and data quality.
Version Control: Proficient in Git based workflows and version control.
Communication Skills: Excellent written and verbal communication skills, with the ability to explain complex technical concepts to both technical and non-technical stakeholders.
Education: Bachelor or Masters degree in Computer Science, Engineering, or a related field (or equivalent practical experience).

**About Virtusa**

Teamwork, quality of life, professional and personal development: values that Virtusa is proud to embody. When you join us, you join a team of 30,000 people globally that cares about your growth — one that seeks to provide you with exciting projects, opportunities and work with state of the art technologies throughout your career with us.

Great minds, great potential: it all comes together at Virtusa. We value collaboration and the team environment of our company, and seek to provide great minds with a dynamic place to nurture new ideas and foster excellence.

Virtusa was founded on principles of equal opportunity for all, and so does not discriminate on the basis of race, religion, color, sex, gender identity, sexual orientation, age, non-disqualifying physical or mental disability, national origin, veteran status or any other basis covered by appropriate law. All employment is decided on the basis of qualifications, merit, and business need.

  • Pyspark+databricks

    3 months ago

    Chennai, Tamil Nadu, India Cognizant Full time

    **Exp: 4 to 13 years** **Skill: Data Bricks+Pyspark** **Location : Bangalore/Hyderabad/Kolkota/Pune/Chennai** **Technical Skills**: Python,PySpark,Azure Data Lake Store,Databricks Workflows,Databricks SQL **Responsibilities**: - Develop and optimize data solutions using Azure Data Lake Store to support business needs. - Utilize Python to create...

  • Pyspark Dev

    7 months ago

    Chennai, Tamil Nadu, India Virtusa Full time

    Pyspark Dev & QA Mandatory Skills BIG Data technology mentioned below Hadoop / Big Data (HDFS, PYTHON, SPARK-SQL, MapReduce) with PySpark Build CI/CD pipelines is required Outstanding coding, debugging and analytical skills Spark APIs to cleanse, explore, aggregate, transform, store & analyse available data Knowledge of installing, configuring, debugging...

  • Databricks - de / Sde

    6 months ago

    Chennai, Tamil Nadu, India Tiger Analytics Full time

    Strong in Pyspark, Python, PLSQL, SQL - Desirable to have ETL with batch and streaming (Kinesis). - Build the solution for optimal extraction, transformation, and loading of data from a wide variety of data sources using data ingestion and transformation components. The following technology skills are required - Advanced working SQL knowledge and experience...

  • Senior Data Engineer

    3 weeks ago

    Chennai, Tamil Nadu, India Hexaware Technologies Full time

    Job Summary:We are seeking a highly skilled Senior Data Engineer to join our team in Chennai, Bengaluru, Mumbai, or Pune. As an Azure Data Engineer, you will design, implement, and maintain data pipelines using Databricks, Data Factory, SQL, and Pyspark/Spark.About the Role:Collaborate with cross-functional teams to understand data requirements and optimize...

  • Senior Databricks Engineer/Databricks Technical Lead/Data Architect

    2 days ago

    Chennai, India VIDPRO CONSULTANCY SERVICES Full time

    Role : Senior Databricks Engineer / Databricks Technical Lead/ Data ArchitectExperience : 3-15 yearsLocation : ChennaiWe are seeking an experienced data scientist who apart from the required mathematical and statistical expertise also possesses the natural curiosity and creative mind to ask questions, connect the dots, and uncover opportunities that lie...

  • Senior Developer

    3 weeks ago

    Chennai, India C2E Consultancy Full time

    Essential Job Functions :- Responsible for design of application considering the cost and best practices- Must be willing to self-learn new technologies, become SMEs, and develop high-quality code in a fast-paced environment.- Strong backend experience, databricks background required.- Mentor junior developers and be hands-on in development work.- Work with...

  • Pyspark developer

    2 weeks ago

    Chennai, India Tata Consultancy Services Full time

    Greetings from TCS!! TCS is hiring for Data Engineer (Pyspark Developer) Required Skill Set: Pyspark, Python (Pandas and Numpy), Azure Synapse/ADF/Databricks , SQL and relational databases, Spark SQL, Spark Scripting, UNIX Shell Scripting, ETL, Data Warehousing, Dev Ops (CI/CD) Desired Experience Range: 6 to 10 Years Job Location: Chennai / Pune...

  • Pyspark Developer

    1 month ago

    chennai, India Tata Consultancy Services Full time

    Greetings from TCS!!TCS is hiring for Data Engineer (Pyspark Developer)Required Skill Set: Pyspark, Python (Pandas and Numpy), Azure Synapse/ADF/Databricks , SQL and relational databases, SparkSQL, Spark Scripting, UNIX Shell Scripting, ETL, Data Warehousing, DevOps (CI/CD)Desired Experience Range: 6 to 10 YearsJob Location: Chennai / PuneMust Have:Data...

  • Pyspark Developer

    2 months ago

    Chennai, India Tata Consultancy Services Full time

    Greetings from TCS!!TCS is hiring for Data Engineer (Pyspark Developer)Required Skill Set: Pyspark, Python (Pandas and Numpy), Azure Synapse/ADF/Databricks , SQL and relational databases, SparkSQL, Spark Scripting, UNIX Shell Scripting, ETL, Data Warehousing, DevOps (CI/CD)Desired Experience Range: 6 to 10 YearsJob Location: Chennai / PuneMust Have:Data...

  • Pyspark Developer

    2 months ago

    Chennai, India Tata Consultancy Services Full time

    Greetings from TCS!!TCS is hiring forData Engineer (Pyspark Developer)Required Skill Set:Pyspark, Python (Pandas and Numpy), Azure Synapse/ADF/Databricks , SQL and relational databases, SparkSQL, Spark Scripting, UNIX Shell Scripting, ETL, Data Warehousing, DevOps (CI/CD)Desired Experience Range:6 to 10 YearsJob Location:Chennai / PuneMust Have:Data...

  • Pyspark developer

    1 month ago

    Chennai, India Tata Consultancy Services Full time

    Greetings from TCS!!TCS is hiring for Data Engineer (Pyspark Developer)Required Skill Set: Pyspark, Python (Pandas and Numpy), Azure Synapse/ADF/Databricks , SQL and relational databases, Spark SQL, Spark Scripting, UNIX Shell Scripting, ETL, Data Warehousing, Dev Ops (CI/CD)Desired Experience Range: 6 to 10 YearsJob Location: Chennai / PuneMust...

  • Azure Databricks

    6 months ago

    Chennai, Tamil Nadu, India Cognizant Full time

    **Technical Lead** **Qualification**: Bachelors in science, engineering or equivalent ** Responsibility**: **Project Planning and Setup**: - Understand the project scope, identify activities/ tasks, task level estimates, schedule, dependencies, risks and provide inputs to Module Lead for review. - Provide inputs to testing strategy, configuration,...

  • Pyspark Developer

    2 months ago

    Chennai, India Tata Consultancy Services Full time

    Greetings from TCS!!TCS is hiring for Data Engineer (Pyspark Developer)Required Skill Set: Pyspark, Python (Pandas and Numpy), Azure Synapse/ADF/Databricks , SQL and relational databases, SparkSQL, Spark Scripting, UNIX Shell Scripting, ETL, Data Warehousing, DevOps (CI/CD)Desired Experience Range: 6 to 10 YearsJob Location: Chennai / PuneMust Have:Data...

  • Pyspark Developer

    2 months ago

    Chennai, India Tata Consultancy Services Full time

    Greetings from TCS!! TCS is hiring for Data Engineer (Pyspark Developer) Required Skill Set: Pyspark, Python (Pandas and Numpy), Azure Synapse/ADF/Databricks , SQL and relational databases, SparkSQL, Spark Scripting, UNIX Shell Scripting, ETL, Data Warehousing, DevOps (CI/CD) Desired Experience Range: 6 to 10 Years Job Location: Chennai / Pune Must Have:...

  • Pyspark Developer

    2 months ago

    Chennai, India Tata Consultancy Services Full time

    Greetings from TCS!!TCS is hiring for Data Engineer (Pyspark Developer)Required Skill Set: Pyspark, Python (Pandas and Numpy), Azure Synapse/ADF/Databricks , SQL and relational databases, SparkSQL, Spark Scripting, UNIX Shell Scripting, ETL, Data Warehousing, DevOps (CI/CD)Desired Experience Range: 6 to 10 YearsJob Location: Chennai / PuneMust Have:Data...

  • Director of Databricks Engineering

    4 weeks ago

    Chennai, Tamil Nadu, India LatentView Analytics Full time

    At LatentView Analytics, we are seeking an experienced Director of Databricks Engineering to lead our data engineering team and drive the design, implementation, and optimization of our Databricks platform. This role requires a strong technical background in Databricks, Python, Big Data, Apache Spark, SQL, and Spark SQL.The ideal candidate will have at least...

  • Azure Databricks

    2 weeks ago

    Chennai, India Hexaware Technologies Full time

    Experience: 4.10 years to 9 yearsNotice Period: Immediate to 60 daysInterview: Saturday(21st Dec 2024)- Face to Face DiscussionNo. of rounds: 2 RoundsLocation:Hexaware Technologies, SIPCOT IT Park, Navalur Siruseri Chennai, Tamil Nadu, 601301JD:• Solid Hands-on experience with Azure Databricks - Pyspark coding and Spark SQL coding - Must have• Solid...

  • Colan Infotech

    3 months ago

    Chennai, India Colan Infotech Pvt Ltd Full time

    Role : Azure Data EngineerExperience : 5 Years Skill Set : Pyspark / Scala Spark, Data Factory, Databricks, Python, SQL. - Must have cloud knowledge in AzureRoles and Responsibilities :- Should have programming skills with the ability to write optimized and reusable high-quality code.- Design, develop and maintain scalable data pipelines using Pyspark /...

  • Colan Infotech

    1 month ago

    Chennai, India Colan Infotech Pvt Ltd Full time

    Role : Azure Data EngineerExperience : 5 Years Skill Set : Pyspark / Scala Spark, Data Factory, Databricks, Python, SQL. - Must have cloud knowledge in AzureRoles and Responsibilities :- Should have programming skills with the ability to write optimized and reusable high-quality code.- Design, develop and maintain scalable data pipelines using Pyspark /...

  • Azure Databricks

    2 weeks ago

    Chennai, India Hexaware Technologies Full time

    Experience: 4.10 years to 9 yearsNotice Period: Immediate to 60 daysInterview: Saturday(21st Dec 2024)- Face to Face DiscussionNo. of rounds: 2 RoundsLocation: Hexaware Technologies, SIPCOT IT Park, Navalur Siruseri Chennai, Tamil Nadu, 601301JD:• Solid Hands-on experience with Azure Databricks - Pyspark coding and Spark SQL coding - Must have• Solid...