PySpark Developer

3 weeks ago


Kolkata, India Corporate Resources Full time
Job Description

1. Hands on experience in MS Azure Data bricks and Hadoop ecosystem components like DBFS, Parquet, Delta Tables, HDFS, Map Reduce programming, Kafka, Spark & Event Hub.2. In depth understanding of Spark Architecture including Spark Core, Spark SQL, Data Frames, Spark Streaming, RDD caching, Spark MLib.3. Hands on experience in Scripting languages like Scala & Python.4. Hands on experience in Analysis, Design, Coding & Testing phases of SDLC with best practices.5. Expertise in using Spark SQL with various data sources like JSON, Parquet and Key Value Pair.6. Experience in creating tables, partitioning, bucketing, loading and aggregating data using Spark SQL/Scala.7. Migrating the code from Traditional DW Environments to Apache Spark and Scala using Spark SQL, RDD.8. Experience in transferring data from RDBMS/BLOB/ADLS to Data bricks using ADF.9. Experience in Azure Database (PaaS) OR Azure SQL Data warehouse.10. Experience in orchestrating the E2E data loads using the Azure Data Factory V2 / Logic Apps.11. Experience in Cosmos DB.12. Good knowledge on SQL Queries & Stored Procedures.IndustryIT-Software / Software ServicesFunctional AreaIT Software - Application Programming, MaintenanceJob RoleSoftware DeveloperJob TypePermanentQualification

UG QualificationB.Tech/B.E - ComputersPG QualificationAny Post Graduate - Any SpecializationDoctorateDoctorate Not Required - NoneDesired Candidate Profile

Please refer to the Job description above

  • Bangalore/Chennai/Pune/Gurgaon/Gurugram/Kolkata, India Pylon Management Consulting Full time

    Primary Roles and Responsibilities:- Developing Modern Data Warehouse solutions using Databricks and Azure Stack- Ability to provide solutions that are forward-thinking in data engineering and analytics space- Triage issues to find gaps in existing pipelines and fix the issues- Work with business to understand the need in reporting layer and develop data...

  • AWS Data Engineer

    1 month ago


    Bangalore,Pune,Kolkata,Mumbai,Chennai, India connect io Full time

    Role : AWS Data Engineer Mandatory Skills : 5+ Years in AWS Services Hands on exp in Glue and Pyspark. Experience in Python. Hands on exp in Airflow/Step Function. Exposure Lambda, API : Creating data models that can be used to extract information from various sources and store it in a usable format Maintaining the integrity of data by designing backup...

  • Senior Data Engineer

    4 weeks ago


    Kolkata, India Nilasu consulting Full time

    Summary :- We are seeking a highly skilled and experienced Senior Data Engineer to join our team.- You will play a key role in designing, developing, and maintaining large-scale data processing pipelines using Python and Spark/PySpark.- Your expertise in distributed computing frameworks and DevOps tools will be instrumental in building efficient and scalable...

  • Senior Data Engineer

    3 weeks ago


    Bangalore,Kolkata,Pune, India Nilasu consulting Full time

    Summary :- We are seeking a highly skilled and experienced Senior Data Engineer to join our team.- You will play a key role in designing, developing, and maintaining large-scale data processing pipelines using Python and Spark/PySpark.- Your expertise in distributed computing frameworks and DevOps tools will be instrumental in building efficient and scalable...

  • Senior Data Engineer

    3 weeks ago


    Bangalore/Kolkata/Pune, IN Nilasu consulting Full time

    Summary :- We are seeking a highly skilled and experienced Senior Data Engineer to join our team.- You will play a key role in designing, developing, and maintaining large-scale data processing pipelines using Python and Spark/PySpark.- Your expertise in distributed computing frameworks and DevOps tools will be instrumental in building efficient and scalable...

  • Data Ingestion

    4 weeks ago


    Kolkata, India IKSEN INDIA PVT LTD Full time

    **Technical Lead - Data Ingestion** **Requirements and Skills**: - Overall 9+ years of experience in developing large scale data pipelines in a cloud/on-Prem environment. - 3-4 years of experience in Cloud projects - Minimum of 2 years of hands-on experience with AWS technology - Minimum 4 projects successfully implemented - Highly Proficient in any or...


  • Kolkata, India Genpact Full time

    A Data Architectwho can design and implement data modernization solutions in the cloud is responsible for developing and implementing data architecture strategies for organizations transitioning their data systems to the cloud. They collaborate with stakeholders to understand business requirements and translate them into scalable, secure, and high-performing...

  • Data Scientist

    4 weeks ago


    Kolkata, India Cognizant Full time

    **- - Use the knowledge of wide variety of AI/ML techniques and algorithms to find what combinations of these techniques can best solve the problem - Improve Model accuracy to deliver greater business impact - Estimate business impact due to deployment of model - Working with tools and scripts for sufficiently pre-processing the data & feature engineering...

  • Senior Consultant

    4 weeks ago


    Bangalore/Gurgaon/Gurugram/Pune/Chennai/Kolkata, IN VIDPRO CONSULTANCY SERVICES Full time

    Primary Roles and Responsibilities :- Developing Modern Data Warehouse solutions using Databricks and Azure Stack - Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development requirements.- Triage issues to find gaps in existing pipelines and fix the...

  • Senior Consultant

    1 month ago


    Bangalore,Gurgaon,Gurugram,Pune,Chennai,Kolkata, India VIDPRO CONSULTANCY SERVICES Full time

    Primary Roles and Responsibilities : - Developing Modern Data Warehouse solutions using Databricks and Azure Stack - Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development requirements.- Triage issues to find gaps in existing pipelines and fix...

  • Azure Data Engineer

    2 weeks ago


    Bangalore/Pune/Kolkata/Chennai/Gurgaon/Gurugram, India VIDPRO CONSULTANCY SERVICES Full time

    Primary Roles and Responsibilities: - Developing Modern Data Warehouse solutions using Databricks and Azure Stack - Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development requirements.- Triage issues to find gaps in existing pipelines and fix the...

  • Azure Data Engineer

    2 weeks ago


    Bangalore/Pune/Kolkata/Chennai/Gurgaon/Gurugram, IN VIDPRO CONSULTANCY SERVICES Full time

    Primary Roles and Responsibilities:- Developing Modern Data Warehouse solutions using Databricks and Azure Stack - Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development requirements.- Triage issues to find gaps in existing pipelines and fix the...

  • Data Engineer

    2 weeks ago


    Bangalore/Pune/Kolkata/Gurgaon/Gurugram/Chennai, India VIDPRO CONSULTANCY SERVICES Full time

    Mandatory Skills : Snowflake/Azure Data Factory/ PySpark / DatabricksPrimary Roles and Responsibilities : - Developing Modern Data Warehouse solutions using Snowflake, Databricks and ADF.- Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development...

  • Data Engineer

    3 weeks ago


    Bangalore/Pune/Kolkata/Gurgaon/Gurugram/Chennai, IN VIDPRO CONSULTANCY SERVICES Full time

    Mandatory Skills : Snowflake/Azure Data Factory/ PySpark / DatabricksPrimary Roles and Responsibilities :- Developing Modern Data Warehouse solutions using Snowflake, Databricks and ADF.- Ability to provide solutions that are forward-thinking in data engineering and analytics space- Collaborate with DW/BI leads to understand new ETL pipeline development...

  • Azure Data Engineer

    1 month ago


    Delhi NCR,Mumbai,Bangalore,Hyderabad,Pune,Chennai,Kolkata,Coimbatore, India Changeleaders Counsultancy Full time

    Job Description : - Min 3+ years of IT experience, including experience in designing Azure data platform with large scale implementation experience.- Hands-on-experience in developing data lake solutions using Azure (Azure data factory for ingestion, Data lake gen 2 and Azure SQL server for storage, Azure analysis service for transformations, Azure data...

  • Azure Data Engineer

    4 weeks ago


    Delhi NCR/Mumbai/Bangalore/Hyderabad/Pune/Chennai/Kolkata/Coimbatore, IN Changeleaders Counsultancy Full time

    Job Description : - Min 3+ years of IT experience, including experience in designing Azure data platform with large scale implementation experience.- Hands-on-experience in developing data lake solutions using Azure (Azure data factory for ingestion, Data lake gen 2 and Azure SQL server for storage, Azure analysis service for transformations, Azure data...

  • Data Science Role

    3 weeks ago


    Bangalore/Kolkata/Mumbai/Hyderabad, India iimjobs Full time

    Data Science - Supply Chain /Manufacture /Retail AnalyticsData Science - Manager /Sr Manager - Diversity / Female only Skills- Gen AI / Conversational AI /Retail Analytics/Supply Chain / Healthcare Analytics/ Manufacturing/IOT/Utility AnalyticsMust Have: 8+ years of experience Indepth knowledge of supply chain processes, logistics, inventory management, and...


  • Bangalore/Chennai/Delhi NCR/Pune/Kolkata, India Zyoin group Full time

    Job Description : Mandatory Skills : Python/ PySpark / Spark with Azure/ AWS Databricks Primary Roles and Responsibilities : - Developing Modern Data Warehouse solutions using Databricks and AWS/ Azure Stack - Ability to provide solutions that are forward-thinking in data engineering and analytics space - Collaborate with DW/BI leads to understanding...

  • Data Engineer

    1 week ago


    Bangalore/Pune/Kolkata/Gurgaon/Gurugram/Mumbai/Ahmedabad, India Sterlite Technologies Limited Full time

    Role as a Data engineer :- Develop and build AI/ML model ready data ponds using Azure Databricks Delta lake- Improve upon the data ingestion models, ETL jobs, and alarm to maintain data integrity and data availability.- Partner with AI/ML product teams proactively identify the data needs for rapid experimentation, stable production deployment and ensure the...

  • Data Engineer

    1 week ago


    Bangalore/Pune/Kolkata/Gurgaon/Gurugram/Mumbai/Ahmedabad, IN Sterlite Technologies Limited Full time

    Role as a Data engineer :- Develop and build AI/ML model ready data ponds using Azure Databricks Delta lake- Improve upon the data ingestion models, ETL jobs, and alarm to maintain data integrity and data availability.- Partner with AI/ML product teams proactively identify the data needs for rapid experimentation, stable production deployment and ensure the...