ITI Data | PySpark Data Engineer

6 hours ago

delhi, India ITI Data Full time
Location : India
Type : Full-time
Experience : 10 – 13 years
Functions : Consulting, Finance, Information Technology, Big Data Engineering
Industries : Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, Healthcare
Job Description
We are looking for a PySpark solutions developer and data engineer that is able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building a data standardized and curation needs on Hadoop cluster. This is high visibility, fast-paced key initiative will integrate data across internal and external sources, provide analytical insights, and integrate with the customer’s critical systems.
Key Responsibilities
Ability to design, build and unit test applications on Spark framework on Python .
Build PySpark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and NoSQL databases as well.
Develop and execute data pipeline testing processes and validate business rules and policies
Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's .
Experience in skilled in Hadoop, Kafka, Scala, Spark.
Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats ( Avro, Parquet, ORC etc ) and compression codec respectively.
Ability to design & build real-time applications using Apache Kafka & Spark Streaming Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec . Build data tokenization libraries and integrate with Hive & Spark for column-level obfuscation Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories
Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings
Work collaboratively with onsite and offshore team. Develop & review technical documentation for artifacts delivered.Ability to solve complex data-driven scenarios and triage towards defects and production issuesAbility to learn-unlearn-relearn concepts with an open and analytical mindsetParticipate in code release and production deployment

  • ITI Data | PySpark Data Engineer

    2 days ago

    Delhi, India ITI Data Full time

    Location:IndiaType: Full-timeExperience: 10 – 13 yearsFunctions: Consulting, Finance, Information Technology, Big Data EngineeringIndustries: Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, HealthcareJob DescriptionWe are looking for a PySpark solutions developer...

  • PySpark Data Engineer

    2 days ago

    Delhi, India ITI Data Full time

    Location : IndiaType : Full-timeExperience : 10 – 13 yearsFunctions : Consulting, Finance, Information Technology, Big Data EngineeringIndustries : Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, HealthcareJob DescriptionWe are looking for a PySpark...

  • PySpark Data Engineer

    2 days ago

    Delhi, India ITI Data Full time

    Location:IndiaType: Full-timeExperience: 10 – 13 yearsFunctions: Consulting, Finance, Information Technology, Big Data EngineeringIndustries: Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, HealthcareJob DescriptionWe are looking for a PySpark solutions developer...

  • Pyspark data engineer

    2 days ago

    Delhi, India ITI Data Full time

    Location : IndiaType : Full-timeExperience : 10 – 13 yearsFunctions : Consulting, Finance, Information Technology, Big Data EngineeringIndustries : Capital Markets, Investment Banking, Alternative Investments, Financial Services, Management Consulting, Information Technology and Services, HealthcareJob DescriptionWe are looking for a Py Spark...

  • AWS Data Engineer

    1 month ago

    delhi, India ITI Data Full time

    Job Description We are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....

  • AWS Data Engineer

    4 months ago

    Delhi, India ITI Data Full time

    Job DescriptionWe are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources. There...

  • AWS Data Engineer

    1 month ago

    delhi, India ITI Data Full time

    Job DescriptionWe are looking for an AWS Data with primary skills on PySpark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources. There...

  • Aws data engineer

    4 weeks ago

    Delhi, India ITI Data Full time

    Job DescriptionWe are looking for an AWS Data with primary skills on Py Spark development who will be able to design and build solutions for one of our Fortune 500 Client programs, which aims towards building an Enterprise Data Lake on AWS Cloud platform, build Data pipelines by developing several AWS Data Integration, Engineering & Analytics resources....

  • Data Warehouse Engineer | Data Warehouse Engineer | delhi

    2 weeks ago

    delhi, India Data Warehouse Engineer Full time

    Experience : 2- 5 YearsPrimary Skills : Strong SQL knowledge, Data warehouse concepts and ETL hands on experience, Azure services like Data Factory, Azure Data Lake, and Good analysis skills.Good to have skills: Power BI, Databricks, Python.

  • Data Warehouse Engineer

    2 weeks ago

    Delhi, India Data Warehouse Engineer Full time

    Experience : 2- 5 YearsPrimary Skills : Strong SQL knowledge, Data warehouse concepts and ETL hands on experience, Azure services like Data Factory, Azure Data Lake, and Good analysis skills.Good to have skills: Power BI, Databricks, Python.

  • Data Engineer

    5 days ago

    Delhi, India Tata Consultancy Services Full time

    Job DescriptionName of the position: Data engineerLocation : Bangalore, Hyderabad,Mumbai, Pune ,Chennai,NCRSkill Requirements & Experience- Overall 8+ years of work experience in Data Warehouse(DWH) Development- Experience in Azure, Python , SQL , Pyspark- Hands on exp in Azure data factory, data bricks- Exposure and experience to Cosmos- Understanding of...

  • Senior Data Integration Engineer

    2 weeks ago

    Delhi, India Senior Data Integration Engineer Full time

    Must Have Skills/Skill Requirement:Design and architect integration solutions to connect various enterprise applications, systems, and databases.Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications.Utilize Azure Integration Services such as Azure Logic Apps,...

  • Senior Data Integration Engineer | Senior Data Integration Engineer | delhi

    2 weeks ago

    delhi, India Senior Data Integration Engineer Full time

    Must Have Skills/Skill Requirement:Design and architect integration solutions to connect various enterprise applications, systems, and databases.Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications.Utilize Azure Integration Services such as Azure Logic Apps,...

  • Senior Data Integration Engineer

    2 weeks ago

    Delhi, India Senior Data Integration Engineer Full time

    Must Have Skills/Skill Requirement:- Design and architect integration solutions to connect various enterprise applications, systems, and databases.- Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications.- Utilize Azure Integration Services such as Azure Logic...

  • Senior data integration engineer

    2 weeks ago

    Delhi, India Senior Data Integration Engineer Full time

    Must Have Skills/Skill Requirement:Design and architect integration solutions to connect various enterprise applications, systems, and databases.Develop and implement integration workflows, APIs, and data pipelines to enable smooth communication and data exchange between different applications.Utilize Azure Integration Services such as Azure Logic Apps,...

  • Data Engineer for Azure Synapse and Pyspark Development

    3 days ago

    Delhi, Delhi, India Tata Consultancy Services Full time

    About Tata Consultancy ServicesTata Consultancy Services (TCS) is a global leader in IT services, consulting, and business solutions. With over 50 years of experience, TCS has been at the forefront of driving innovation and growth through technology.Job Title: Data Engineer for Azure Synapse and Pyspark DevelopmentThis role offers an exciting opportunity to...

  • Python, Pyspark Data Engineer

    5 days ago

    Delhi, India KC Executive Search Full time

    Technical Stack: Python, Spark, AWS, PySparkResponsibilitiesDevelop and enhance data-processing, orchestration, monitoring, and more by leveraging popular open-source software, AWS, and GitLab automation.Collaborate with product and technology teams to design and validate the capabilities of the data platform Identify, design, and implement process...

  • Pyspark/Python Data Engineer

    4 months ago

    Delhi, India Genpact Full time

    Genpact (NYSE: G) is a global professional services and solutions firm delivering outcomes that shape the future. Our 125,000+ people across 30+ countries are driven by our innate curiosity, entrepreneurial agility, and desire to create lasting value for clients. Powered by our purpose – the relentless pursuit of a world that works better for people – we...

  • Lead data engineer

    4 weeks ago

    Delhi, India Wavicle Data Solutions Full time

    Job Description:We are seeking a highly experienced Lead Data Engineer with over 8 years of expertise in data engineering.As a Lead Data Engineer, you will play a pivotal role in architecting and implementing data solutions.Your proficiency in Python, Py Spark, AWS, Databricks, SQL, and leadership skills will be crucial for success.Key Responsibilities:Lead...

  • Data Platform Engineer

    5 days ago

    Delhi, India OSD Data Services Full time

    Data Platform EngineerLocation: RemoteType: InternshipAbout UsAt OSD Data, we’re redefining how businesses leverage their data. Our mission is to empower organizations with a seamless, unified, and scalable data infrastructure that enables smarter decision-making and faster innovation. With a focus on data lakehouse technologies, we’ve built a...