Data engineer(Databricks,pyspark, ADF,SQL)

1 day ago

Chennai, Tamil Nadu, India Risk Resources LLP Full time ₹ 15,00,000 - ₹ 25,00,000 per year
Overview: The Data Engineer is a crucial member of our data team, responsible for designing, building, and maintaining robust data infrastructure. Utilizing platforms such as Databricks and Azure Data Factory, this role focuses on creating efficient data pipelines and managing large datasets for analysis. By employing tools like PySpark and SQL, the Data Engineer ensures data integrity, accessibility, and performance, thus empowering business intelligence initiatives across the organization. With the escalating importance of data-driven decision-making, the Data Engineer plays an integral role in transforming raw data into actionable insights, facilitating the delivery of high-quality data to stakeholders. Collaboration with data analysts, data scientists, and other cross-functional teams is essential to enhance data usage throughout the organization. The ideal candidate will have a strong background in data systems, excellent technical skills, and the ability to work in a dynamic environment while meeting stringent deadlines.
  • Design, develop, and maintain scalable data pipelines using Databricks and PySpark.
  • Implement data transformation processes within Azure Data Factory.
  • Create and optimize SQL queries for efficient data retrieval.
  • Collaborate with data scientists and analysts to understand data requirements.
  • Ensure data integrity and quality through rigorous testing and validation processes.
  • Build and maintain data models to support business intelligence efforts.
  • Monitor and troubleshoot data pipeline performance issues.
  • Document data engineering processes, architectures, and workflows.
  • Develop and enforce best practices for data governance and security.
  • Participate in Agile development cycles and project management activities.
  • Stay updated on the latest data technologies and industry trends.
  • Perform ad-hoc data analysis and report generation as required.
  • Assist in capacity planning and optimization of data storage solutions.
  • Engage in continuous improvement of data management processes.
  • Ensure compliance with data privacy regulations and policies.
  • Provide support and training to team members on data engineering tools and practices.
Required Qualifications: - Bachelor's degree in Computer Science, Information Technology, or related field. Proven experience as a Data Engineer. Strong proficiency with Databricks. Extensive experience in PySpark. Hands-on experience with Azure Data Factory (ADF). Solid knowledge of SQL and relational databases. Experience with big data technologies such as Hadoop or Spark. Familiarity with data modeling and warehousing. Understanding of ETL (Extract, Transform, Load) processes. Ability to work with large datasets and optimize performance. Strong analytical and problem-solving skills. Excellent communication and teamwork abilities. Experience with cloud technologies, particularly Azure. Understanding of data security best practices. Proven ability to manage multiple tasks and meet deadlines.

  • ADF Data Engineer

    3 days ago

    Chennai, Tamil Nadu, India Recode Solutions Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Overview:We are seeking an experienced Data Engineer to join our team in a Developer. This role focuses on enhancing and supporting existing Data & Analytics solutions by leveraging Azure Data Engineering technologies. The engineer will work on developing, maintaining, and deploying IT products and solutions that serve various business users, with a strong...

  • Azure Data Engineer

    3 days ago

    Chennai, Tamil Nadu, India DataEngine Full time ₹ 12,00,000 - ₹ 36,00,000 per year

    Hiring for Azure Data EngineerExperience: 8+ yearsLocation: Chennai, Hyderabad, BangaloreNotice Period: Immediate to 30 daysKey ResponsibilitiesData Pipelines: Build pipelines for raw data ingestion and orchestration using Azure Data Factory (ADF) and DatabricksData Transformation: Develop and maintain innovative Azure solutions, perform transformation using...

  • Databricks Data Engineer

    4 weeks ago

    Chennai, Tamil Nadu, India, Tamil Nadu Tata Consultancy Services Full time

    About the RoleThe Databricks Data Engineer will play a crucial role in migrating and optimizing data workflows, ensuring high performance and reliability in our data operations.Experience- 3 to 6 yearsLocation - Kolkata /Chennai/TrivandrumRequired Technical Skill Set ------ Databricks, AWS, Python & PySparkGood to have- Experience in AWS platform (S3, EC2,...

  • Azure Databricks Lead

    2 days ago

    Chennai, Tamil Nadu, India GIG Consultants Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Description : Key Responsibilities : - Lead the design, development, and optimization of large-scale data solutions using Azure Databricks, Azure Data Factory (ADF), and Azure Synapse Analytics. - Architect and implement efficient ETL/ELT pipelines for ingesting, transforming, and processing structured and unstructured data. - Develop...

  • Manager - Data Engineering

    1 week ago

    Chennai, Tamil Nadu, India LatentView Analytics Full time ₹ 20,00,000 - ₹ 25,00,000 per year

    Designation: Manager – Data EngineeringLevel: L4Experience: 9 to 15 yearsLocation: ChennaiJob Description:We are seeking a highly skilled and motivated Manager with strong experience in the Databricks Domain to join our growing team.Responsibilities:Architect and design solutions to meet functional and non-functional requirements.Lead the design,...

  • Data Engineer

    3 days ago

    Chennai, Tamil Nadu, India CX Data Labs Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    HiRole - Azure Data EngineerLocation - Chennai, Gurugram (5days Office)Full timeShift timing 2pm to 11pmKey Responsibilities:· Develop, maintain, and optimize scalable ETL/ELT pipelines using PySpark and Databricks.· Collaborate with cross-functional teams to design and implement data models and data integration solutions.· Create and maintain robust SQL...

  • Azure Data Factory,Azure Python, Pyspark, Databricks, SQL Professional

    1 week ago

    Chennai, Tamil Nadu, India Suzva Software Technologies Full time ₹ 15,00,000 - ₹ 25,00,000 per year

    Experience with Azure datafactory,Azure ,Python, Pyspark, Databricks, SQL Key ResponsibilitiesDesign and build ETL/ELT pipelines to ingest, transform, and load data from clinical, omics, research, and operational sources.Optimize performance and scalability of data flows using tools like Apache Spark, Databricks, or AWS Glue.Collaborate with domain experts...

  • Senior Data Engineer – Databricks

    3 days ago

    Chennai, Tamil Nadu, India Zillion Technologies Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Senior Data Engineer – DatabricksLocation: Hyderabad/ChennaiWork Schedule: Timings 2:30 PM to 11:30 PMRole Overview:We are seeking a highly skilled Senior Data Engineer with deep expertise in Databricks to design, build, and optimize scalable data pipelines and architectures. You will collaborate with cross-functional teams including data scientists,...

  • Azure Data Engineer

    1 week ago

    Chennai, Tamil Nadu, India D2KSS SOFTWARE SERVICES PRIVATE LIMITED Full time ₹ 8,00,000 - ₹ 24,00,000 per year

    Industry & Sector:IT services and enterprise data engineering — delivering cloud-native analytics, ETL/ELT pipelines, and modern data platform implementations on Microsoft Azure for large enterprise customers. This is an on-site, hands-on engineering role supporting strategic data modernization initiatives across business domains.Primary Job TitleSenior...

  • Data Engineer

    5 days ago

    Chennai, Tamil Nadu, India Work Fox Solutions Full time ₹ 9,00,000 - ₹ 12,00,000 per year

    Data EngineerMaintain ETL pipelines using Databricks and PySpark to extract, transform, and load data from various sources.Troubleshoot and resolve issues related to data transformations or loading processes in Databricks.Develop and maintain documentation for ETL processes, configurations, and data mappings.Strong proficiency in Databricks with hands-on...