Data Engineering Azure databricks

1 day ago

Thoothukudi, India EXL Full time

Data Engineer (DE) Consultant is responsible for designing, developing, and maintaining data assets and data related products by liaising with multiple stakeholders.

Responsibilities:

  • Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.
  • Create the data integration and data diagram documentation.
  • Lead the data validation, UAT and regression test for new data asset creation.
  • Create and maintain data models, including schema design and optimization.
  • Create and manage data pipelines that automate the flow of data, ensuring data quality and consistency.

Qualifications and Skills:

  • Strong knowledge on Python and Pyspark
  • Expectation is to have ability to write Pyspark scripts for developing data workflows.
  • Strong knowledge on SQL, Hadoop, Hive, Azure, Databricks and Greenplum
  • Expectation is to write SQL to query metadata and tables from different data management system such as, Oracle, Hive, Databricks and Greenplum.
  • Familiarity with big data technologies like Hadoop, Spark, and distributed computing frameworks.
  • Expectation is to use Hue and run Hive SQL queries, schedule Apache Oozie jobs to automate the data workflows.
  • Good working experience of communicating with the stakeholders and collaborate effectively with the business team for data testing.
  • Expectation is to have strong problem-solving and troubleshooting skills.
  • Expectation is to establish comprehensive data quality test cases, procedures and implement automated data validation processes.
  • Degree in Data Science, Statistics, Computer Science or other related fields or an equivalent combination of education and experience.
  • 4-7 years of experience in Data Engineer.
  • Proficiency in programming languages commonly used in data engineering, such as Python, Pyspark, SQL.
  • Experience in Azure cloud computing platform, such as developing ETL processes using Azure Data Factory, big data processing and analytics with Azure Databricks.
  • Strong communication, problem solving and analytical skills with the ability to do time management and multi-tasking with attention to detail and accuracy.

  • Data Engineer

    2 weeks ago

    Thoothukudi, India Innodata Inc. Full time

    We are looking for a Data Engineer with strong experience in CI/CD practices, Databricks (Spark), Python, Github and SQL. The ideal candidate should have hands-on expertise in building and automating data pipelines, managing multi-environment deployments, and working with modern DevOps/configuration management tools.Key ResponsibilitiesDesign, implement, and...

  • Freelance Data Engineer – Python, SQL, AWS

    4 weeks ago

    Thoothukudi, Tamil Nadu, India ThreatXIntel Full time

    Company DescriptionThreatXIntel is a startup cyber security company that offers customized, affordable solutions to protect businesses and organizations from cyber threats. The company provides services like cloud security, web and mobile security testing, and DevSecOps. ThreatXIntel takes a proactive approach to security by continuously monitoring and...

  • Data engineer

    4 days ago

    Thoothukudi, India Everise Full time

    Company OverviewJoin us on our mission to elevate customer experiences for people around the world. As a member of the Everise family, you will be part of a global experience company that believes in being people-first, celebrating diversity and incubating innovation. Our dedication to our purpose and people is being recognized by our employees and the...

  • Senior Manager, Data Governance Engineering

    1 week ago

    Thoothukudi, India EDB Full time

    EnterpriseDB is seeking an experienced and visionary Senior Manager of Data Governance Engineering to lead our global data governance strategy and engineering execution. This role will focus on building and maintaining the next-generation data ecosystem—spanning transactional, analytical, and AI workloads—while ensuring robust governance, traceability,...

  • Audit Consultant

    3 weeks ago

    Thoothukudi, Tamil Nadu, India RED Global Full time

    Role: Audit ConsultantStart Date: ASAPEnd Date: 12months+ extensionCapacity: 5 days a weekLocation: 100% RemoteLanguage: EnglishJob Description:We are looking for an audit consulting engagement for assessing their current Databricks implementation on Azure and integration with AzureDevOps.Based on the findings in the audit and recommendation, we can build...

  • Senior Test Lead

    3 weeks ago

    Thoothukudi, Tamil Nadu, India NovintiX Full time

    ???? ???????????????????? ???????????? ????????????????????????????Novinti X is a fast-growing engineering and digital transformation consultancy serving top Life Sciences, Med Tech, and Healthcare clients across the US, Ireland, and Europe.We specialise in delivering high-impact digital programs, engineering services, and talent solutions that accelerate...

  • Data Platform Engineer

    3 weeks ago

    Thoothukudi, Tamil Nadu, India Tekgence Inc Full time

    Data Platform Engineer Work Location will be Hybrid in Mumbaicontract duration: 12+ monthsStrong Software Engineering SkillsSolid programming expertise, especially in Python, with strong fundamentals in computer science, data structures, algorithms, and system design.Proven problem-solving ability with a focus on writing clean, maintainable, and efficient...

  • Data and Business Intelligence Analyst

    3 weeks ago

    Thoothukudi, Tamil Nadu, India Opportunity International EduFinance Full time

    APPLICATION PROCESSPlease submit your application using the below link:Please note, any applications not submitted via the above form cannot be considered. Applications are reviewed on a rolling basis.POSITION SUMMARYThe Data and Business Intelligence Analyst will play a key role in building and maintaining EduFinance's data infrastructure while delivering...

  • Big Data Engineer

    4 weeks ago

    Thoothukudi, Tamil Nadu, India Idyllic Services Full time

    Job Title: Big Data Engineer – ScalaLocation: Bangalore, Chennai, Gurgaon, Pune, Mumbai.Experience: 7–10 Years (Minimum 3+ years in Scala)Notice Period: Immediate to 30 DaysMode of Work: Hybrid Click the link below to learn more about the role and take the AI Interview to begin your application journey: https://sl1nk.com/XiPeNRole OverviewWe are looking...

  • Staff Data Engineer

    3 weeks ago

    Thoothukudi, Tamil Nadu, India Commonwealth Bank Full time

    Organization:At CommBank, we never lose sight of the role we play in other people's financial wellbeing. Our focus is to help people and businesses move forward to progress. To make the right financial decisions and achieve their dreams, targets, and aspirations. Regardless of where you work within our organisation, your initiative, talent, ideas, and energy...