Data Engineering Azure databricks

3 weeks ago

Thoothukudi, India EXL Full time

Data Engineer (DE) Consultant is responsible for designing, developing, and maintaining data assets and data related products by liaising with multiple stakeholders.

Responsibilities:

  • Work with stakeholders to understand the data requirements to design, develop, and maintain complex ETL processes.
  • Create the data integration and data diagram documentation.
  • Lead the data validation, UAT and regression test for new data asset creation.
  • Create and maintain data models, including schema design and optimization.
  • Create and manage data pipelines that automate the flow of data, ensuring data quality and consistency.

Qualifications and Skills:

  • Strong knowledge on Python and Pyspark
  • Expectation is to have ability to write Pyspark scripts for developing data workflows.
  • Strong knowledge on SQL, Hadoop, Hive, Azure, Databricks and Greenplum
  • Expectation is to write SQL to query metadata and tables from different data management system such as, Oracle, Hive, Databricks and Greenplum.
  • Familiarity with big data technologies like Hadoop, Spark, and distributed computing frameworks.
  • Expectation is to use Hue and run Hive SQL queries, schedule Apache Oozie jobs to automate the data workflows.
  • Good working experience of communicating with the stakeholders and collaborate effectively with the business team for data testing.
  • Expectation is to have strong problem-solving and troubleshooting skills.
  • Expectation is to establish comprehensive data quality test cases, procedures and implement automated data validation processes.
  • Degree in Data Science, Statistics, Computer Science or other related fields or an equivalent combination of education and experience.
  • 4-7 years of experience in Data Engineer.
  • Proficiency in programming languages commonly used in data engineering, such as Python, Pyspark, SQL.
  • Experience in Azure cloud computing platform, such as developing ETL processes using Azure Data Factory, big data processing and analytics with Azure Databricks.
  • Strong communication, problem solving and analytical skills with the ability to do time management and multi-tasking with attention to detail and accuracy.

  • Data Engineer

    3 weeks ago

    Thoothukudi, India Dunboxed Full time

    Dunboxed is a boutique data services firm specializing in Data Analytics, Business Intelligence (BI), and Data Engineering. We don't just deliver services; we build strategic partnerships that drive real business value for our clients. As a growing firm, we are agile, innovative, and focused on quality. Every team member plays a critical role in our...

  • Data Engineer

    1 week ago

    Thoothukudi, India AAA Global Full time

    Data Engineer | Mumbai Location: India Industry: Proprietary Trading / Financial Markets Experience: 4+ years Employment Terms: Full-time, Permanent Compensation: Competitive, market-aligned Introduction We are seeking an experienced Data Engineer to strengthen our core Data Engineering team. In this key role, you will ensure the secure, scalable, and...

  • Data engineer

    4 weeks ago

    Thoothukudi, India Everise Full time

    Company OverviewJoin us on our mission to elevate customer experiences for people around the world. As a member of the Everise family, you will be part of a global experience company that believes in being people-first, celebrating diversity and incubating innovation. Our dedication to our purpose and people is being recognized by our employees and the...

  • Data Architect

    3 weeks ago

    Thoothukudi, India Tata Consultancy Services Full time

    We are hiring for Data Architect. Role: Data ArchitectExperience: 8 years to 15 yearsLocation: Hyderabad / Pune/Bangalore/Delhi / MumbaiTechnical skill - AWS, Azure, GCP, Snowflake, Redshift, BiqQuery, Python, Talend, IICS, Hadoop, Spark, Kafka, Databricks Interested candidates can send their resume on below mail ID along with below...

  • Java Developer

    7 days ago

    Thoothukudi, India Recro Full time

    Tech StackBackend & Infra: Java (Spring Boot), Microservices, AWS ECS, Redis, ScyllaDB, MongoDB, GraphQL, AWS, Lambda, SQSData & AI: Databricks, Spark, Kinesis, Claude, Cursor, LangChainObservability: Prometheus, Grafana, CloudWatchRole : Senior Backend Engineer (Java / Microservices)Location: Remote (Work from Home)We’re hiring an experienced backend...

  • Lead data engineer

    1 week ago

    Thoothukudi, India ATOM Systems Private Limited Full time

    Job DescriptionName of the position : Data EngineerLocation :  Coimbatore / Remote No. of resources needed: 01 Mode: Contract to HireYears of experience: 15+ Years About the RoleWe are seeking a highly skilled and driven Data Engineering Lead to lead our data engineering team. The ideal candidate combines strong leadership and technical...

  • Senior data modeler

    1 week ago

    Thoothukudi, India Philodesign Technologies Inc Full time

    Position: Senior Data ModelerExperience: 7+ YearsShift: General ShiftLocation: RemoteRole OverviewWe are seeking an experienced Senior Data Modeler to design, develop, and maintain conceptual, logical, and physical data models that support business and analytical requirements. The ideal candidate will translate complex business needs into efficient and...

  • Rpa engineer

    7 days ago

    Thoothukudi, India PropertyLoop Full time

    Job Title: RPA Engineer – Web-Form AutomationLocation: India – Remote (core overlap with GMT + IST)Employment Type: Full-time, PermanentReports To: Lead Technologist & Product Strategy Liaison (UK)About Us:Property Loop is a UK-based technology company that specialises in streamlining operations through automated processes. Our platform collects...

  • DevOps Engineer

    3 weeks ago

    Thoothukudi, India Flowmentum, Inc. Full time

    🚀 We're Hiring: DevOps & Platform Engineer (Azure | .NET | Terraform | PowerShell | Azure DevOps CI) 🔧 SRE-Focused | Remote | Flexible HoursWe're looking for a skilled DevOps Engineer with hands-on experience in Azure, .NET, Terraform, PowerShell scripting, and CI/CD using Azure DevOps. If you're passionate about automation, infrastructure-as-code, and...

  • Lead Systems Development Engineer

    4 weeks ago

    Thoothukudi, India JLL Full time

    Lead Systems Development EngineerJLL TechnologiesWhat this job involves:#JLLTechAmbitionsKey ResponsibilitiesDesign and implement cloud infrastructure across Azure, AWS, and GCP environmentsArchitect, deploy and optimize Azure-based solutions including compute, storage, networking, and securty servicesLead cloud migration and modernization initiatives with...