ETL Specialist

6 days ago

Erode, India Responsive Full time

We are seeking a highly skilled ETL Specialist with expertise in building, maintaining, and optimizing data pipelines using Python scripting. The ideal candidate will have experience working in a Linux environment, managing large-scale data ingestion, processing files in S3, and balancing disk space and warehouse storage efficiently. This role will be responsible for ensuring seamless data movement across systems while maintaining performance, scalability, and reliability.Key Responsibilities:ETL Pipeline Development: Design, develop, and maintain efficient ETL workflows using Python to extract, transform, and load data into structured data warehouses.Data Pipeline Optimization: Monitor and optimize data pipeline performance, ensuring scalability and reliability in handling large data volumes.Linux Server Management: Work in a Linux-based environment, executing command-line operations, managing processes, and troubleshooting system performance issues.File Handling & Storage Management: Efficiently manage data files in Amazon S3, ensuring proper storage organization, retrieval, and archiving of data.Disk Space & Warehouse Balancing: Proactively monitor and manage disk space usage, preventing storage bottlenecks and ensuring warehouse efficiency.Error Handling & Logging: Implement robust error-handling mechanisms and logging systems to monitor data pipeline health.Automation & Scheduling: Automate ETL processes using cron jobs, Airflow, or other workflow orchestration tools.Data Quality & Validation: Ensure data integrity and consistency by implementing validation checks and reconciliation processes.Security & Compliance: Follow best practices in data security, access control, and compliance while handling sensitive data.Collaboration with Teams: Work closely with data engineers, analysts, and product teams to align data processing with business needs.Skills Required:Proficiency in Python: Strong hands-on experience in writing Python scripts for ETL processes.Linux Expertise: Experience working with Linux servers, command-line operations, and system performance tuning.Cloud Storage Management: Hands-on experience with Amazon S3, including handling file storage, retrieval, and lifecycle policies.Data Pipeline Management: Experience with ETL frameworks, data pipeline automation, and workflow scheduling (e.g., Apache Airflow, Luigi, or Prefect).SQL & Database Handling: Strong SQL skills for data extraction, transformation, and loading into relational databases and data warehouses.Disk Space & Storage Optimization: Ability to manage disk space efficiently, balancing usage across different systems.Error Handling & Debugging: Strong problem-solving skills to troubleshoot ETL failures, debug logs, and resolve data inconsistencies.Nice to Have:Experience with cloud data warehouses (e.g., Snowflake, Redshift, BigQuery).Knowledge of message queues (Kafka, RabbitMQ) for data streaming.Familiarity with containerization tools (Docker, Kubernetes) for deployment.Exposure to infrastructure automation tools (Terraform, Ansible).Qualifications:Bachelor’s degree in Computer Science, Data Engineering, or a related field.3+ years of experience in ETL development, data pipeline management, or backend data engineering.Strong analytical mindset and ability to handle large-scale data processing efficiently.Ability to work independently in a fast-paced, product-driven environment.

  • Senior ETL Engineer – Data Integrity Specialist

    4 days ago

    erode, India beBeeInformatica Full time

    Job OverviewWe are seeking an experienced ETL Specialist with strong expertise in ETL/Data Pipeline Testing and API Testing to join our organization. The ideal candidate will possess a deep understanding of data flows, SQL skills, and familiarity with Informatica tools.The selected individual will be responsible for performing end-to-end testing of ETL...

  • ETL Testing Specialist

    2 days ago

    erode, India beBeeData Full time

    ETL Testing ProfessionalWe are seeking a highly skilled ETL testing professional to support the success of our organization.Job Description:Design and execute comprehensive test plans to guarantee data accuracy and reliability. Identify defects in ETL processes and collaborate with development teams to implement fixes and enhancements.Key...

  • Senior Data Validation Specialist

    2 days ago

    erode, India beBeeData Full time

    Job OpeningWe are seeking a highly skilled QA Consultant with expertise in ETL and data pipeline testing, as well as API testing.The ideal candidate will have a strong understanding of data flows, SQL skills, and familiarity with Informatica tools such as IDMC/IICS, CDI, and MDM SaaS.Perform comprehensive end-to-end testing of ETL pipelines across cloud and...

  • Pharmaceutical Data Specialist

    4 days ago

    erode, India beBeeData Full time

    Pharmaceutical Data SpecialistWe are seeking an experienced Pharmaceutical Data Specialist to join our team on a long-term contract basis. This role is ideal for individuals with hands-on experience in the pharmaceutical industry, who possess strong analytical and problem-solving skills.The successful candidate will be responsible for developing,...

  • AI Applications Specialist

    4 days ago

    erode, India beBeeArtificialintelligence Full time

    AI Applications SpecialistWe're seeking a skilled AI Applications Specialist to join our team. As an AI Applications Specialist, you will be responsible for designing and deploying end-to-end AI systems from prototype to production.You will work with the latest LLM technologies and solve real-world challenges across energy, defense, finance, and data...

  • Migrating Complex Data Specialist

    4 days ago

    erode, India beBeeDataMigration Full time

    Data Migration ExpertMigrate complex data from legacy systems to our modern platform with ease. As a Data Migration Specialist, you'll leverage your technical expertise to develop efficient methods for migrating client data into our platform.This role requires collaboration with AI coding assistants to create streamlined processes for transforming large...

  • Data Validation Specialist

    4 days ago

    erode, India beBeeDataValidation Full time

    Effective data management is critical to informed business decision-making.Your RoleYou will be responsible for designing, developing, and executing comprehensive ETL test plans to validate and verify data extraction, transformation, and loading processes.Key responsibilities include validating data accuracy and completeness, identifying defects, and...

  • Data Pipeline Specialist

    2 days ago

    erode, India beBeeDataEngineer Full time

    About this RoleWe are seeking a skilled Data Engineer to design, develop, and maintain robust ETL/ELT pipelines for various data sources. The ideal candidate will have strong expertise in SQL (PostgreSQL preferred) and Python, with hands-on experience building ETL pipelines and working in Lakehouse architecture.Key ResponsibilitiesData Engineering &...

  • Data Solutions Specialist

    4 days ago

    erode, India beBeeDataEngineer Full time

    Job OpportunityWe are seeking a skilled professional to design and implement data solutions that drive business outcomes.The ideal candidate will have strong hands-on experience in Python programming and PySpark, as well as expertise in AWS services, Apache Spark, and Hadoop ecosystem.Responsibilities include developing and testing complex data pipelines for...

  • Data Integration Specialist

    4 days ago

    erode, India beBeeEtl Full time

    Senior ETL DeveloperThe Senior Consultant plays a crucial role in integrating and mapping customer data from client source systems to our industry-leading platform. The position involves, but is not limited to:Utilizing strong technical data migration, scripting, and organizational skills to ensure the client data is converted efficiently and accurately to...