Senior Python Data Engineer

4 days ago

Kollam, India Whatjobs IN C2 Full time

About the Project: Join a dynamic engineering team working on a high-impact tax reporting platform for the 2025 fiscal season. The core goal is to modernize and significantly accelerate the generation of Excel-based reports, reducing report generation time from minutes to seconds. This involves ingesting data from multiple upstream systems, transforming it using efficient data processing libraries, and serving it through APIs. The team emphasizes scalability, maintainability, and developer productivity, incorporating spec-driven development powered by AI tools. You’ll be contributing to the backend architecture and data pipeline powering this transformation, helping to evolve a high-performance system that's central to the client’s reporting domain. Responsibilities: Design, build, and maintain high-performance data processing pipelines using Python libraries (Pandas required; Polars nice-to-have). Develop and expose RESTful APIs using FastAPI or similar frameworks. Consume and process normalized Parquet files from multiple upstream sources to generate dynamic Excel reports. Contribute to a spec-driven development workflow (using GitHub Copilot, Claude, etc.) to scaffold and generate API/data pipeline code. Optimize report generation logic for speed and scalability, currently targeting sub-20 second response times. Integrate with messaging and storage mechanisms (e.G., Service Bus, Storage Accounts). Collaborate on infrastructure-as-code automation using Bicep similar tools (Terraform, CDK). Participate in design discussions for future migration to Snowflake and/or a data lake architecture. Contribute to CI/CD pipelines using GitHub Actions. Required Skills and Experience: Strong proficiency in Python for data processing with hands-on expertise in Pandas. Ability to quickly learn new frameworks such as Polars if needed. Experience building backend services or APIs using frameworks like FastAPI. Solid understanding of data modeling principles (Star Schema) and handling normalized datasets. Familiarity with enterprise messaging patterns and data integration from various sources (API-based and file-based). Experience working with GitHub and CI/CD pipelines (GitHub Actions or similar). Infrastructure-as-Code experience with Bicep or comparable tools (Terraform, AWS CDK). Comfort with spec-driven development and leveraging AI tools like GitHub Copilot for scaffolding. Nice-to-Have / Preferred Qualifications: Experience with Polars (not required). Experience with orchestration tools like Apache Airflow or ETL frameworks like DBT. Exposure to Snowflake (streams, tasks, stored procedures). Experience working with DuckDB and/or DocTV/OpenTI XLarge for report generation. Knowledge of Angular or frontend plugins for Excel is a plus but not required. Familiarity with async workflows and distributed processing concepts. Soft Skills & Cultural Fit: Strong ownership mindset, proactive in identifying and solving problems. Collaborative team player, able to work cross-functionally with architects, engineers, and stakeholders. Comfortable working in a distributed, multicultural team across multiple time zones (Europe, India, U.S.). Adaptable and open to learning emerging tools and technologies. Work Environment / Time Zone Expectations: Global team with flexibility on candidate location. Current team members are distributed across the US, Latin America, Europe, and India. Candidates from other regions will be considered, provided there is reasonable overlap with core collaboration hours.

  • Senior Data Engineer

    2 weeks ago

    Kollam, India Tata Consultancy Services Full time

    Role**: Senior Data EngineerRequired Technical Skill Set: Senior Data EngineerDesired Experience Range: 7 - 12 yrsNotice Period: Immediate to 90Days onlyLocation of Requirement: BangaloreWe are currently planning to do a Virtual InterviewJob Description:Primary Skills (Java, Python, etc.)Python, Scala/Java, Spark, SQL, Snowflake, BDD, Databricks, AWS Key...

  • Senior data engineer

    7 days ago

    Kollam, India Veraxion Full time

    We’re Hiring: Senior Data Engineer (Azure) We are looking for a Senior Data Engineer who can design, build, and scale modern data platforms that power analytics, decision-making, and AI across the organization. If you love solving complex data problems, architecting scalable pipelines, and working with cutting-edge Azure technologies — this role is for...

  • Senior data engineer

    7 days ago

    Kollam, India Pacific Data Integrators Full time

    Role: Senior Data Engineer Location: Remote Job Type: Full-time Shift time: Open to work in EST shift (5 PM to 2 AM IST) Key Responsibilities Lead the design, development, and implementation of complex data integration solutions using Informatica Intelligent Data Management Cloud (IDMC). Develop, document, unit test, and maintain high-quality ETL...

  • Data Engineer

    2 weeks ago

    Kollam, India Aceolution Full time

    Job Title: Data Engineer – Python Expert(Freelance Role)Location: Remote / HybridEmployment Type: Contract/ FreelanceRole Summary We are looking for a seasoned Senior Data Engineer to architect, build, and own the data pipelines that power our large language model (LLM) development. As a senior Individual Contributor (IC), you will be the team's expert on...

  • Senior Data Engineer

    3 days ago

    Kollam, India American Inference Full time

    Note: Please apply directly through for this position. We kindly request that candidates refrain from contacting company officials via email or messages regarding this role. About the Company We are an AI and Data Consulting Startup transforming how businesses leverage technology through four core service lines: - Consulting Services: AI Strategy,...

  • Senior backend engineer

    2 weeks ago

    Kollam, India Pramaan Labs Full time

    Senior Backend Engineer Location:  India - Remote Type:  Full-time Compensation: ₹20-50 LPA basis experience We are hiring a Senior Backend Engineer  on behalf of a fast-growing Saa S startup that is building intelligent inventory and forecasting systems for modern e-commerce businesses. As one of the first backend hires, you will work closely with the...

  • Senior LLM Engineers

    3 weeks ago

    Kollam, India MillionLogics Full time

    Company DescriptionMillionLogics, a trusted Oracle Partner, is an IT solutions provider with a global presence, blending innovation and strategic vision. With offices in London, UK, and a development hub in Hyderabad, India, we offer a diverse range of services, including Data & AI, Cloud Solutions, and IT Consulting. Our mission is to transform enterprises...

  • Data Engineer

    3 weeks ago

    Kollam, India Digitalzone Full time

    About the RoleAs a Data Engineer, you will design, build, and optimize data pipelines and real-time systems that power AI-driven decisioning and analytics.ResponsibilitiesDevelop and maintain scalable ETL/ELT pipelines using Python, Airflow, and dbt Build and optimize real-time streaming pipelines using Kafka, RabbitMQ, Spark and event-driven microservices...

  • Customer engineer

    2 days ago

    Kollam, India Konecta Full time

    In this role, you will be the technical point of contact for the customer – a trusted partner for their IT leadership – as the primary engineer on the project, working with multifunctional teams both internally and externally. You are a customer-facing data engineer who genuinely loves solving problems and can handle client expectations while working on...

  • Backend Python Engineer

    3 weeks ago

    Kollam, India Turing Full time

    Role Overview:We’re looking for experienced Python engineers to collaborate with one of the world’s top Large Language Model (LLM) companies. Your work will directly help improve how AI models think, reason, and code.In this role, you’ll generate and evaluate high-quality data used to fine-tune and benchmark LLMs. You’ll design prompts, analyze model...