Data Engineer

20 hours ago

ahmedabad, India Tata Electronics Tepl Full-time €6,00,000 - €10,00,000 Contract

Free with email or Google

Save this job and keep your search organized

Create a free account to save jobs, create alerts and return to this listing from your dashboard.

Free with email or Google

By continuing, you agree to our Terms & Privacy Policy.

Job Responsibilities -

Architect and implement scalable offline data pipelines for manufacturing systems including AMHS, MES, SCADA, PLCs, vision systems, and sensor data.
Design and optimize ETL/ELT workflows using Python, Spark, SQL, and orchestration tools (e.g., Airflow) to transform raw data into actionable insights.
Lead database design and performance tuning across SQL and NoSQL systems, optimizing schema design, queries, and indexing strategies for manufacturing data.
Enforce robust data governance by implementing data quality checks, lineage tracking, access controls, security measures, and retention policies.
Optimize storage and processing efficiency through strategic use of formats (Parquet, ORC), compression, partitioning, and indexing for high-performance analytics.
Implement streaming data solutions (using Kafka/RabbitMQ) to handle real-time data flows and ensure synchronization across control systems.
Building dashboards using analytics tools like Grafana.
Good Understanding of Hadoop ecosystem.
Develop standardized data models and APIs to ensure consistency across manufacturing systems and enable data consumption by downstream applications.
Collaborate cross-functionally with Platform Engineers, Data Scientists, Automation teams, IT Operations, Manufacturing, and Quality departments.
Mentor junior engineers while establishing best practices, documentation standards, and fostering a data-driven culture throughout the organization.

Essential Attributes -

Expertise in Python programming for building robust ETL/ELT pipelines and automating data workflows.
Proficiency with Hadoops ecosystem.
Hands-on experience with Apache Spark (PySpark) for distributed data processing and large-scale transformations.
Strong proficiency in SQL for data extraction, transformation, and performance tuning across structured datasets.
Proficient in using Apache Airflow to orchestrate and monitor complex data workflows reliably.
Skilled in real-time data streaming using Kafka or RabbitMQ to handle data from manufacturing control systems.
Experience with both SQL and NoSQL databases, including PostgreSQL, Timescale DB, and MongoDB, for managing diverse data types.
In-depth knowledge of data lake architectures and efficient file formats like Parquet and ORC for high-performance analytics.
Proficient in containerization and CI/CD practices using Docker and Jenkins or GitHub Actions for production-grade deployments.
Strong understanding of data governance principles, including data quality, lineage tracking, and access control.
Ability to design and expose RESTful APIs using FastAPI or Flask to enable standardized and scalable data consumption.

Qualifications -

Desired Experience Level -