Data Engineer

Overview:

We are looking for a highly skilled Python Data Engineer to join our team in an on-premise data engineering environment. The ideal candidate will have experience in ETL tools, data processing technologies, data orchestration, and relational databases. Additionally, you should be proficient in Python scripting for data engineering tasks and have experience working with Spark, PySpark, and other relevant data technologies. While cloud tools are a good-to-have, this position primarily focuses on on-premise data infrastructure.

This is an excellent opportunity to work on exciting projects that require developing scalable data pipelines, real-time data streaming, and optimizing data processing tasks using Python.

Key Responsibilities:

• ETL Development & Optimization: Design, develop, and optimize ETL pipelines using open-source or cloud ETL tools (e.g., Apache Nifi, Talend, Pentaho, Airflow, AWS Glue).
• Python Scripting for Data Engineering: Write Python scripts to automate data extraction, transformation, and loading (ETL) processes. Ensure that the code is optimized for performance and scalability.
• Big Data Processing: Work with Apache Spark and PySpark to process large datasets in a distributed computing environment. Optimize Spark jobs for performance and resource efficiency.
• Job Orchestration: Use Apache Airflow or other orchestration tools to schedule, monitor, and automate data pipeline workflows.
• Data Streaming: Design and implement real-time data streaming solutions using technologies like Apache Kafka or AWS Kinesis for high-throughput, low-latency data processing.
• File Formats & Table Formats: Work with open-source table formats like Apache Parquet, Apache Avro, or Delta Lake, and other structured/unstructured data formats for efficient data storage and access.
• Database Management: Work with relational databases (e.g., PostgreSQL, MySQL, SQL Server) for data storage, management, and optimization. Understand database concepts such as normalization, indexing, and query optimization.
• SQL Expertise: Write and optimize complex SQL queries for data extraction, transformations, and aggregation across large datasets. Ensure queries are efficient and scalable.
• BI & Data Warehouse Knowledge: Exposure to BI tools and data warehousing concepts is a plus, ensuring the data is structured in a way that supports analytics and reporting.

Required Skills & Experience:

• ETL Tools: Experience working with open-source ETL tools such as Apache Nifi, Talend, or Pentaho. Cloud-based tools like AWS Glue or Azure Data Factory are good to have.
• Python Scripting: Proficiency in Python for automating data processing tasks, writing data pipelines, and working with libraries such as Pandas, Dask, PySpark, etc.
• Big Data Technologies: Experience with Apache Spark and PySpark for distributed data processing, along with optimization techniques.
• Data Orchestration: Experience using Apache Airflow or similar tools for scheduling and automating data pipelines.
• Data Streaming: Experience with Apache Kafka or AWS Kinesis for building and managing real-time data pipelines.
• Open-Source File Formats: Knowledge of Apache Parquet, Apache Avro, Delta Lake, or similar open-source table formats for efficient data storage and retrieval.
• Relational Databases: Strong experience with at least one relational database (e.g., PostgreSQL, MySQL, SQL Server) and a solid understanding of database concepts like indexing, normalization, and query optimization.
• SQL Expertise: Strong skills in writing and optimizing complex SQL queries for data extraction, transformations, and aggregation.

Nice to Have:

• BI/Analytics Tools: Familiarity with BI tools like Power BI, Tableau, Looker, or similar reporting and data visualization platforms.
• Data Warehousing: Knowledge of data warehousing principles, schema design (e.g., star/snowflake), and optimization techniques for large datasets.
• Cloud Technologies: Experience with cloud data platforms like Databricks, Snowflake, or Azure Synapse is beneficial, though the role is focused on on-prem environments.
• Containerization: Familiarity with containerization tools like Docker or Kubernetes for deploying data engineering workloads.

Educational Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related field (or equivalent work experience).

Additional Qualities:

• Excellent problem-solving and troubleshooting skills.
• Ability to work both independently and in a collaborative environment.
• Strong communication skills, both written and verbal.
• Detail-oriented with a focus on data quality and performance optimization.
• Proactive attitude and the ability to take ownership of projects.