Senior Data Engineer Position

Role Overview:

Seeking an expert Big Data Engineer to work on exciting projects that require developing scalable data pipelines, real-time data streaming, and optimizing data processing tasks.

Key Responsibilities:

• ETL Development & Optimization
  • Design, develop, and optimize ETL pipelines using open-source or cloud ETL tools (e.g., Apache Nifi, Talend, Pentaho, Airflow, AWS Glue).
• Python Scripting for Data Engineering
  • Write Python scripts to automate data extraction, transformation, and loading (ETL) processes. Ensure that the code is optimized for performance and scalability.
• Big Data Processing
  • Work with Apache Spark and PySpark to process large datasets in a distributed computing environment. Optimize Spark jobs for performance and resource efficiency.
• Job Orchestration
  • Use Apache Airflow or other orchestration tools to schedule, monitor, and automate data pipeline workflows.
• Data Streaming
  • Design and implement real-time data streaming solutions using technologies like Apache Kafka or AWS Kinesis for high-throughput, low-latency data processing.
• File Formats & Table Formats
  • Work with open-source table formats like Apache Parquet, Apache Avro, or Delta Lake, and other structured/unstructured data formats for efficient data storage and access.
• Database Management
  • Work with relational databases (e.g., PostgreSQL, MySQL, SQL Server) for data storage, management, and optimization. Understand database concepts such as normalization, indexing, and query optimization.
• SQL Expertise
  • Write and optimize complex SQL queries for data extraction, transformations, and aggregation across large datasets. Ensure queries are efficient and scalable.
• BI & Data Warehouse Knowledge
  • Exposure to BI tools and data warehousing concepts is a plus, ensuring the data is structured in a way that supports analytics and reporting.

Required Skills & Qualifications:

• ETL Tools
  • Experience working with open-source ETL tools such as Apache Nifi, Talend, or Pentaho. Cloud-based tools like AWS Glue or Azure Data Factory are beneficial.
• Python Scripting
  • Proficiency in Python for automating data processing tasks, writing data pipelines, and working with libraries such as Pandas, Dask, PySpark, etc.
• Big Data Technologies
  • Experience with Apache Spark and PySpark for distributed data processing, along with optimization techniques.
• Data Orchestration
  • Experience using Apache Airflow or similar tools for scheduling and automating data pipelines.
• Data Streaming
  • Experience with Apache Kafka or AWS Kinesis for building and managing real-time data pipelines.
• Open-Source File Formats
  • Knowledge of Apache Parquet, Apache Avro, Delta Lake, or similar open-source table formats for efficient data storage and retrieval.
• Relational Databases
  • Strong experience with at least one relational database (e.g., PostgreSQL, MySQL, SQL Server) and a solid understanding of database concepts like indexing, normalization, and query optimization.
• SQL Expertise
  • Strong skills in writing and optimizing complex SQL queries for data extraction, transformations, and aggregation.

Benefits:

• Nice to Have
  • Familiarity with BI tools like Power BI, Tableau, Looker, or similar reporting and data visualization platforms.
• Cloud Technologies
  • Experience with cloud data platforms like Databricks, Snowflake, or Azure Synapse is beneficial, though the role is focused on on-prem environments.
• Containerization
  • Familiarity with containerization tools like Docker or Kubernetes for deploying data engineering workloads.

Additional Requirements:

• Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related field (or equivalent work experience).

Additional Qualities:

• Excellent problem-solving and troubleshooting skills.
• Ability to work both independently and in a collaborative environment.
• Strong communication skills, both written and verbal.
• Detail-oriented with a focus on data quality and performance optimization.
• Proactive attitude and the ability to take ownership of projects.