Python Data Architect

As a skilled Python data engineering professional, you will design, develop, and optimize ETL pipelines using open-source or cloud ETL tools. You will also work on exciting projects that require developing scalable data pipelines, real-time data streaming, and optimizing data processing tasks using Python.

Key Responsibilities:

• Design, develop, and optimize ETL pipelines using Apache Nifi, Talend, Pentaho, Airflow, and AWS Glue.
• Write Python scripts to automate data extraction, transformation, and loading (ETL) processes, ensuring that the code is optimized for performance and scalability.
• Work with Apache Spark and PySpark to process large datasets in a distributed computing environment, optimizing Spark jobs for performance and resource efficiency.
• Use Apache Airflow or other orchestration tools to schedule, monitor, and automate data pipeline workflows.
• Design and implement real-time data streaming solutions using technologies like Apache Kafka or AWS Kinesis for high-throughput, low-latency data processing.
• Work with open-source table formats like Apache Parquet, Apache Avro, or Delta Lake, and other structured/unstructured data formats for efficient data storage and access.
• Work with relational databases like PostgreSQL, MySQL, or SQL Server for data storage, management, and optimization, understanding database concepts such as normalization, indexing, and query optimization.
• Write and optimize complex SQL queries for data extraction, transformations, and aggregation across large datasets, ensuring queries are efficient and scalable.

Required Skills & Qualifications:

• Experience working with open-source ETL tools like Apache Nifi, Talend, or Pentaho, with cloud-based tools like AWS Glue or Azure Data Factory being beneficial.
• Proficiency in Python for automating data processing tasks, writing data pipelines, and working with libraries like Pandas, Dask, PySpark, etc.
• Experience with Apache Spark and PySpark for distributed data processing, along with optimization techniques.
• Experience using Apache Airflow or similar tools for scheduling and automating data pipelines.
• Experience with Apache Kafka or AWS Kinesis for building and managing real-time data pipelines.
• Knowledge of Apache Parquet, Apache Avro, Delta Lake, or similar open-source table formats for efficient data storage and retrieval.
• Strong experience with at least one relational database like PostgreSQL, MySQL, or SQL Server, with a solid understanding of database concepts like indexing, normalization, and query optimization.
• Strong skills in writing and optimizing complex SQL queries for data extraction, transformations, and aggregation.

Nice to Have:

• Familiarity with BI tools like Power BI, Tableau, Looker, or similar reporting and data visualization platforms.
• Knowledge of data warehousing principles, schema design (e.g., star/snowflake), and optimization techniques for large datasets.
• Experience with cloud data platforms like Databricks, Snowflake, or Azure Synapse is beneficial, though the role is focused on on-prem environments.
• Familiarity with containerization tools like Docker or Kubernetes for deploying data engineering workloads.

Educational Qualifications:

• Bachelor's or Master's degree in Computer Science, Engineering, Information Systems, or a related field (or equivalent work experience).

Additional Qualities:

• Excellent problem-solving and troubleshooting skills.
• Ability to work both independently and in a collaborative environment.
• Strong communication skills, both written and verbal.
• Detail-oriented with a focus on data quality and performance optimization.
• Proactive attitude and the ability to take ownership of projects.