• Design, build, and optimize ETL/ELT workflows for structured and unstructured datasets.
• Develop data pipelines for ingestion, transformation, and loading into data warehouses or data lakes.
• Write efficient SQL queries, transformations, stored procedures, and data scripts.
• Implement incremental loads, change-data-capture (CDC), and automated data processes.

2. Data Integration

• Integrate data from multiple internal and external systems (APIs, DBs, files, applications).
• Seamlessly connect ETL pipelines to cloud storage, relational databases, streaming sources, or third-party platforms.
• Handle data profiling, cleansing, mapping, and validation.

3. Data Quality & Validation

• Develop data quality rules, validation checks, reconciliation processes, and exception handling.
• Monitor data pipelines and resolve issues proactively.
• Ensure accuracy, completeness, and consistency of all processed data.

4. Data Architecture & Modelling Support

• Collaborate with data architects to design scalable data models and ETL frameworks.
• Implement SCD (Slowly Changing Dimensions), fact and dimension design, and OLAP/OLTP optimizations.
• Contribute to data governance, metadata management, and documentation standards.

5. Automation, Scheduling & Performance

• Automate workflows using schedulers (Airflow, ADF, Control-M, etc.).
• Tune ETL performance (query optimization, parallel processing, partitioning).
• Assist in CI/CD pipeline setup for ETL deployments (Git, DevOps pipelines).

6. Collaboration & Stakeholder Engagement

• Work with BI teams, analysts, developers, and business owners to understand requirements.
• Translate functional requirements into technical ETL designs.
• Document data flows, mapping specifications, and transformation logic clearly.

QUALIFICATION REQUIRMENTS AND EXPERIENCE :

• 2–5+ years experience in data engineering or ETL development.
• Strong SQL background is essential.
• Bachelor’s degree in Computer Science, Information Systems, Engineering, or equivalent experience.

KNOWLEDGE:

• SSIS / Informatica / Talend / Matillion / Pentaho / Azure Data Factory / AWS Glue / GCP Dataflow / Airflow / dbt (for ELT)
• SAP Data Services (if applicable) Programming / Querying / Strong SQL (required) / Python or Scala (for transformations, scripting, automation) / Bash/PowerShell for system scripting (nice-to-have).
• Databases & Storage. SQL Server, Oracle, MySQL, PostgreSQL, Cloud storage (ADLS, S3, GCS), Data warehouses (Snowflake, Redshift, BigQuery, Synapse).
• Data Engineering & Integration, API integrations (REST, SOAP), File formats (CSV, Parquet, Avro, JSON, XML)
• Data modelling (dimensional modelling, star schema, SCD types)
• Data Quality & Governance
• Great Expectations or similar DQ tools
• Metadata management tools (Collibra, Purview, Alation)
• DevOps & Deployment
• Git / GitHub / Azure DevOps / GitLab
• CI/CD pipelines for ETL
• Scheduler tools (Airflow, Control-M, cron jobs)
• Required Competencies
• Solid analytical and problem-solving skills.
• Strong communication and documentation capabilities.
• Ability to work in agile or hybrid project environments.\Detail-oriented and committed to data accuracy.
• Ability to handle large, complex datasets.
• Nice-to-Have Skills
• Experience with modern ELT (dbt, Databricks workflows).
• Streaming experience (Kafka, Spark Streaming, EventHub).
• Data lakehouse concepts (Delta Lake, Iceberg, Hudi).
• Working knowledge of BI tools (Power BI, Tableau).
• Banking or financial sector experience.

 Certifications such as:

o Azure/AWS/GCP Data Engineer

• o Informatica/Talend/ADF professional certifications