AI Talent

AI Talent is an Australian-owned company established in 2010, specializing in providing exceptional AI professionals for impactful projects. With over 25 years of industry experience, they offer…

AI TalentArtificial IntelligenceAustraliaCloudData ScienceIT StaffingMachine LearningRecruitmentaitalent.com.au

AI Talent

AI Talent is an Australian-owned company established in 2010, specializing in providing exceptional AI professionals for impactful projects. With over 25 years of industry experience, they offer…

AI TalentArtificial IntelligenceAustraliaCloudData ScienceIT StaffingMachine LearningRecruitmentaitalent.com.au

HQSydney, AU

Team Size48

Open Jobs18

Total Funding-

Latest FundraiseUnknown

Join the Team

Senior Databricks Developer

HybridSydney, New South Wales, AU

Hybrid • Sydney, New South Wales, AU

Startup jobs. A lot of them.

Your next opportunity is in here somewhere. Sign up to explore 52,000+ startups and their open roles. No spam. No gamification. Just jobs.

52,000+

Startups

66,000+

Open Roles

1,500+

New This Week

Technical Writer

Part-timeAustin, US

Part-time • Austin, US

DevOps Engineer

Full-timeAmsterdam, NL

Full-time • Amsterdam, NL

Data Scientist

Full-timeHaifa

Full-time • Haifa

Product Designer

InternshipHaifa

Internship • Haifa

Data Scientist

Full-timeTel Aviv

Full-time • Tel Aviv

Technical Writer

Full-timeHamburg, DE

Full-time • Hamburg, DE

We're partnering with a vital leader in the Government Health sector, dedicated to optimising public health outcomes, resource allocation, and policy analysis through a modern, cloud-native data platform. For the right candidate with the necessary skills and experience, we are pleased to offer 482 visa sponsorship .

This client requires a Senior Databricks Developer to serve as a technical leader for their advanced Analytics and Research workloads. You will be instrumental in architecting and building highly scalable data pipelines using PySpark/Scala within the Databricks Lakehouse Platform. This role demands expertise in Delta Lake, performance optimisation , and ensuring strict data governance and security for sensitive patient data, adhering to all compliance and privacy regulations .

What You'll Do

What You'll Bring

Lead the design and development of large-scale, resilient, and performant ETL/ELT data pipelines using PySpark/Scala within Databricks notebooks and jobs.

Architect and manage the Delta Lake environment, focusing on data ingestion, quality enforcement (using Delta Live Tables or similar), and schema evolution for complex public health datasets.

Optimise Databricks clusters, notebooks, and Spark jobs for cost-efficiency and performance, specifically targeting bottlenecks in high-volume batch and streaming workloads.

Define and enforce data governance practises within the Lakehouse, utilizing Unity Catalogue for centralized metadata and access control, adhering to government standards.

Collaborate closely with government analysts and data scientists to transition analytical models and research findings into scalable, production-ready pipelines.

Champion CI/CD and MLOps practises for Databricks notebooks and workflows, utilizing tools like Azure DevOps or Jenkins.

Mentor and guide junior engineers on Databricks development standards, Spark optimisation , and modern data engineering practises .

6+ years of progressive professional experience in Data Engineering, with at least 3 years dedicated to developing solutions on the Databricks Platform .

Expert-level proficiency in PySpark and/or Scala for distributed data processing.

Mandatory hands-on experience with Delta Lake architecture , including DLT, time travel, and VACUUM operations.

Deep understanding of cloud infrastructure ( Azure preferred) and how Databricks integrates with cloud storage (ADLS Gen2) and services.

Expert proficiency in SQL and dimensional modelling principles.

Proven experience with CI/CD, Infrastructure as Code (e.g., Terraform), and Databricks command-line tools for automation.

Exceptional communication and problem-solving skills, with the ability to analyse complex requirements and design resilient solutions in a highly regulated environment.