Data Engineer – Scalable Databricks Pipelines & Platform …, City of Sydney

About the Role This is a technical delivery role focused on building reliable, scalable, and well‑governed data solutions. You will play a key role in ensuring SCI’s data is accessible, trusted, and ready to support intelligence products, analytics, and machine learning initiatives. Working closely with the Product Manager, Business Analyst, Technology team, and implementation partners, you will contribute to the delivery and ongoing evolution of SCI’s modern data platform. Key Responsibilities
- Design, build, and maintain scalable data pipelines from source systems into Databricks and SCI’s intelligence platform.
- Develop and support medallion architecture layers, ensuring data is transformed, curated, and quality assured.
- Monitor pipeline performance and data quality, proactively resolving issues and maintaining reliability.
- Maintain documentation for pipelines, integrations, transformation logic, and data lineage to support governance and audit requirements.
- Operate and support SCI’s data platform, including environment management, catalogue governance, and role‑based access controls.
- Design and maintain API integrations and external data connections supporting intelligence products and reporting.
- Support the delivery and ongoing enhancement of the Property Risk Index through reliable datasets and pipelines.
- Partner with Product Managers, Business Analysts, actuarial teams, and implementation vendors to deliver technical solutions and build internal capability.
- Enable machine learning and analytics initiatives by providing trusted, structured, and accessible datasets.
- Contribute to agile delivery practices and continuous improvement across SCI’s data engineering capability. About You You are a practical, delivery‑focused Data Engineer who enjoys solving complex data challenges and building scalable solutions that create real business impact. Skills & Experience
- Minimum 3 years’ experience as a Data Engineer with hands‑on experience building and maintaining production‑grade data pipelines.
- Solid experience with Databricks or comparable cloud data platforms such as Snowflake, Azure Synapse, or AWS Redshift.
- Advanced SQL and Python skills, including complex transformation logic and data quality validation.
- Practical experience using AI coding assistants (e.g. Claude Code, Copilot) to improve productivity in data pipeline development, testing, and documentation.
- Experience working with REST APIs, authentication methods, and integration error handling.
- Understanding of data governance concepts including data lineage, cataloguing, and role‑based access control.
- Experience working within agile delivery teams alongside Product Managers, Business Analysts, and data scientists.
- Azure cloud platform experience, including Azure Data Factory or equivalent orchestration tools.
- Exposure to AI/ML feature pipelines, vector stores, or supporting data platforms for AI workloads (e.g. Azure OpenAI, Databricks AI).
- Experience within insurance or financial services environments. Flexibility Flexibility is the cornerstone of how we work from our North Sydney office. Individual development plans tailored to your goals, supported by training and professional qualifications. Leave Benefits Options to purchase additional leave, plus paid parental leave including superannuation. Salary continuance insurance and salary sacrifice options. Access to our Employee Assistance Program (EAP) for you and your family, including mental health, nutrition, exercise, and financial wellbeing support. Working at SCI At Strata Community Insurance, we embrace diversity and inclusivity, welcoming people from all backgrounds, cultures, and experiences. We encourage you to bring your full self to work and are committed to creating an environment where everyone can thrive. You must have full working rights in Australia to be considered for this role. #J-18808-Ljbffr Apply on Kit Job: kitjobau.com/job/3rtupq