Evaluation & Research

Overview

This research examined the prevalence of mental health conditions among people seeking alcohol and other drug (AOD) treatment in Victoria and assessed how AOD services respond to these co-occurring needs. The Victorian Alcohol and Drug Association (VAADA) commissioned this research to better understand the resourcing demands and treatment approaches in the sector.

[View keynote VAADA Conference presentation here]

Need

Despite multiple policy initiatives over two decades, people with co-occurring mental health and AOD needs continue to face significant barriers to integrated care. The Royal Commission into Victoria's Mental Health System (2021) found little substantial progress in improving outcomes for this group.

Approach

The research gathered data through a sector-wide survey of 29 AOD organisations, focus groups with clinicians and executives, and interviews with people with lived experience of mental illness and substance use problems.

Key Outcomes

·       83% of AOD clients had a mental health condition (46% formally diagnosed, 37% with symptoms but no diagnosis)

·       AOD services spent more than half of their clinical time supporting clients with co-occurring needs

·       Most commonly reported co-occurring conditions: Anxiety, trauma, depressive, and personality disorders

Capabilities engaged

·       Mixed-methods research

·       Stakeholder consultation

·       Lived experience co-design and participation

·       Quantitative and qualitative data analysis

·       Systems-level policy analysis

·       Evidence based reporting

Population Groups Supported

·       People with co-occurring mental health and substance use issues

·       Family members and carers supporting people with a dual diagnosis

·       AOD service providers and clinicians

Impact

The findings highlight the need to expand funding for mental health support and specialist mental health training in AOD service settings. The research demonstrated that AOD services are often the primary point of care for people with co-occurring needs who face barriers accessing mental health services.

Client: Victorian Alcohol and Drug Association (VAADA)

Project Size: $30K-$50K

Overview

We supported the Youth Support and Advocacy Service (YSAS) to develop a program logic, complete an outcomes evaluation, and calculate some estimated government avoided costs for their Transformer youth crime prevention program. The project demonstrated significant positive outcomes across multiple dimensions and quantified the economic benefits of early intervention with at-risk young people.

Need

Young people from culturally diverse backgrounds who are in contact with the criminal justice system face multiple challenges including alcohol and other drug (AOD) issues and disengagement from education, employment, and training (EET). These young people require intensive, culturally responsive support to address complex needs and prevent further entrenchment in the criminal justice system. Government stakeholders needed evidence of program effectiveness and economic value.

Approach

We designed a evaluation framework that measured outcomes across three timeframes: short-term outcomes (safety, stability, and security), intermediate outcomes (connectedness, self-worth, hope and motivation, and agency), and longer-term impacts (reduced AOD risk and harm, reduced criminal justice system engagement, and increased EET engagement). Data from 90 young people who participated in the program over the past year was analysed.

Key outcomes

The Transformer Program demonstrated:

·       A 60% reduction in AOD risk and harm among participants

·       A 72% reduction in criminal justice system engagement

·       A 68% increase in EET engagement over a 12-month period

·       Significant avoided government costs in youth justice and welfare

Population groups supported

·       Young people from culturally diverse backgrounds

·       Young people in contact with the criminal justice system

·       Young people experiencing AOD risk and harm

·       Young people disengaged from education, employment and training

·       Families of young people facing multiple challenges

Impact

The evaluation demonstrated that intensive case management combined with development programs, education and employment links, and family support services created significant positive change for young people at risk. The avoided costs modelled showed strong potential economic benefits to the community through reduced government costs.

Capabilities engaged

·       Literature and practice review

·       Program logic development

·       Outcome evaluation design and implementation

·       Avoided costs research and analysis

·       Mixed-methods data collection and analysis

·       Stakeholder engagement

Client: Youth Support and Advocacy Service (YSAS)

 Project size: $30-$50K

Overview

We assisted the Housing Justice Partnership to develop a Theory of Change and complete an outcome and impact evaluation. The evaluation included an preliminary estimate of cost savings to government through earlier intervention and integrated service delivery.

Need

People experiencing homelessness face significant barriers to accessing legal assistance despite having complex legal needs. Traditional legal service models often fail to reach this vulnerable population, resulting in unresolved legal issues that can perpetuate homelessness.

Approach

We designed a mixed-methods evaluation using client service data from 292 clients, interviews with clients and staff, surveys, and journey mapping to assess the effectiveness of co-locating legal services within housing and homelessness programs.

Key outcomes

78% of legal matters for clients experiencing homelessness were satisfactorily resolved or resolved in part

67% of clients with tenancy matters achieved improved housing security

69% of clients experiencing homelessness reported reduced stress after receiving legal assistance

$650,000 in estimated health, justice and welfare costs avoided through early intervention

Population groups supported

People experiencing or at risk of homelessness (88% of clients)

People with complex legal needs, particularly civil and criminal matters

People experiencing financial disadvantage (100% of clients)

People living with disability and/or mental health issues (69% of clients)

Impact

The evaluation demonstrated that accessible, integrated legal and housing services significantly improved both legal and non-legal outcomes for vulnerable clients. Co-location of services at housing sites proved particularly effective in reaching clients who would otherwise not access legal assistance, while also building capacity among housing workers to identify legal issues earlier.

Capabilities engaged

Theory of Change development

Mixed-methods evaluation design

Stakeholder consultation and engagement

Journey mapping and cost avoidance analysis

Qualitative thematic analysis

Evaluation framework development

Evidence-based reporting and knowledge translation

Client: Inner Melbourne Community Legal (IMCL), The Salvation Army, and VincentCare

Project size: $30K-$50K

Overview

We assisted The Salvation Army and Gladstone Regional Council in developing a Theory of Change and monitoring and evaluation framework for the Philip Street Communities and Families Precinct. This collaborative project established a clear vision and pathway for how the Precinct would create positive change in the Gladstone community through integrated service delivery.

Need

The Gladstone region needed a coordinated approach to community services delivery to address fragmentation in the social support system. The Philip Street Precinct was developed to strengthen health and wellbeing across the region by clustering services together, improving partnerships, and encouraging community ownership of shared spaces.

Approach

We facilitated a collaborative process with key stakeholders to develop a comprehensive Theory of Change that mapped how the Precinct's activities would lead to desired outcomes. We then designed a fit-for-purpose evaluation framework with clear indicators to measure progress across three key domains: foundations for change, enabling conditions, and community awareness and engagement.

Key outcomes

·       Development of a comprehensive service model grounded in place-based and asset-based community development approaches

·       Creation of a Theory of Change visualizing pathways from activities to long-term community impacts

·       Design of an evaluation framework with practical tools for measuring progress and impact

·       Establishment of clear governance structures that embed community voice in decision-making

Population groups supported

·       General Gladstone community members seeking to improve wellbeing and social connections

·       Vulnerable community members requiring targeted support services

·       Community groups and associations seeking spaces for activities

·       First Nations people, youth, seniors, people with disabilities, and culturally diverse communities

Impact

The Theory of Change and evaluation framework provided a strong foundation for the Precinct's operations, ensuring services remained aligned with community needs and priorities. The framework enabled partners to track progress, demonstrate value, and continuously improve service delivery through evidence-based decision making.

Capabilities engaged

·       Theory of Change development and facilitation

·       Evaluation framework design

·       Stakeholder consultation and engagement

·       Service model documentation

·       Governance structure development

·       Community engagement strategy

·       Mixed-methods evaluation design

Client: The Salvation Army, Gladstone Regional Council, and GAPDL Communities for Children

Project size: <$30K

Overview

We assisted the Youth Support and Advocacy Service (YSAS) to complete an outcome evaluation of their Family and Youth Early Intervention Program (FaYEIP). The evaluation analysed five years of program data (2017-2022) to assess effectiveness in supporting young people at risk of developing alcohol and other drug (AOD) problems and their families.

Need

Young people with emerging AOD problems face significant service gaps in Victoria. There is a lack of early intervention services that can prevent progression to more harmful substance use, and poor coordination between youth AOD and family support systems. FaYEIP was established to address these gaps by providing integrated early intervention before problems become entrenched.

Approach

We designed a retrospective outcomes evaluation that analysed data from 218 client files (72% of all clients) to assess changes across five key outcome areas. The evaluation included staff-reported data, client satisfaction surveys, and stakeholder consultations. We also documented the program's Theory of Change and service model to contextualise findings.

Key outcomes

·       74% of young people reduced their progression to more harmful substance use

·       88% of young people improved family connections and communication

·       97% of families increased their confidence and capacity to support young people

·       61% of young people improved their education and community connections

Population groups supported

·       Young people aged 12-18 years at risk of developing AOD problems

·       Families and caregivers of young people with emerging AOD issues

·       Young people following first engagement with police

·       Culturally diverse communities across Southern Metropolitan Melbourne

Impact

The evaluation demonstrated that FaYEIP's integrated approach effectively prevents progression to more harmful substance use while strengthening family relationships. Providing services to both young people and their families together yielded the strongest outcomes. The program successfully diverts young people from the criminal justice system and creates pathways to additional support when needed.

Capabilities engaged

·       Literature review

·       Outcome evaluation design and implementation

·       Theory of Change development

·       Mixed-methods data analysis

·       Service model documentation

·       Stakeholder consultation

·       Attribution analysis

Client: Youth Support and Advocacy Service (YSAS)

Project size: $30K-$50K

Overview

We completed a comprehensive review of the Wilum Housing Development Program, which provides transitional accommodation to young people recovering from substance use problems. The review examined the program's service model, operational processes, and outcomes, while identifying opportunities to enhance its effectiveness through a stronger focus on youth development principles.

Need

The Wilum Program had not been reviewed since its inception and had not kept pace with recent developments in youth housing approaches. The program was unable to meet the demand for supported accommodation places and lacked clarity around its service model. Young people recovering from substance use problems required a structured pathway to develop independent living skills and transition to long-term housing.

Approach

We conducted a systematic review using multiple data sources including consultations with staff and managers, analysis of program documentation and tools, and desktop research on evidence-based practices in youth housing. We examined the program's theoretical frameworks, eligibility criteria, referral pathways, assessment processes, and service delivery activities to identify strengths, gaps and opportunities for improvement.

Key outcomes

·       Developed recommendations to reposition Wilum as an aspirational development program rather than supported accommodation

·       Identified the need to adapt youth foyer principles, particularly the Open Talent framework, to strengthen the program's development focus

·       Recommended restructuring staff roles from AOD Outreach Workers to Development Coaches

·       Identified strategies to strengthen housing provider relationships and improve property utilisation

·       Developed a phased approach to program expansion that would create clearer pathways to long-term housing

Population groups supported

·       Young people aged 18-21 years recovering from substance use problems

·       Young people demonstrating readiness for independent living with support

·       Young people committed to education, employment and training activities

·       Young people from diverse backgrounds including First Nations, LGBTIQA+, and culturally diverse communities

Impact

The review provided a clear roadmap for transforming Wilum from a support-focused program to a development-focused initiative that better prepares young people for independent living. The recommendations emphasised creating structured pathways to long-term housing while building young people's skills, community connections, and participation in education and employment.

Capabilities engaged

·       Service model review and development

·       Evidence-based practice research

·       Stakeholder consultation and engagement

·       Program evaluation and outcomes assessment

·       Housing pathway development

·       Organisational change management

Resource planning and allocation

Client: Youth Support and Advocacy Service (YSAS)

Project size: <$30K