How Melbourne's Archives Ended Up Full of Duplicate Images — and What It Took to Finally Fix It

Victoria's public sector holds tens of millions of digital image files across dozens of agencies, and a significant portion of them are exact or near-exact duplicates. That is the unambiguous conclusion reached by a 2024 audit of digital asset management practices commissioned through the Victorian Government's Digital Strategy framework — a finding that has quietly shaped procurement decisions and IT reform efforts across Melbourne ever since.

The problem did not appear overnight. It accumulated over roughly fifteen years, beginning with the first wave of mass digitisation drives that swept through state institutions after 2008, when the Rudd government's stimulus funding reached cultural agencies and public libraries. PROV — the Public Record Office Victoria, headquartered on Macarthur Place in Carlton — was among the first to flag the issue internally, as scanning contractors delivered batches of files in incompatible formats with inconsistent naming conventions. Without a shared taxonomy, agencies simply stored what they received, duplicates and all.

A System Built for Speed, Not Sustainability

The real acceleration came between 2015 and 2020. State Library Victoria, on Swanston Street in the CBD, ran multiple concurrent digitisation grants during that period, each administered by different internal teams using different cataloguing tools. Museums Victoria, whose collections management spans the Melbourne Museum in Carlton Gardens and the Immigration Museum on Flinders Street, faced similar fragmentation. Each institution's IT environment grew independently, and cross-agency deduplication was nobody's specific responsibility.

Cloud migration made things worse before it made them better. When agencies shifted storage to platforms managed under whole-of-government contracts from around 2018 onward, files were often migrated in bulk without prior deduplication. Storage costs under those arrangements are typically billed per gigabyte, meaning duplicate files have represented a direct, ongoing budget line — not just an organisational inconvenience. The Department of Government Services has not publicly released a total cost figure attributable to duplicate storage across the Victorian public sector, but the broader national picture gives a sense of scale: the Australian National Audit Office's 2023 review of digital asset management across federal agencies found that redundant file storage accounted for a measurable share of avoidable cloud spend at multiple departments.

Private sector institutions in Melbourne were not immune either. The Australian Centre for the Moving Image, based at Federation Square, undertook a major collection audit in 2022 that identified duplicate master files dating back to its 2002 opening. The State Archives and Records Authority of New South Wales publicly documented a comparable problem in 2021, which gave Victorian counterparts a documented case study to draw on when making the argument internally for dedicated remediation resources.

What the Cleanup Actually Involves

Duplicate image replacement — the process of identifying redundant files, designating a canonical master, and systematically retiring or re-pointing references to the duplicates — is technically straightforward but administratively complex. The challenge is not finding duplicates. Modern perceptual hashing tools can do that in hours across collections of millions of files. The challenge is governance: deciding which copy is the authoritative one, updating downstream references in catalogues and public-facing portals, and ensuring provenance metadata is preserved.

Victoria's approach, coordinated through the Office of the Chief Information Officer and supported by the Digital Strategy unit within the Department of Government Services, has involved piloting a shared deduplication workflow across three agencies since late 2025. State Library Victoria's Redmond Barry Building reading room collections were among the first to go through the process. The pilot is expected to inform a broader rollout to other Victorian public institutions across the 2026-27 financial year.

For cultural institutions and government agencies still managing legacy collections, the practical priority is establishing a single source of truth before the next procurement cycle. Institutions that have not yet mapped their storage environments against a consistent metadata standard are likely to find any future migration — to updated cloud contracts or new collection management systems — significantly more expensive and error-prone than it needs to be. The window to get ahead of the problem, before the next wave of AI-assisted cataloguing tools reshapes expectations around collection quality, is narrowing.