Melbourne's Digital Archivists Race to Fix a Decade of Duplicate Images Swamping Council and Cultural Records

Melbourne's State Library on Swanston Street flagged the problem formally in late June: tens of thousands of duplicate image files clogging its digital collections system, some triplicated across servers, others mislabelled and orphaned since at least 2019. The library is now part of a broader push, quietly gathering momentum across Victorian public institutions, to deploy automated duplicate-image-replacement software before the end of the 2025–26 financial year.

The timing matters because Victorian government agencies face a hard deadline. The Department of Government Services confirmed in a May circular — referenced in budget supplementary documents tabled in the Legislative Council — that agencies must complete digital asset audits by August 31, 2026, or risk losing access to the shared cloud storage subsidy that currently offsets infrastructure costs. For institutions that have accumulated years of unmanaged image libraries, that is a very short runway.

Who Is Affected and What the Audit Found

The City of Melbourne's digital communications team, operating out of offices in Little Collins Street, found during a May internal review that its media asset library contained duplicate image files accounting for a material share of total storage consumption — a finding that prompted escalation to the council's IT governance committee in June. The council declined to confirm the exact storage figure publicly while the audit is ongoing.

At RMIT University's digital media faculty in the CBD, archivists working on the university's institutional repository reported a similar pattern. Staff there began trialling a hash-based duplicate detection tool in April, scanning collections of digitised photographic prints and event photography accumulated since 2012. Hash-based detection works by generating a unique fingerprint for each image file; exact duplicates produce identical hashes and can be flagged for automated removal or replacement with a single canonical version.

The Victorian Collections platform, which aggregates digital objects from more than 200 museums and historical societies statewide, has also been drawn into the conversation. Platform administrators acknowledged at a May stakeholder briefing in Carlton that some contributing organisations had uploaded the same image file multiple times under different metadata records — a structural problem that duplicate-image-replacement workflows are now being designed to address systematically.

Storage costs are not trivial. Commercial cloud storage pricing for large institutions typically runs between $25 and $45 per terabyte per month depending on the access tier, and organisations that have allowed duplicate proliferation over several years may be paying for two or three times the storage they actually need. For a mid-sized council or cultural institution holding five to ten terabytes of image assets, the annual waste can reach tens of thousands of dollars.

Tools, Vendors and What Comes Next

Two software vendors are currently being evaluated through a Victorian Government procurement panel: one Melbourne-based company operating out of Cremorne, and a Sydney firm. Both offer similar core functionality — perceptual hashing to catch near-duplicate images, not just exact copies, which is critical for photographic collections where the same image may have been resaved at different resolutions or with different colour profiles.

Perceptual hashing is the key differentiator from older tools. Unlike exact-match detection, it can identify two JPEG files that are visually identical but differ in file size by 15 per cent — the kind of discrepancy that routine compression or format conversion creates, and that has made earlier deduplication attempts only partially effective.

For institutions not yet engaged with the procurement panel, the practical advice from digital preservation specialists is straightforward: begin with a read-only audit before touching any files. Automated replacement tools can make irreversible decisions if configured incorrectly, and cultural collections have specific legal obligations around maintaining provenance records under the Public Records Act 1973 (Vic).

The State Library has indicated it expects to complete its first-pass deduplication of non-archival working files by late September 2026, which would put it ahead of most peer institutions. For the dozens of smaller councils and galleries across Melbourne — from Moreland to Port Phillip — the August deadline is already close enough to cause concern among digital records officers who spoke at a Local Government Professionals Victoria session in Melbourne last month.