SF City Hall's Duplicate Image Problem: Key Decisions Ahead After Years of Digital Neglect

San Francisco's Department of Technology has flagged a growing problem across multiple city agencies: thousands of duplicate digital images clogging shared servers, inflating storage costs, and complicating public records requests that depend on clean, searchable archives. The question now is how — and how fast — the city moves to fix it.

The issue matters right now because several city departments are in the middle of significant digital infrastructure overhauls. The San Francisco Planning Department, which handles environmental review documents and project renderings for everything from Mission District infill housing to Tenderloin SRO renovations, relies on image archives that staff and members of the public search regularly. When the same image appears under three different file names across two separate servers, it slows searches, creates version-control headaches, and can trip up California Public Records Act responses that are already subject to strict statutory deadlines.

Where the Problem Lives

The duplication issue is concentrated in a handful of high-volume departments. The San Francisco Municipal Transportation Agency, which manages Muni and oversees capital projects along corridors like Van Ness Avenue and the Central Subway extension, generates large volumes of project photography, inspection imagery, and before-and-after documentation. The Department of Public Works, responsible for street conditions from the Embarcadero to Ocean Avenue, similarly accumulates redundant site photos uploaded by field crews using different devices and naming conventions.

At City Hall, the Office of the City Administrator has been quietly piloting a deduplication protocol since late 2025 using software that identifies bit-for-bit identical files as well as near-duplicate images — shots of the same location taken seconds apart. Sources familiar with municipal IT procurement say the licensing costs for enterprise-grade deduplication tools typically run between roughly $40,000 and $120,000 annually for a government deployment of San Francisco's scale, though the city has not publicly confirmed its specific contract figures.

The San Francisco Public Library's digital collections branch at the main branch on Larkin Street has dealt with this problem longer than most. Librarians managing the San Francisco Historical Photograph Collection — which includes tens of thousands of digitized images spanning more than a century — developed internal deduplication workflows years ago, in part because donor submissions routinely included copies of images already in the archive. That institutional knowledge may prove useful as other city agencies now grapple with the same challenge at larger scale.

Decisions That Will Shape the Outcome

Three choices will define what happens next. First, city technology officials need to decide whether to pursue a centralized deduplication pass across all agency servers simultaneously or tackle departments one at a time. A phased approach reduces operational risk but stretches the timeline; a centralized push could take months to execute but would yield faster results for public records compliance.

Second, departments must settle on a master file-naming standard. Without one, new duplicates will accumulate as quickly as old ones are removed. The Planning Department's current internal protocol requires date stamps and project numbers in file names, but that convention is not uniformly enforced across other agencies.

Third, and most politically consequential, is who pays. San Francisco's Department of Technology budget has faced pressure in recent years as the city worked to close a multi-year general fund shortfall. Technology modernization projects have been deprioritized before; this one could be again unless department heads make a unified case to the Mayor's Office of Public Policy and Finance before the next budget cycle.

The timeline is not open-ended. Several city departments face a July 2027 deadline to comply with updated state digital records retention standards. Agencies that arrive at that deadline with bloated, poorly organized image archives will face a harder compliance process — and potentially higher costs — than those that start deduplication work now. The decisions made at Civic Center over the next six months will determine whether San Francisco gets ahead of that deadline or scrambles to meet it.