How San Francisco's City Agencies Spent Years Drowning in Duplicate Images — and What Finally Forced a Fix

San Francisco's municipal agencies collectively store hundreds of thousands of redundant digital images across their internal servers — duplicate photographs, scanned permit documents, and duplicated infrastructure records that have quietly consumed storage budgets and slowed public records requests for the better part of a decade. A directive issued this spring by the city's Department of Technology is now pressing departments to audit and consolidate those files by the end of fiscal year 2027.

The problem didn't appear overnight. It grew from a patchwork of decisions made between roughly 2014 and 2022, when city departments scaled up digital record-keeping at different speeds, with different software platforms and almost no coordination on file-naming conventions. The Planning Department on Van Ness Avenue was running one document management system. The Department of Public Works, headquartered on South Van Ness, used another. The Municipal Transportation Agency had its own. None of them talked to each other automatically, and duplicates multiplied every time a file crossed departmental lines — which, in a city bureaucracy, happens constantly.

The Tech Boom Made It Worse Before Anyone Noticed

Cloud storage got cheap fast during San Francisco's tech boom years. Between 2015 and 2020, the cost of storing a gigabyte of data on enterprise cloud platforms dropped so sharply that agencies stopped scrutinizing what they were actually holding. There was little incentive to deduplicate when the bill was low and the political pressure was elsewhere — focused on homelessness on Market Street, fentanyl response in the Tenderloin, or housing production numbers downtown. Digital housekeeping was nobody's campaign issue.

The reckoning came gradually. By 2023, the Department of Technology's internal audits found that some agencies were storing three or four copies of the same permit-application image, sometimes across both on-premises servers at City Hall and contracted cloud infrastructure. A citywide digital storage budget that had held relatively flat began climbing. The MTA's digital archive, which includes decades of infrastructure photography from Muni subway tunnels and overhead wire inspections, was flagged specifically as a high-duplication environment in a 2024 internal review, according to documents described in a budget committee staff report from that year.

The San Francisco Public Library's digital collections team at the Civic Center branch had actually solved a version of this problem earlier than most. Librarians working on the San Francisco Historical Photograph Collection developed a de-duplication workflow around 2019 that used perceptual hashing — a method that identifies visually identical or near-identical images even when file names differ — to clean up thousands of scanned archival prints. That internal model is now cited by the Department of Technology as a template other agencies could follow.

What the New Directive Requires

The spring 2026 directive doesn't mandate a single software solution. Instead, it requires every department with more than 50,000 image files in city storage to complete a duplication audit by March 31, 2027, and submit a remediation plan within 90 days after that. Departments that contract with vendors for document management — several do, through agreements administered out of the Civic Center complex — must include vendor cooperation clauses in any renewals after January 1, 2027.

For residents, the practical impact is mostly invisible but real. Public records requests that involve image files — building permits, code enforcement photos, infrastructure inspection records — have historically taken longer to fulfill in part because staff had to manually locate the authoritative version of a file among duplicates. Faster retrieval is one of the stated goals of the cleanup effort, alongside storage cost reduction.

The city's Housing Accelerator Office, stood up in 2023 to speed permit approvals for residential construction, has already begun its own image-file audit as part of a broader workflow overhaul. That office processes thousands of permit-application photo packages annually and flagged duplication as a bottleneck in internal documentation from late 2025.

For departments that haven't started yet, the clock is running. The Department of Technology has scheduled a technical assistance session for August 2026 at its offices on 1 South Van Ness Avenue, open to staff from any agency still mapping their file inventories. The fiscal year 2027 deadline is firm, and the storage savings are expected to help offset costs elsewhere in a capital budget that has little slack.