San Francisco Takes a Tech-Forward Approach to Purging Duplicate Images From City Records — But How Does It Stack Up Against London and Singapore?

San Francisco's city government is sitting on a digital records problem it can no longer ignore. The Department of Technology, headquartered on Seventh Street in SoMa, has been quietly rolling out a duplicate-image detection initiative across multiple municipal departments since January 2026, targeting the hundreds of thousands of redundant photographs, scanned permits, and inspection records that have accumulated across city servers over the past two decades. The effort touches everything from Planning Department building files to SFMTA transit infrastructure documentation stored across shared drives in Civic Center.

The push matters now for two reasons. First, California's AB 473 — a 2024 state records modernization statute — set a compliance deadline of December 31, 2026 for all counties and major municipalities to audit and index their digital asset repositories. Second, the post-pandemic explosion in remote inspection workflows, which flooded city systems with smartphone-captured images often uploaded multiple times by different field crews, made the backlog acute. San Francisco is not alone in feeling the pressure, but how the city is responding sets it apart — for better and worse — from peers globally.

The city's Office of Digital Services, which operates under the broader umbrella of the Department of Technology, partnered in February with a Mission District-based civic-tech firm to deploy perceptual hashing software — a technique that identifies visually identical or near-identical images even when file names and metadata differ. The Department of Building Inspection, whose records office sits on Cesar Chavez Street, has been the first large agency to run a full deduplication pass, working through an estimated 1.2 million image files covering permit documentation dating to 2005. Program managers have described the effort internally as a prerequisite for the city's broader Digital Equity and Open Data obligations under the Mayor's Office of Housing and Community Development compliance framework.

How San Francisco Compares to London and Singapore

London's Government Digital Service began a comparable program in 2023 through its Crown Commercial Service procurement framework, giving borough councils access to centrally licensed deduplication tools. By mid-2025, the Greater London Authority reported clearing more than 4 million redundant files from shared Planning and Transport for London repositories — a scale San Francisco has not yet approached. Singapore's Government Technology Agency, GovTech, went further still, embedding duplicate-detection checkpoints directly into upload workflows across 23 ministries by 2024, meaning redundant images are flagged before they enter storage rather than cleaned up retroactively. Both cities benefited from more centralized IT governance structures than San Francisco, where departments historically managed their own systems with limited coordination.

San Francisco's federated department structure is the acknowledged complication. The Planning Department, the Department of Public Works, and SFMTA each maintain separate content management systems with different metadata standards, which means a photograph of, say, a cracked sidewalk on Valencia Street in the Mission might exist in three separate databases under three different file names. Unifying those records requires not just deduplication software but negotiation between agencies that have operated independently for years. The cost of the current initiative — funded through a $2.3 million allocation in the city's Fiscal Year 2025-26 technology budget — is modest compared to London's centralized spend but represents a meaningful commitment given competing demands on city IT resources.

What Comes Next for City Agencies and Residents

The Department of Technology has set an internal target of completing the Building Inspection deduplication pass by September 2026, which would give the city roughly three months to extend the process to Planning and Public Works records before the state deadline. If the rollout holds to schedule, the city expects to recover several terabytes of server capacity currently costing roughly $180,000 annually in cloud storage fees — savings that program managers plan to redirect toward expanding the open-data portal maintained through DataSF, the city's public data platform.

For residents and developers who regularly pull permit records from the San Francisco Open Data Portal, the practical effect should be cleaner, faster searches. Duplicate image files have been a known irritant for anyone using the portal's building permit database, where searching a Tenderloin address can return the same inspection photograph a dozen times. Contractors who work with the Planning Department's SFPlanning online portal have filed repeated feedback requests about the issue over the past three years. Whether the deduplication effort reaches its December target depends heavily on cooperation between agencies that have not always moved in lockstep — a challenge London and Singapore solved through top-down mandates that San Francisco, with its strong tradition of departmental autonomy, has so far chosen not to replicate.