SF City Archive Pushes Duplicate Image Replacement to New Phase This Week

San Francisco's Department of Technology confirmed this week that a system-wide audit of duplicate images stored across city databases had moved into its active replacement phase, a step that city records managers say affects more than 40 city departments and touches everything from permit documentation at the Department of Building Inspection on Seventh Street to historical photographs archived at the San Francisco Public Library's main branch on Larkin Street.

The work is technical, unglamorous, and easy to overlook. But the volume of redundant digital assets sitting inside municipal servers has ballooned since 2020, when pandemic-era digitisation programs pushed staff to upload documents rapidly without consistent naming conventions or deduplication protocols. The result: storage costs climbed, file retrieval slowed, and staff at multiple agencies were, by some accounts, routinely working with outdated or incorrectly labelled image files.

What the Replacement Process Actually Involves

Duplicate image replacement, in this context, means more than deleting extras. The city's protocol — developed under the San Francisco Department of Technology's Digital Services unit — requires that each duplicate be assessed before removal, a legacy image preserved if it carries historical metadata, and a verified master file substituted in its place across all linked systems. The process follows a chain-of-custody standard that the city adopted formally in March 2025 as part of its broader Open Data and Records Modernisation Initiative.

The San Francisco Public Library system alone holds an estimated 1.2 million digitised photographs across its San Francisco History Center collections, many of which were scanned multiple times at different resolutions during separate grant-funded projects between 2018 and 2023. Library staff have been coordinating with the Department of Technology since January to identify which version of each image qualifies as the canonical master file. As of late June, that reconciliation had covered roughly 60 percent of the History Center's digital catalogue, according to the city's published project tracker on DataSF, the city's open data portal.

The Department of Building Inspection, which processes tens of thousands of permit applications annually from its offices on Seventh Street SoMa, uses image records to document site conditions, code violations, and inspection outcomes. Staff there have flagged the duplication problem as operationally significant: inspectors pulling photos from the system to prepare violation reports have occasionally retrieved outdated images from prior permit cycles rather than the most recent inspection visit. The replacement project is intended to eliminate that risk by creating a single verified record per inspection event.

Why the Timing Matters

The city chose this week to escalate the project's pace partly because the July 4 holiday weekend reduced demand on core municipal servers, giving the Department of Technology a window to run batch replacement jobs with lower risk of disrupting active services. The department had previously scheduled a similar push for spring but postponed it after a separate network infrastructure upgrade in April ran over its maintenance window.

Municipal cloud storage rates for the city's contracted services run roughly $0.023 per gigabyte per month under its current vendor agreement — figures cited in budget materials presented to the Board of Supervisors' Budget and Appropriations Committee earlier this year. Eliminating redundant image files across departments is projected to reduce stored data volume measurably, though the city has not published a final savings figure pending completion of the audit.

The Civic Innovation team at Code for San Francisco, the volunteer civic-tech brigade that meets weekly at various SoMa venues, has been tracking the project and encouraging the Department of Technology to publish its deduplication methodology as a reusable open-source tool. Other California cities — Oakland and San Jose have both cited storage-management challenges in their own IT budget documents — could potentially adapt San Francisco's framework.

For residents and businesses who regularly interact with city permitting or records systems, the practical takeaway is straightforward: any image-based document retrieved from city portals after mid-July should reflect the verified master file. Anyone who downloaded permit records or code-enforcement photographs before the replacement batch runs completes should re-pull those records to confirm they have the current version. The Department of Technology's help desk on Mayor Street in Civic Center can assist agencies or members of the public who encounter mismatched files during the transition window.