SF's Digital Records Overhaul Hits a Snag: Duplicate Images Are Clogging City Databases, Officials Say

San Francisco's push to digitize decades of paper records has produced an unexpected headache: thousands of duplicate images buried inside city databases, inflating storage costs and frustrating departments trying to modernize public access to government files. The problem has drawn pointed commentary from technology officers, archivists, and open-government advocates over the past several months, with some describing the situation as a foreseeable consequence of rushed digitization without adequate quality controls.

The issue cuts across multiple agencies. The San Francisco Planning Department, which maintains permit records going back well into the twentieth century, began an accelerated scanning program in 2023 to clear a backlog of physical documents at its offices on Rahaul Vallejo Street in SoMa. The SF Digital Services division, housed under the Office of the City Administrator at City Hall, has separately been consolidating legacy databases as part of its broader "DataSF" modernization effort. Both initiatives, sources familiar with the programs say, fed duplicate images into shared repositories when automated ingestion systems failed to flag files that had already been uploaded.

What Officials and Experts Are Saying

The City Administrator's Office has acknowledged the redundancy problem exists but has not yet released a comprehensive audit of its scope. Digital preservation specialists who work with municipal governments nationally say the San Francisco situation is not unusual for cities that scaled up scanning programs quickly—particularly after COVID-19 court and office closures created a sudden backlog of documents needing rapid digitization in 2020 and 2021.

Advocates tied to the San Francisco chapter of the Electronic Frontier Foundation have raised the issue in the context of public records requests, noting that duplicate file structures can obscure which version of a document is authoritative. When a building permit image exists in three slightly different scanned forms inside the Planning Department's system, it complicates responses to California Public Records Act requests, which carry a ten-day response deadline under state law. The EFF has not filed formal complaints related to this specific issue, but representatives have flagged it in public-facing materials about government transparency in the Bay Area.

SFMTA's technology division has been grappling with a related version of the problem. The agency manages surveillance and transit camera footage across the Muni Metro system, including footage from underground stations at Civic Center, Powell Street, and Van Ness. Retention policies require certain footage to be stored for defined periods, but duplicate uploads from camera systems that sync footage redundantly have added unnecessary volume to storage contracts. SFMTA's approved budget for fiscal year 2025–2026 runs to approximately $1.4 billion, and while storage costs represent a small line item, technology officers across the agency have described the redundancy as an operational nuisance compounding other IT transition pressures.

Pressure to Act Before the Next Budget Cycle

The Board of Supervisors' Government Audit and Oversight Committee is expected to take up broader IT efficiency questions later this summer, with a hearing tentatively scheduled for August. Committee members have signaled interest in understanding how much duplicate data storage is costing the city annually, a figure the Controller's Office has not yet published in any public document.

Private-sector technologists in the South of Market and Mission Bay corridors—where many of San Francisco's AI and data infrastructure firms are based—say automated deduplication tools have become standard practice in enterprise environments. Several have argued in op-eds and public panels that city governments lag years behind in adopting these tools, partly because procurement rules make it difficult to move quickly and partly because legacy systems were not built with interoperability in mind.

For residents trying to pull historical permit data through the city's Accela permit portal or access archived Muni planning documents through DataSF's public interface at data.sfgov.org, the practical effect is confusion: duplicate images sometimes surface with conflicting metadata, making it hard to determine which scan is the correct or most recent version. The City Administrator's office says it is reviewing options for an automated deduplication pass across shared repositories, with a timeline expected to be announced before the end of calendar year 2026.