SF's Digital Records Crisis: What Officials and Experts Say About the City's Duplicate Image Problem

San Francisco's municipal agencies are sitting on digital storage systems so clogged with duplicate images and redundant files that the city's Department of Technology flagged the problem in its fiscal year 2025–26 infrastructure review as a meaningful driver of rising cloud storage costs. Officials and digital records specialists say the issue is no longer a back-office nuisance — it is slowing permit processing at the Planning Department on Raoul Wallenberg Place and creating search headaches inside the San Francisco Public Library's Digital Collections unit at the main branch on Larkin Street.

The timing matters because City Hall is simultaneously pushing a housing production emergency agenda that depends on faster permit workflows. Every drag on Planning Department systems — including bloated image repositories full of duplicate site photographs and scanned drawings — adds friction to a process that Mayor Daniel Lurie has publicly identified as a bottleneck. Digital archivists and records managers say duplicate image accumulation is one of the least glamorous but most persistent causes of that friction.

What the Specialists Are Saying

Records management professionals who work with Bay Area public agencies describe a pattern that is almost universal in large municipal systems: staff photograph the same site multiple times across multiple permit applications, scans of paper drawings get uploaded in triplicate by different reviewers, and no automated deduplication tool flags the redundancy before storage costs compound. The San Francisco Planning Department processes thousands of permit applications annually, and each application can generate dozens of image attachments. Without a systematic culling protocol, those files accumulate in cloud storage environments billed by the gigabyte.

Digital preservation specialists at the San Francisco Public Library have been grappling with a parallel version of the problem in the library's historical photograph collection, which includes tens of thousands of images sourced from donors, city agencies, and digitization projects stretching back to the early 2000s. Deduplication is considered standard practice in library science, but resource constraints — the library system's current operating budget is set against citywide fiscal pressure — have limited how aggressively staff can run deduplication workflows across the full archive.

Technology consultants who advise Bay Area governments say off-the-shelf deduplication software has become significantly more capable since 2023, with AI-assisted tools now able to flag near-duplicate images — same scene, slightly different angle or compression — rather than only exact byte-for-byte copies. The practical result, according to firms that have run pilots with California counties, is storage reduction of between 20 and 40 percent in image-heavy repositories, translating to measurable annual savings on cloud licensing contracts.

City Programs and Next Steps

The Department of Technology's current contract with its primary cloud provider runs through June 2027, giving the city a roughly 12-month window to demonstrate storage efficiencies before the next renewal negotiation. Digital records consultants say that window is also an opportunity to embed deduplication standards into the city's broader data governance framework — a framework that, as of this year, is still being revised following the retirement of the city's previous Chief Data Officer position.

At the neighborhood level, the practical stakes are visible at the San Francisco Planning Department's public counter at 49 South Van Ness Avenue, where permit applicants and architects wait for document reviews. Staff there say digital retrieval speed matters when reviewers need to pull prior permit photographs for comparison. A repository cleaned of duplicates retrieves files faster and costs less to maintain — basic arithmetic that technology officials say is straightforward to act on once political and budget priority aligns.

For city residents and business owners filing permits along corridors like Fillmore Street or in the Dogpatch industrial zone, the immediate practical advice from records specialists is consistent: submit image files in standardized formats, avoid resubmitting previously uploaded documents, and confirm with Planning staff which file naming conventions reduce the chance of unintentional duplication on the department's end. Longer term, watchdog groups including the San Francisco Civil Grand Jury have previously flagged digital records management as an area deserving closer oversight — and officials say a formal audit recommendation on storage practices could come as early as the fall 2026 session.