SF City Agencies Race to Fix Duplicate Image Problem Plaguing Digital Records This Week

San Francisco's Department of Technology confirmed this week that a duplicate image replacement effort—targeting thousands of redundant photo and document scans stored across city servers—has moved into an active remediation phase, with affected systems spanning everything from Planning Department permit files to public-facing portals maintained by the Office of Digital Services on Seventh Street.

The timing matters. The city is deep into a broader push to modernize its data infrastructure before a hard deadline tied to the San Francisco Digital Equity Initiative, a program that received renewed funding allocation in the spring 2026 budget cycle. Redundant image files have been clogging shared storage, slowing database queries for frontline workers, and producing inconsistent results when residents pull records through SF.gov—the city's consolidated service portal that launched its latest iteration in early 2025.

What Went Wrong, and Where It Showed Up

The problem surfaced most visibly in two places: the San Francisco Planning Department's online permit tracker, where scanned building diagrams were appearing in duplicate or triplicate across search results, and the San Francisco Public Library's digital archive at the Main Branch on Larkin Street, where staff flagged that hundreds of historical photograph entries had been indexed multiple times after a system migration last fall. Neither issue threatened public safety, but both drew complaints from contractors, researchers, and community advocates who rely on accurate record retrieval.

City IT staff identified the root cause as an automated ingestion script that failed to run deduplication checks when files were transferred from legacy servers during the Department of Technology's 2025 infrastructure consolidation. That migration, which ran from August through November 2025, moved more than 4 terabytes of records into a new cloud environment managed under the city's contract with a third-party vendor. According to a department briefing document circulated to the Board of Supervisors in June, roughly 11 percent of image files transferred during that window were assigned duplicate identifiers—creating downstream confusion in any system that pulled from the consolidated storage layer.

The Tenderloin-based nonprofit OpenSF Data Collective, which monitors public records accessibility for low-income residents navigating city services, noted the issue publicly in May when it published a report on SF.gov search reliability. The group documented cases where residents searching for affordable housing application forms were retrieving outdated scanned versions of documents rather than current PDFs, partly because of the identifier conflicts created by duplicate image records.

The Fix, and What Comes Next

The Department of Technology began deploying an automated replacement script on July 1, targeting the highest-traffic databases first. The Planning Department's permit portal is expected to be cleared of duplicate entries by July 11, according to the department's published remediation timeline posted to the city's DataSF transparency page. The library's digital archive cleanup is scheduled to follow in the week of July 14.

The full remediation across all affected systems is projected to take until mid-August. That window includes manual review of approximately 3,200 flagged records where the automated script cannot confidently determine which version of a duplicated image is canonical—those cases will go to departmental staff for individual sign-off.

Practically speaking, residents who have run into dead links or garbled search results on SF.gov permit pages or on the library's digital catalog at sfpl.org should try those searches again after July 11. Anyone who submitted a records request to the Planning Department since January 2026 and received a response citing unclear or incomplete scanned attachments can contact the department's counter at 1650 Mission Street to request a reprocessed version of their documents at no additional cost.

The episode has also added fuel to an ongoing debate at City Hall about how aggressively San Francisco should invest in preventive data governance—rather than reactive cleanup—as more city services migrate to digital-first delivery. A proposal before the Budget and Finance Committee would allocate $2.1 million in the next fiscal year for a dedicated data quality office within the Department of Technology. That proposal is expected to come to a vote before the August recess.