SF Agencies Remove Thousands of Duplicate Images From Public Records

San Francisco's Department of Technology moved this week to expand its duplicate image replacement protocol across at least four city agencies, affecting thousands of digital records stored in the Civic Center data infrastructure. The push, which picked up speed after a backlog audit completed in late June, targets redundant and degraded image files embedded in public-facing databases — everything from permit records at the Department of Building Inspection to case files managed through the city's 311 system.

The timing matters. San Francisco is mid-way through a broader digital modernization effort tied to its two-year IT reform roadmap, and unresolved data redundancy has been flagged by the city's Budget and Legislative Analyst office as a friction point in efforts to streamline services. Duplicate image files consume storage, slow retrieval times for city staff, and in several documented cases have caused version-control errors in publicly searchable permit histories — an issue that directly affects contractors, attorneys, and residents who rely on those records at the Department of Building Inspection's offices on Mission Street.

What the Cleanup Actually Involves

The process is less glamorous than it sounds. City IT staff and contracted technicians are running automated deduplication scripts against image repositories, then flagging uncertain matches for human review before any file is permanently replaced or archived. The San Francisco Public Library's digital collections team, based at the main branch on Larkin Street in Civic Center, has been running a parallel but separate version of this protocol on its historical photograph archive since early 2025 — and that project has offered some practical lessons for the broader city effort.

The Library's archive work surfaced a specific challenge: automated tools misidentify visually similar but legally distinct images at a measurable rate. The library's internal documentation, shared with the Department of Technology in May, noted that manual review catch rates matter significantly when the underlying records carry legal or historical weight. Building permit images, for instance, can determine whether a property modification was approved — replacing the wrong version of a scanned document has real consequences for homeowners in neighborhoods like the Sunset and the Excelsior, where older housing stock means older and more complicated permit trails.

The 311 system, which processes tens of thousands of service requests each month across San Francisco, also stores photographic evidence submitted by residents — cracked sidewalks, illegal dumping, encampments. Those images feed into case resolution workflows managed by Public Works and other departments. Duplicate entries in that pipeline have caused cases to be logged twice, distorting the city's own performance metrics on response times.

What Comes Next for City Records and the Public

The Department of Technology has set an internal target of completing the first-phase deduplication review across the four pilot agencies before the end of the third quarter — meaning September 30. A second phase, which would extend the protocol to the Planning Department's voluminous permit and environmental review image library, is penciled in for early 2027.

For residents and professionals who regularly pull city records, the practical upshot is that some permit histories may temporarily show gaps or display placeholder images during the replacement window. The Department of Building Inspection's public portal has carried a notice since July 1 advising users that certain image records may be unavailable or under review.

Archivists and open-government advocates have raised concerns about audit trails — specifically, whether replaced image files will be logged in a publicly accessible change record rather than silently overwritten. That question is not yet resolved. The San Francisco Sunshine Ordinance Task Force is scheduled to take it up at its next regular meeting, which falls later this month.

Anyone who relies on city image records for active litigation or permit applications should download local copies now and flag any discrepancies directly to the relevant agency before September 30. The Department of Technology's service desk can be reached through SF.gov.