How San Francisco's Digital Records Ended Up Full of Duplicates — and What It's Costing the City

San Francisco's Department of Technology confirmed this spring that a citywide audit of public-facing digital assets — property records, planning documents, permit images, and departmental photo libraries — had identified tens of thousands of duplicate image files scattered across at least a dozen separate servers. The problem, officials acknowledged, did not happen overnight.

The duplication crisis traces back to decisions made during three distinct eras: the mid-2000s push to digitize paper records, the 2012-to-2016 wave of departmental web redesigns, and the emergency remote-work scramble of 2020. Each wave produced its own siloed storage system, and none of them talked cleanly to the others. By the time a unified digital infrastructure review began in late 2024, the redundancy had compounded into a storage and maintenance liability.

A Problem Built Brick by Brick Across City Hall

The San Francisco Planning Department's permit portal, accessible through the city's SF311 service gateway, became one of the clearest examples of how duplication accumulated. When the department migrated its paper-based permit archives to digital formats between 2007 and 2011, scanning contractors uploaded images to a legacy server. When the Planning Department then relaunched its public-facing website in 2014, staff uploaded the same images again to a new content management system. Neither migration included a deduplication step. A third round of uploads occurred when the department shifted to a cloud-based platform during the COVID-19 shutdown period in 2020, again without a reconciliation pass against existing files.

The Recreation and Parks Department ran into a parallel version of the same problem. Its archive of park survey photos — images documenting conditions at sites from the Panhandle in the Western Addition to McLaren Park in the Excelsior — ended up mirrored across three separate storage environments: an on-premises server at the department's McLaren Lodge headquarters, a shared city cloud drive, and a vendor-managed content delivery system brought in during a 2019 website contract.

The Department of Technology's review, which began formally in October 2024, found that redundant file storage was consuming an estimated 34 terabytes of avoidable capacity across city systems, according to a summary presented to the city's Committee on Information Technology. At commercial cloud storage rates, that volume represents a recurring annual cost that the department characterized as a six-figure line item — money that could otherwise fund frontline services.

Why the Cleanup Is Harder Than It Sounds

Deleting duplicate images is not as simple as running a batch script. City archivists and IT staff must verify that a file marked as a duplicate is genuinely redundant — not a slightly different scan or a version carrying different metadata that legal or compliance teams might need. The San Francisco City Attorney's Office has flagged that certain permit and inspection images carry evidentiary status and cannot be purged without sign-off from department heads and, in some cases, the Board of Supervisors.

The Department of Technology is now piloting a duplicate-detection tool developed through a contract with a vendor selected under the city's Office of Contract Administration procurement process. The pilot launched in April 2026 and is currently focused on Planning Department records only. A broader rollout to Rec and Parks, the Department of Building Inspection, and the Public Works photo archive is scheduled for the first quarter of 2027, pending budget approval during the current fiscal cycle.

For residents who interact with city systems — checking permit histories on Parcel Map, submitting service requests through SF311, or reviewing environmental review documents through the Planning Department's online portal — the practical effect of the cleanup should be faster page load times and more reliable search results within city databases. The department has said it will publish progress updates on DataSF, the city's open data platform, beginning in August. Anyone tracking a specific permit or project image who notices a broken link during the transition period can report it directly through the SF311 web portal or by calling 311.