SF City Hall's Digital Archive Has a Duplicate Image Problem — And the Numbers Are Staggering

San Francisco's Department of Technology is sitting on a digital storage crisis measured not in gigabytes but in wasted public dollars. An internal audit of the city's municipal image repositories — covering everything from Planning Department permit records to SFMTA street-condition photo logs — found that duplicate image files account for roughly 34 percent of total stored assets across the city's shared servers, according to a departmental review circulated to agency heads this spring.

That figure matters because San Francisco pays for cloud and on-premises storage through multi-year vendor contracts that scale directly with data volume. When nearly one in three image files is a functional copy of another, the city is not just burning disk space — it is paying for it, and has been for years.

Where the Redundancy Lives

The worst offenders, per the audit summary, are high-volume intake systems: the Planning Department's SFPlanning portal on Kearny Street, which processes thousands of permit application photo packets annually, and the Department of Building Inspection's online submission system, which serves contractors and homeowners across every neighborhood from the Sunset to Bayview-Hunters Point. Both platforms allow applicants to upload image files without any automated de-duplication check at the point of intake. The result is a compounding pile of identical or near-identical JPEGs and PDFs that migrate into archival storage and stay there indefinitely.

The city's 311 service request system — which logs pothole, graffiti, and encampment complaints with attached photos from residents — contributes a separate and substantial layer. A single block of, say, Cesar Chavez Street with visible damage can generate dozens of 311 reports from different callers, each uploading the same or nearly identical image from a slightly different angle. Those files feed directly into the city's Salesforce-based case management infrastructure and are rarely pruned.

The Moscone Center-area fiber network hub that routes data between City Hall and satellite offices on Van Ness Avenue handles much of this traffic. Technologists who work inside the city's IT ecosystem say the storage sprawl predates the current administration and has grown steadily since the 2015 push to digitize legacy paper records.

The Dollar Count Is Getting Harder to Ignore

The city's Department of Technology allocated approximately $47 million to enterprise IT infrastructure in the fiscal year 2025–26 budget, a figure that includes storage contracts with vendors including Microsoft Azure and on-premises hardware maintenance at the city's data center on Turk Street near the Civic Center. The audit does not break out a precise line-item cost for redundant image storage specifically, but analysts working from the 34 percent redundancy figure and the proportional storage budget put the unnecessary expenditure somewhere between $800,000 and $1.4 million annually — real money in a budget cycle where Mayor Daniel Lurie's administration has been combing every department for savings to close a projected shortfall.

Automated de-duplication tools — software that computes cryptographic hashes of image files and flags or removes identical copies — are commercially available and widely deployed in private-sector environments. The San Francisco Public Library system piloted one such tool in its digital collections unit on Larkin Street in 2024, reducing its own image archive by roughly 18 percent in a six-month trial. The library's experience is now being cited internally as the closest local proof-of-concept for a city-wide rollout.

The Department of Technology is expected to bring a formal proposal for a de-duplication protocol to the city's Committee on Information Technology before the end of the third quarter of 2026. If approved, implementation would likely begin with the Planning and Building Inspection systems — the highest-volume intake points — before extending to 311 and other city platforms. Residents and contractors who regularly submit photo documentation through city portals would see no change to the front-end upload experience; the de-duplication process would run silently on the back end after submission. The practical upside for the public: faster search and retrieval in public records requests, which currently bog down in part because queries have to crawl through redundant file masses. For a city government that has made digital transparency a stated priority, cleaning up the numbers is the first step toward actually delivering on it.