SF Officials and Tech Experts Push to Fix a Digital Duplication Crisis Costing City Agencies Thousands

San Francisco's municipal tech infrastructure is sitting on a problem nobody wanted to name out loud until recently: duplicate images — the same photograph, permit photo, or scanned document filed multiple times across city systems — are eating storage budgets, slowing public portals, and undermining data integrity at agencies that can least afford the headache. The Department of Technology and the Controller's Office have both flagged digital asset redundancy as a line-item concern in internal budget discussions this fiscal year, according to city budget documents reviewed this spring.

The timing matters. The city is in the middle of a broader push to modernize how it stores and retrieves public records, a process accelerated by the 2024 passage of San Francisco's Digital Services Reform ordinance and the ongoing overhaul of the SF.gov platform. When your core content management system is bloated with redundant files, the entire stack slows down — and in a city spending heavily to make permitting, housing applications, and transit information more accessible online, every millisecond of lag has a political cost.

What Officials Are Saying — and Not Saying

City officials have been careful not to attach dollar figures to the problem in public settings, but the conversation is happening. At a Department of Technology open forum held at City Hall in April 2026, staff raised the issue of image deduplication as part of a broader storage optimization discussion tied to the city's contract with its cloud infrastructure vendor. The SF Digital Services team, which operates out of 1 Dr. Carlton B. Goodlett Place, has been piloting content hash verification tools since late 2025 to identify redundant files before they enter the city's document management system.

Outside government, the message from the technology community is more pointed. Engineers and archivists at the Internet Archive — the nonprofit based in the Richmond District on Funston Avenue — have long argued that deduplication is not a cosmetic fix but a foundational data hygiene requirement. The Archive manages petabytes of content and has developed open-source perceptual hashing tools that some city IT consultants have cited in internal planning documents as a model worth adapting for municipal use.

The San Francisco-based open government nonprofit OpenSF, which tracks city digital services from its office in SoMa, has been pressing the Controller's Office for more transparency on how storage contracts are structured. Their concern: city departments are paying cloud storage rates on duplicate files that could be eliminated with standard deduplication software already licensed by some agencies but not consistently deployed across the enterprise.

The Numbers Behind the Clutter

Cloud storage costs are not trivial at municipal scale. Enterprise cloud contracts for mid-sized city governments have been running between $2 million and $6 million annually in recent years, depending on volume and service tier — figures that city budget analysts have used as benchmarks in planning sessions. Even a 10 to 15 percent reduction in stored volume through deduplication could yield six-figure savings annually, according to frameworks used by comparable urban IT departments in Chicago and New York.

The SF Planning Department's permit portal, which serves thousands of users weekly from the Tenderloin to the Outer Sunset, has been a particular flashpoint. Applicants frequently upload the same site photos multiple times when portal sessions time out and restart, generating redundant image files that staff must then manually review. The Planning Department has not publicly quantified the backlog, but IT staff have described the issue in internal meeting notes obtained through a public records request filed in May 2026.

For residents and small business owners trying to navigate the city's online permitting system — particularly along corridors like Valencia Street in the Mission or on Clement Street in the Inner Richmond — the practical effect shows up as slower load times and occasional file errors when uploading documentation.

The Department of Technology is expected to release updated digital infrastructure guidance later this summer as part of its fiscal year 2026–27 technology roadmap. Advocates say that document will be the clearest signal yet of whether city leadership treats deduplication as a priority fix or kicks it further down the backlog. For anyone watching how San Francisco spends its technology budget, that roadmap is worth reading carefully.