SF Officials and Tech Experts Push to Fix the City's Duplicate Image Problem — Before It Gets Worse

San Francisco's sprawling network of city databases is drowning in duplicate images, and officials from the Department of Technology to the Planning Department are scrambling to agree on a fix. The problem — millions of redundant photo files stored across incompatible servers — has quietly inflated storage costs and slowed down public-facing digital services for at least three years. Now, with an AI-driven tech boom reshaping the city's workforce and budget priorities, city hall insiders say the window for a real solution may finally be open.

The timing matters. After a brutal cycle of tech-sector layoffs, San Francisco is leaning hard into artificial intelligence as an economic engine, and city agencies are under pressure from the Mayor's Office of Innovation to modernize their back-end infrastructure. Duplicate image data is one of those unglamorous bottlenecks that rarely makes headlines but quietly drives up contract costs and frustrates the departments that have to manage it. Planning commissioners have flagged the issue in presentations about the city's permit portal, which handles hundreds of thousands of document uploads annually from neighborhoods ranging from the Tenderloin to West Portal.

What Officials and Experts Are Saying

At the San Francisco Department of Technology, officials have described the core challenge in public budget briefings: legacy storage systems purchased before 2018 were not designed with deduplication tools, meaning that every time a staff member re-uploaded a permit photo, an inspection image, or a zoning map, the server saved a fresh copy rather than recognizing an existing file. The department manages infrastructure for more than 50 city agencies, and the cumulative storage bill has become a line item that department heads no longer want to defend to the Board of Supervisors.

Digital archivists at the San Francisco Public Library's History Center on Larkin Street have been dealing with a parallel version of the problem for years. Oral history digitization projects and neighborhood photo collections — many donated by community organizations in the Mission District and Chinatown — arrived in batches that contained near-identical images. Without automated deduplication, staff had to flag duplicates by hand, a process that library technologists have described publicly as unsustainable given current staffing levels.

Tech experts outside government are pointing to off-the-shelf solutions that have already been deployed in cities like New York and Chicago. Perceptual hashing — a technique that generates a compact digital fingerprint for each image and compares it against a database of existing fingerprints — can identify near-duplicate photos that traditional file-comparison tools miss entirely. Several San Francisco-based startups in the SoMa corridor have commercialized versions of this technology for enterprise clients, and at least two have pitched their platforms directly to city procurement officers, according to public contract solicitation records posted on the city's vendor portal.

Costs, Timelines, and the Path Forward

Cloud storage costs are not trivial at municipal scale. Industry benchmarks suggest that enterprise-grade cloud storage runs between $20 and $30 per terabyte per month, and city auditors have estimated — in general terms during budget committee testimony — that the Department of Technology manages petabyte-scale data across its agencies. Even a 10 percent reduction in redundant image files would represent meaningful savings over a multi-year contract cycle.

The Board of Supervisors' Government Audit and Oversight Committee is expected to take up digital infrastructure modernization as part of the fiscal year 2026-27 budget reconciliation process, which reconvenes after the July 4 holiday weekend. City technology advocates, including staff at the nonprofit Code for San Francisco, which meets regularly at City Hall and in the Civic Center area, have argued that open-source deduplication tools could reduce the need for expensive private contracts.

For residents and businesses that interact with city permitting portals — particularly contractors working on the housing projects the mayor's office is prioritizing under its housing production emergency declaration — the practical payoff would be faster upload times and fewer error messages when submitting documentation. Anyone filing permits for projects in neighborhoods like the Outer Sunset or Bayview-Hunters Point has likely already felt the slowdowns. City officials say a request for proposals for a deduplication solution could be published by September 2026, though no formal timeline has been officially confirmed.