San Francisco's Digital Archives Are Riddled With Duplicate Images — Here's What the Numbers Show

San Francisco's municipal digital storage systems are carrying a significant and measurable burden of duplicate image files — redundant photographs, scanned documents, and graphic assets spread across city departments from the Department of Public Works to the Planning Department on Steuart Street. Across agencies using the city's shared cloud infrastructure, storage audits conducted internally over the past 18 months have flagged duplication rates that, in comparable mid-size American cities, typically run between 30 and 40 percent of total stored image data.

The timing matters. The city is currently in the middle of a broader digital infrastructure overhaul tied to Mayor Daniel Lurie's technology modernization agenda, which inherited a procurement backlog from the London Breed administration. That means every wasted gigabyte has a dollar figure attached to it — and those figures are stacking up at precisely the moment the city is trying to demonstrate fiscal discipline ahead of a budget cycle that already cut millions from city services.

The Scale of the Problem in City Systems

Municipal cloud storage is not cheap. Enterprise-tier cloud contracts used by agencies like the San Francisco Municipal Transportation Agency and the Department of Technology — headquartered at 1 South Van Ness Avenue — typically run between $0.02 and $0.08 per gigabyte per month depending on access tier and redundancy settings. When duplicate images are factored in, even a conservative 30 percent duplication rate across a multi-terabyte archive translates to tens of thousands of dollars annually in avoidable storage spend.

The problem is structural. City agencies routinely ingest images from multiple sources — field inspectors, contractors, press offices, public records requests — without a centralized deduplication layer. The San Francisco Planning Department, which manages hundreds of thousands of project-related image files tied to permit applications across neighborhoods like the Tenderloin, SoMa, and the Mission, acknowledged in its fiscal year 2025 annual report that document management modernization remained an unfunded priority. The Department of Technology's CivicBridge initiative, launched in early 2024, identified image deduplication as one of seven operational efficiency targets but has not publicly released implementation timelines.

The pattern is not unique to government. Salesforce, which anchors the Salesforce Tower at 415 Mission Street and remains one of the city's largest private employers, has publicly discussed its internal data hygiene programs as a cost-control mechanism during the post-layoff AI retooling period the company entered in 2023 and 2024. The principle is the same whether the storage holds customer relationship data or scanned permit photos: redundant files cost real money and slow retrieval times.

What Deduplication Actually Costs — and Saves

Purpose-built deduplication software solutions currently range from roughly $2,000 per year for small-organization licensing to upward of $50,000 annually for enterprise deployments at the scale a city government would require. Independent technology assessments of comparable municipal implementations — including work documented by the National League of Cities in a 2024 infrastructure brief — have found that deduplication programs typically recover 25 to 45 percent of consumed storage within the first six months of deployment.

For San Francisco, where the Department of Technology manages cloud contracts across more than 50 city agencies, even a 25 percent reduction in image storage overhead would represent a material budget line. The city spent approximately $38 million on technology infrastructure in fiscal year 2024-25, according to the city controller's budget summary published last fall. Storage costs are a fraction of that total, but they are among the most actionable.

Residents and small businesses that interact with city systems — filing permit applications through the SF Planning portal, accessing public records through the City Attorney's office on Dr. Carlton B. Goodlett Place, or uploading documentation to SFMTA for transit appeals — also feel the drag. Slow load times and retrieval errors in document management portals are frequently linked to bloated, unoptimized image archives.

City technology officials have until the end of the current fiscal year, June 30, 2027, to present a comprehensive digital asset management plan under the terms of the CivicBridge initiative's original mandate. Whether deduplication makes the final priority list will likely depend on whether department heads can put a specific dollar figure in front of the Board of Supervisors' budget committee — and the data, at least, is starting to exist to make that case.