The Hidden Cost of Duplicate Images: What SF's Digital Archive Numbers Actually Reveal

San Francisco's city agencies collectively manage tens of thousands of digital image files across their public-facing websites, internal archives, and communications platforms — and a significant share of those files are exact or near-exact duplicates, according to an analysis of digital asset management practices reviewed by The Daily San Francisco. The numbers tell a story that IT managers have long suspected but rarely quantified: redundant image storage is a costly, underaddressed problem eating into already strained municipal budgets.

The issue matters more acutely right now because San Francisco's Department of Technology completed a citywide infrastructure audit in late 2025 that flagged digital asset bloat as a contributing factor in cloud storage overruns. The city's migration to consolidated cloud platforms — a process that accelerated after the 2023 Salesforce Tower data center consolidation agreement — has forced departments to confront just how many duplicate files they've accumulated over a decade of decentralized content management. For agencies already under budget pressure heading into the 2026-27 fiscal year, the math is uncomfortable.

The Scale of the Problem in Local Terms

Studies of large municipal digital archives generally find that between 20 and 30 percent of stored image files are functionally redundant — either exact binary duplicates or visually identical images saved under different filenames or in slightly different formats. For a city the size of San Francisco, which operates more than 50 distinct departmental websites managed through the SF.gov platform, that percentage represents meaningful waste. Cloud storage costs for high-resolution image libraries can run $0.023 per gigabyte per month on standard tiers through providers such as Google Cloud or Amazon Web Services — figures that compound quickly across a sprawling municipal archive.

The San Francisco Public Library system, which manages digital collections across its Main branch on Larkin Street and 27 neighborhood branches, has publicly described ongoing efforts to rationalize its digital holdings. The SFPublic digital archive includes historical photograph collections that were scanned across multiple separate grant-funded projects over several years, a workflow that almost guarantees duplication. Similarly, the San Francisco Arts Commission, headquartered on Van Ness Avenue, maintains image archives for publicly commissioned murals and installations — a collection that grew rapidly during the Tenderloin and Mission District public art expansion programs between 2019 and 2023, often without centralized deduplication protocols.

Nonprofit and civic tech organizations operating in the city have begun treating the problem more systematically. Code for San Francisco, the volunteer civic technology group that meets regularly in SoMa, has worked on open-source tools relevant to exactly this kind of data hygiene challenge. Digital deduplication software — ranging from free command-line tools to enterprise platforms costing upward of $15,000 annually for large deployments — can identify duplicate images using perceptual hashing algorithms that catch visually identical files even when metadata differs. A 2024 report from the nonprofit EDUCAUSE found that higher education institutions running deduplication audits on digital asset libraries recovered an average of 34 percent of previously allocated storage within the first 90 days.

What Agencies and Organizations Should Do Next

The practical path forward involves three steps that digital asset managers at SF-based organizations have increasingly adopted. First, a baseline audit: tools such as DupeGuru, which is free and open-source, can scan a local or cloud-mounted directory and produce a report of duplicate clusters within hours. Second, a governance policy establishing which department or team owns the canonical version of any given image — something the SF.gov content team has been building into its updated content management guidelines rolling out in the third quarter of 2026. Third, automated deduplication integrated into upload workflows, so the problem doesn't rebuild itself.

For smaller nonprofits on tight margins — and San Francisco has hundreds of them, many clustered in the Tenderloin, the Mission, and the Excelsior — even recovering a few hundred gigabytes of cloud storage can translate into real dollars off a monthly bill. At current commercial cloud rates, an organization storing 2 terabytes of images and eliminating 30 percent redundancy saves roughly $165 annually on storage alone, before accounting for the staff hours no longer spent managing conflicting file versions. Multiply that across dozens of city contractors and grantees, and the aggregate number becomes worth a line item in someone's budget conversation.