San Francisco Leads a Quiet Digital Cleanup — But Other Cities Are Catching Up Fast

San Francisco's Department of Technology quietly hit a milestone this spring: its citywide digital asset repository, used by agencies from the Planning Department to SF Public Works, had logged more than 340,000 flagged duplicate images by April 2026, the result of a two-year deduplication push that began under the city's Digital Equity and Infrastructure initiative. The effort is unglamorous work — the municipal equivalent of cleaning out a storage unit — but cities from Amsterdam to Seoul are watching closely as they face the same sprawling, redundant image libraries that slow permit processing, inflate storage costs, and create legal headaches around licensing.

The stakes are higher than they might appear. San Francisco's Planning Department alone handles tens of thousands of permit applications a year for properties across neighborhoods from the Sunset District to SoMa, each generating multiple site photos, scanned documents, and inspection images. When duplicates pile up across disconnected systems — some running software that dates to the early 2010s — staff end up working from conflicting records. In a city where housing production is already under an emergency declaration, any friction in the permitting pipeline has real consequences for whether a building gets approved in weeks or months.

What San Francisco Is Doing Differently

The city's approach centers on two programs. The first, managed by the San Francisco Department of Technology in partnership with the Office of Digital Services on 1 Dr. Carlton B. Goodlett Place, uses machine-learning tools to identify near-duplicate images — not just exact copies — across agency servers. The second feeds confirmed duplicates into a centralized archive managed by SF Environment's data team, rather than simply deleting them, preserving an audit trail that attorneys and planners can reference if a property dispute goes to litigation.

That combination — automated detection plus a preserved archive — sets San Francisco apart from the approach taken by, say, Chicago, which in 2024 contracted with a private vendor to run a hard-delete deduplication pass across its city systems. Chicago's method was faster and cheaper upfront, but the city's Law Department subsequently flagged gaps in records tied to at least three zoning appeals, according to reporting by the Chicago Tribune last year. London's Government Digital Service has taken a middle path, setting retention windows of 18 months before duplicates are permanently purged from the Greater London Authority's asset management platform.

New York City's approach is the most decentralized of any major American municipality. The five boroughs each manage their own property image databases with minimal cross-agency deduplication, a structure that advocates at the Urban Tech Hub in Brooklyn have publicly criticized as inefficient. San Francisco, with its comparatively unified city IT structure, has an organizational advantage that genuinely matters here.

The Cost Question — and What Comes Next

Storage is not free. San Francisco's Department of Technology reported in its fiscal year 2025-26 budget documentation that city agencies collectively spent approximately $4.2 million annually on cloud and on-premise storage, a figure that department officials projected could fall by 12 to 18 percent once deduplication targets are met across all participating agencies. That projection remains unverified externally, but comparable figures from Amsterdam's municipal digitization program — which wrapped a similar project in late 2024 — suggest double-digit savings are achievable. Amsterdam reported a 14 percent reduction in storage expenditure within 18 months of completing its deduplication rollout across city housing and planning systems.

Seoul's Smart City operations center, which manages image data for a metro area of roughly 10 million people, completed a full deduplication audit in 2023 and cut its municipal image library from 11.4 million files to just under 7 million — a reduction of more than 38 percent — without losing a single record flagged as legally significant.

For San Francisco residents and businesses navigating the Planning Department or dealing with property permit searches at the Permit Center on 49 South Van Ness Avenue, the practical payoff is faster document retrieval and fewer cases where an inspector or planner pulls the wrong version of a site photo. The city's Digital Services office says it expects all major participating agencies to reach deduplication compliance by the end of calendar year 2026. Whether that deadline holds will depend heavily on whether the remaining agencies — including SF Recreation and Parks, which has been slower to onboard — can complete data mapping before the fiscal year closes in June 2027.