How San Francisco's City Agencies Got Buried Under Duplicate Images — And What It's Costing Taxpayers

San Francisco's Department of Technology has been quietly working through a backlog that city officials acknowledge stretches back nearly a decade: thousands of duplicate images embedded across municipal databases, from the Planning Department's permit portals on South Van Ness Avenue to the Digital Services team's public-records repositories at City Hall. The problem is mundane, unglamorous, and expensive.

The issue matters now because the city is in the middle of a sweeping push to consolidate its digital infrastructure under the DataSF initiative, a program launched to standardize how San Francisco agencies collect, store, and publish information. That consolidation has exposed just how bad the redundancy problem became — and how it got that way.

A Decade of Siloed Systems

The roots go back to the early 2010s, when departments across City Hall began building their own content management systems with little coordination. The San Francisco Municipal Transportation Agency ran one image library for its Muni fleet documentation. The Department of Building Inspection maintained a separate archive for site photographs tied to permits. The Office of Economic and Workforce Development kept its own promotional image bank for programs like Invest in Neighborhoods, which operates across commercial corridors in the Tenderloin, Excelsior, and Bayview-Hunters Point.

None of these systems talked to each other. Contractors uploading photos for a single project on, say, a Mission District storefront renovation might drop the same JPEG into three separate portals — one for the permit, one for the neighborhood program file, one for the city's open-data archive. Multiply that across thousands of projects over ten years and the redundancy compounds fast.

Then the pandemic hit. Between March 2020 and the end of 2021, city agencies accelerated digitization efforts under emergency timelines, pushing paper records online and converting physical inspection reports to digital formats. Staff working remotely from the Sunset District to the East Bay often lacked access to shared drives, so they uploaded what they needed locally and moved on. The speed was necessary. The long-term cost was not fully reckoned with at the time.

The Storage Bill and the Path to Reform

Cloud storage is not free. San Francisco pays for government cloud services through multi-year contracts administered by the Department of Technology, headquartered at 1 South Van Ness Avenue. While the city does not publicly itemize storage costs at the file level, government technology analysts who study municipal IT spending — including researchers at Stanford's Digital Civil Society Lab — have documented that duplicate-image bloat across mid-size city governments can inflate cloud storage expenditures by 15 to 30 percent annually compared to properly deduplicated archives.

For a city operating under a budget that the Controller's Office set at roughly $14.6 billion for fiscal year 2025-2026, even a fractional inefficiency in technology infrastructure translates to real dollars that could be redirected to transit operations or housing programs. The Board of Supervisors' Budget and Legislative Analyst has flagged digital asset management as an area warranting review in at least two prior budget cycles.

The DataSF program, which sits within the City Administrator's Office, began piloting automated deduplication tools in late 2025. The effort targets image files first — they are large, easy to identify as duplicates through hash-matching software, and scattered across the highest-traffic portals, including the SF Planning public map at 49 South Van Ness and the city's 311 service-request database.

The practical effect for residents should eventually be faster load times on permit-lookup pages, lower error rates when contractors submit applications, and a cleaner audit trail for public records requests. City technology staff are also standardizing file-naming conventions so that future uploads are tagged at the point of entry — a reform that is straightforward in theory but requires retraining staff across more than 50 departments.

For now, anyone navigating a permit application or a public records search through the city's online portals should expect intermittent slowdowns through the late summer of 2026 as the deduplication process runs in the background. The Department of Technology has said the core cleanup phase is targeted for completion before the end of the calendar year — leaving the harder work of keeping new duplicates from accumulating as the next challenge for whoever inherits the project.