SF City Agencies Push to Eliminate Duplicate Images From Public Records Databases This Week

San Francisco's Department of Technology moved this week to accelerate a citywide audit of duplicate images embedded in public records databases, a housekeeping effort that has taken on new urgency as storage costs and data integrity concerns mount across multiple agencies. The push, coordinated with the City Administrator's Office at 1 Dr. Carlton B. Goodlett Place, targets redundant image files that have accumulated over years of overlapping document uploads across departments including the Planning Department, the Department of Building Inspection, and the San Francisco Public Library's digital archive program.

The timing matters. San Francisco's broader digital infrastructure overhaul — tied to a multi-year technology modernization bond measure — has forced every major city department to account for storage overhead. Duplicate image files, which can multiply rapidly when permit applications, environmental review packages, and community meeting presentations are uploaded through multiple portals, are now being flagged as a measurable drag on retrieval speeds and annual cloud storage budgets. City IT staff have described the problem internally as systemic rather than incidental, according to agenda documents from the department's June 30 operations meeting, which are publicly available on the city's records portal.

What the Cleanup Involves

The audit covers image assets held across the city's Accela permitting platform, the SF Planning Department's online map and project viewer on Mission Street, and the digitized collections maintained by the San Francisco History Center inside the Main Library on Larkin Street in Civic Center. Technicians are using automated deduplication tools to flag files that share identical hash values — a standard method for identifying exact copies — before a manual review stage determines whether any flagged images should be retained for archival or legal reasons.

The Department of Building Inspection alone manages tens of thousands of permit-related image attachments filed each year for properties across neighborhoods from the Outer Sunset to the Bayview. Officials have not yet released a final count of confirmed duplicates, but the scope of the audit covers records dating back to 2014, when the city first began migrating paper-based permit files to its current digital systems. That twelve-year window represents a significant volume of accumulated data, and city IT staff have indicated the deduplication process is expected to run through at least the end of August.

The SF Public Library's digital preservation program, which has been digitizing historical photographs of San Francisco's neighborhoods and infrastructure since 2018, is also part of the review. The History Center's collection includes images of landmarks from the Ferry Building to Alamo Square, and librarians have flagged that donor-submitted scans sometimes duplicate images already held in the permanent collection. A staff report circulated in late June noted the library had identified more than 800 potential duplicate image records in one collection category alone, though that figure has not been independently verified by The Daily San Francisco.

Why This Week's Push Matters for Residents

For San Franciscans navigating the city's planning and permitting systems — a process that has drawn sustained criticism for delays and data gaps — the practical benefit of cleaner image records is real. Permit reviewers at 49 South Van Ness Avenue, the city's main permitting hub, rely on uploaded site photographs and architectural drawings to process applications. Duplicate or mislabeled images can send staff back to applicants for resubmission, adding days or weeks to timelines that already stretch across months for many project types.

The deduplication work also feeds into a longer-term city goal of building a unified asset management system that can be accessed by multiple departments without redundancy. That project, which city technology planners have tied to state digital infrastructure grant programs, is scheduled for a phased rollout beginning in the first quarter of 2027.

Residents who have submitted images through public-facing portals — including neighborhood plan feedback surveys and 311 service requests — are not expected to be affected by the audit. The city has said personal submissions are stored in separate systems. Anyone with questions about specific permits or archival records can contact the Department of Technology's public records liaison at its offices on Seventh Street in SoMa or through the city's online service portal.