'My Family's History Is Just Gone': SF Residents Speak Out on Duplicate Image Purges Erasing Community Archives

An automated image-deduplication process running inside at least two San Francisco city-contracted digital archive platforms has deleted thousands of photographs submitted by residents and community organizations, according to people who have spent recent weeks trying to recover the files. The deletions are not the result of a single incident — they are the cumulative output of software designed to flag and remove visually similar images, a process that community members say was never clearly disclosed when they uploaded their photos.

The issue has surfaced as San Francisco's Department of Technology continues a broader digitization push tied to the city's Open Data initiative and the SFGovTV community media archive program. Several neighborhood groups that submitted photographs to those platforms in 2024 and 2025 say they noticed gaps in their collections earlier this year, and only recently pieced together that a deduplication filter was the cause.

What Communities Are Losing

The losses hit hardest among groups that had already survived physical displacement. At the Filipino Community Center on Sutter Street in the Tenderloin, volunteers spent much of 2024 uploading scanned photographs from the 1970s and 1980s documenting the neighborhood's SoMa Manilatown corridor — the same blocks largely erased by Yerba Buena redevelopment decades ago. Community members say several dozen of those scans, some of them the only surviving copies of specific images, were removed after the deduplication software identified them as near-duplicates of slightly different scans of the same physical print.

The software's logic is straightforward on paper: two images that are 95 percent identical waste storage space. But community archivists point out that two scans of the same photograph are not the same file. Resolution differences, color calibration, and crop framing can make one scan archivally superior to another. The automated system, they say, had no way to know which version to keep.

Similar complaints have come from the Excelsior District, where the Excelsior Action Group had uploaded photographs documenting the neighborhood's annual Outer Mission Street festivals going back to 2019. Members say several years of event photography were collapsed into a fraction of the original file count after a platform update in March 2026.

Pressure Builds on the City

The San Francisco Public Library's San Francisco History Center on Larkin Street, which maintains its own separate archive infrastructure, says it has not been affected by the same deduplication issue — a distinction community members note with some bitterness, since the History Center operates on archival standards that treat every submitted file as distinct regardless of visual similarity.

The gap between the city's professional archival practices and the commercial platforms procured under tech contracts has become a focal point. The city's current Open Data portal contract, which runs through the end of fiscal year 2027, does not appear to include explicit requirements barring automated deduplication on community-submitted image collections, according to documents reviewed by The Daily San Francisco. The contract value has not been independently confirmed by this reporter.

Residents in the Tenderloin and Excelsior are not the only ones affected. Members of the Western Addition's Fillmore Jazz Preservation Project say they submitted more than 400 photographs in late 2024 and can now locate fewer than 280 in their designated folder. The project was documented through a grant from Grants for the Arts, the city's arts funding arm, which distributed more than $11 million to community organizations in its most recent funding cycle. Whether the grant terms require the city to maintain complete file sets is a question project members say they have raised without a clear answer.

Community members and archivists are now pushing for three concrete changes: a mandatory audit of all city-contracted platforms to identify files deleted by deduplication since January 2024, a requirement that any automated file management process disclose its operation to uploading organizations, and a 90-day restoration window during which deleted files could be recovered from backup servers before permanent removal. For residents whose families have no other copies of these images, that 90-day window may already be closing.