Check out the simple Python script I made to clean Salesforce duplicates - 300k records processed.

While my typical focus is on build pipelines and deployment manifests, data quality is a critical—and often overlooked—pre-deployment dependency. A recent engagement involved de-duplicating a Salesforce instance with over 300,000 Contact records prior to a new CRM integration. The business logic for a "duplicate" was specific (matching on email *and* last modified date within a 7-day window), ruling out standard tools.

I built a Python script leveraging the `simple-salesforce` library. Its core responsibilities are:
* Batch querying records via SOQL.
* Applying the custom matching algorithm.
* Safely marking duplicates for deletion (using a custom field `to_delete__c`) rather than direct deletion, allowing for audit and rollback.

The script's effectiveness hinges on idempotency and logging, much like a good CI job. Here is the core structure:

```python
import pandas as pd
from simple_salesforce import Salesforce

def identify_duplicates(records, window_days=7):
"""Groups records by email, flags duplicates within time window."""
df = pd.DataFrame(records)
df['LastModifiedDate'] = pd.to_datetime(df['LastModifiedDate'])
df = df.sort_values(by='LastModifiedDate', ascending=False)

df['to_delete'] = False
for _, group in df.groupby('Email', dropna=False):
if len(group) > 1:
# Compare each record to the most recent in group
baseline = group.iloc[0]
for idx, row in group.iloc[1:].iterrows():
if (baseline['LastModifiedDate'] - row['LastModifiedDate']).days <= window_days:
df.at[idx, 'to_delete'] = True
return df[df['to_delete'] == True]

def main():
sf = Salesforce(username=user, password=pw, security_token=token)
query = "SELECT Id, Email, LastModifiedDate FROM Contact WHERE Email != null"
all_contacts = sf.query_all(query)['records']

duplicates_df = identify_duplicates(all_contacts)
print(f"Identified {len(duplicates_df)} potential duplicates.")

# Batch update marking records
for batch in np.array_split(duplicates_df, len(duplicates_df)//200 + 1):
update_records = [{"Id": rid, "to_delete__c": True} for rid in batch['Id']]
sf.bulk.Contact.update(update_records)
```

**Key considerations for production:**
* The script was executed from a Jenkins pipeline, with parameters for the date window and Salesforce credentials injected via HashiCorp Vault.
* Log output was captured as a pipeline artifact for validation.
* A dry-run mode was essential, performing all logic without the final `bulk.update`.

This approach processed all records in under 90 minutes. The pattern is transferable: isolate the business logic, enforce idempotency, and wrap execution in a repeatable, logged automation framework.

--crusader