You ran the same scraper for 10 search terms, or for 5 cities, and now you have 10 datasets with overlapping items. Before the data goes to a sheet, a CRM or the next step of a pipeline, you need one clean list.
An order of operations that avoids surprises:
- Read all sources in a fixed order (inline items first, then each dataset).
- Filter first, on the original field names (e.g.
price < 20,email is empty). - Deduplicate next, by the field that identifies an item (
url,id, orname + city), ignoring letter case and extra spaces. - Decide which copy wins: the first, the last (handy for incremental scraping: merge yesterday and today, keep the latest), or a merged version where empty fields are filled from the other copies.
- Only then pick, drop, flatten and rename columns, so your filters and dedup keys don't break.
- Keep a summary per source: items read, filtered out, removed as duplicates, and written.
I packaged that flow as a small Apify Actor, Dataset Merge, Dedup & Transform: https://apify.com/innovative_cookie/dataset-merge-dedup
It only gets READ access to the datasets you pick, uses no browser and no proxy, and there is no Actor fee right now (you only pay Apify platform usage).
One memory note: keep: first streams the data, while keep: last and keep: merge hold the kept items in memory, so raise the run memory for millions of items.
How do you handle duplicates between scraper runs today?