So While i was testing my workflow,
i found that the dedupe filter is in the Google sheet node
where only new leads will be upserted.
But the problem was that,
let say i have set the lead count limit as 100 per run,
so for the first time in the Apify node,
It will go and fetch the first 100 people of the Job title, industry etc
what you have asked for.
But let's say when you run for the second time
Then the Apify node will fetch,
some of the previous lead also which were fetched in the first run
Now in the final output(i.e Google sheet) you will not get the duplicate leads
but what it was doing was,
the leads which was refetched by Apify
was travelling all the way through the enrichment process again
which was unnecessarily increasing the enrichment cost (which was already done).
So how did i fixed it?
Simply just before the Enrichment process i have added Two Nodes In the red box you can see:-
~Get Existing Leads(Google sheet Node) - It will get all the leads existing in the Google sheet.
~Filter Out Already Processed (Code node) - It will match the existing Leads by their LinkedIn URLs and Filter out the matching ones, And only allow the Non Matching Leads for the Enrichment process ahead.
So now we have to layers of filters one in the red box and other in the last google sheet node.
And this reduced the unnecessary cost and made my workflow Cost effective and more production ready.
Did you like my approach? let me know in the comments👇