CareerCup

For 1 MB of memory, even bit set would be 8000000 bits, which are not enough for 100TB of data ..any thoughts ?!

On a second thought, you can just sort both files using limited memory with external sort. Then it would just be two pointer logic.

without any precise calculations:

1 - let's say the RAM would be able to hold X items
2 - we will process on multiple chunks
3 - each step we will take the first X/2 from file 1 into L1 and first X/2 from file 2 into L2
4 - process the items in the lists L1 & L2 and remove the items present in both lists.
5 - put back the remaining items in L1 & L2 to their files (this step can be handled more efficiently.)
6 - repeat from step 3