CareerCup

Algorithm.

Assumptions:

1. Numbers are 4 byte integers.
2. Total memory 16 GB, Most of them memory used for loading numbers from persistent storage

1. Load, say, 1 GB numbers
2. Sort them using some sorting algorithm, like quick sort
3. Store/transfer sorted numbers back to persistent storage
4. Load next 1 GB numbers and repeat 1 to 3
5. Load 100 MB numbers from each 1 GB, and merge 10 chunks
6. Repeat this process for all 100 MB chunks of each 1 GB.
7. Repeat this merge process, until all numbers are sorted.

Under these assumptions I would do this:
* Create a bit array of 2**32 bits (only 512MB required).
* Load the huge array by blocks that fit in memory
* If a number in the block has been seen before (bit corresponding to this number is set), output it as a duplicate, otherwise set the bit.

Load about half of the size of available memory data. Check to see if there is a duplicate there. Load the next half from the dataset. compare it with the first set. If duplicate found return its value, otherwise upload the third set and so on until the entire data is checked. Now first set is finished. Ignore the first set and repeat the process until the duplicate is found.