CareerCup

Depth first traversal of a web page with all unique urls hit recorded in a hash to avoid cyclic traversals. On a http 404 / error code, traverse back to the most recent ancestor.

#1 This is an IO task, so we can implement multithreading. This is important to note.
#2 Don't send a GET request, send a HEAD request, that's enough to see if the link is broken.

I'm not sure how else we can make this more efficient, other than that just download all the responses and see which ones are broken! Broken error codes start with 4--.