CareerCup

Don't know why your question got voted down- this is a reasonable design question to be asked during an interview. You sir get a +1.

Launch a new thread to handle each request; each thread increments a sequen number as its first step (synchronized), reads the input data using a (small) bufferred reader and saves it on disk with a name prefixed with that sequence number. The optimal size of the buffer could be determined depending on the disc partition size and the size of physical memory.

In order not to create too many files in one directory, sequences modulo D (e.g.200) can be placed in subfolders (=sequence / D). In order to avoid having too many files open, a semaphor is needed to enforce a maximum # of threads launched. Requests above this threshold will not be accepted.

With respect to scalability, the RESTful web interface needs to be placed behind a load balancer and number of web servers scaled horizontally when # running threads reaches 90% of max # threshold.

Seems to be a neat approach , The above solution covers the case of large data ( for e.g the large images and videos ) also covers scalability . But I do not think it will cater to million of records if thrown at it .
I would recommend , if we are taking Cloud technology in picture .



1. We would chip off the requests coming to the us , in smaller sections like 32 bits , on which the each thread would operate. After chipping these requests would be held in memory till the thread actively gets them and process them further .


2. We would be launching a new thread , off-course to serve the incoming traffic . The thread increments a common sequence lying in-place in memory ( Not on disk since we would be having a bombarding requests coming in) 

3. Each thread for eg is allowed to be carrying itself some small data piece ( for eg 32 bits , equalling the size we chipped off the incoming requests. We would obviously be storing the initial request id while chipping the data off) . Once the sequence is attached to the thread , it will be written to the flat file residing on a high Input / Output disk dive .

4. We can dedicate a huge partition on the disk for incoming requests . On which no other application would be writing on .

5. The folder structure would be designed in a way that a single folder would be dedicated to the each request , for eg a large video would have all files ( related to the video residing in the folder ) .
6. The files will be renamed with thread_id_sequence_number_Data_file, so as to uniquely identify things .

7. We could have monitoring of Zombie processes , and too many file handlers lying ideal . We could cap the number of threads to be spawned at the particular time .

8. Since we have the same on cloud, we could have high availability Load Balancers across Geo having application redundancy across different Geos .
With added level of redundancy we can plan to take periodic backup of disk space .

9 . Obviously we would be removing the folders once the files in complete entity is stored on the backend DB .

10 . For serving this large data files we can devise using CDN functionalities .