CareerCup

NoSQL Solution..esp MongoDB seems to be a good fit here

MongoDB

Initially, i also thought that we would require a NoSQL solution. But thinking again, we don't have a schemaless usecase here. The database is having two properties of integer and string, which could easily be modelled as a table in a MySQL based database. Secondly aggregation is also pretty straightforward when using a MySQL based solution. So the database can be implemented using MySQL Cluster which can scale according to needs.

We can run an agent in each system, that pushes the logs near real time or periodically to an s3 bucket. We can use s3 upload events to an sqs queue. This queue can be processed by an array of servers, which process the logs and upload to the mysql cluster.

On the other end of the cluster, it is merely reading the data and aggregating it according to requirements.

Using Kafka to push data to the storage server, store in HDFS. Using Hive for data pull. Hive also supports aggregation. Remember to use a good partition (App name for the log, time, etc), otherwise Hive query will be slow.