Disclaimer: This is a technical article. Its about a new framework which I developed for WriterBabu.
Social network is a complex ecosystem and the volume of activity and interaction that goes on here is HUGE. The challenges faced by any social network can be divided into four broad categories:
1. Change propagation
Each small action of any user on a social network affects large number of data points and hence it becomes challenging to propagate that effect to all data points without compromising on speed. If done synchronously simple activities like taking a poll and commenting on a post will take more than 500 mili seconds on server( maybe more ) which is totally unacceptable to provide a seamless user experience. WriterBabu's server side response time is within 30 mili seconds on an average. To make things happen asynchronously and still maintain the consistency of each transaction a new framework was needed because doing tasks in batch mode using scheduled cron jobs cause hell lot of inconsistency.
2. Relationships
Another challenge that social networks throw is the challenge to maintain and deliver the relationships between entities. Its all about how entities are connected with each other. And entities are connected in multiple ways and the relationships are needed to process and deliver various actions. Calculating relationships in real time will definitely become a bottleneck once the scale is different.
3. Generating Activity Feed
Pulling data from almost infinite sources and showing in them in real time is impossible and hence there has to be a way to prepare the feed as soon as the activity happens. Every activity should update the activity log and the feed stories. Doing so synchronously is still doable at this scale but as the number of users and activities increase this updating feed and collecting information from various sources will need asynchronous system. Same goes with the notification system. Added challenge is that both the feed and notifications needs to be delivered almost real time, as they happen.
4. Complex rule processor
Complex rule processors are needed to trigger actions. The rules become complex due to the inherent personalised experience that social networks are expected to deliver. Someone is a friend to someone and someone follows someone, someone is anonymous for a post, someone is writing a personal diary. Different level of privacy options and different relationships among users make the conditions really really complex sometimes. Again, the constraint is time and speed.
(Random image to add colors ... )
To face all these challenges a system based on events and triggers was needed, and hence was born Backstack. Every activity triggers a chain reaction of computation and execution. For example an author poll taken on a post will trigger updation of poll count for that post, updation of poll count for that user, reputation score calculation for that user and badge calculation for that user along with innumerable other set of generic actions like feed story updation, generating notification to the author of that post and so on.
As soon as the trigger is fired, a job is pushed into a queue( the event tube) which is picked up by the event listeners on the tube to execute. The framework provides seamless management of mapping a trigger to updation of several data points and in turn mapping data point updation to execution of actions with option of applying a filter based on a rule engine.
I can't go into explaining the code of Backstack but I can for sure list down the stack behind backstack which will strengthen further the backbone of WriterBabu architecture:
PHP for scripting
MongoDB for storing log and other vital data
Graph DB Neo4j for maintaining data and relationships
Beanstalk for job queue
Redis to deliver ready to use data
NodeJs for computation intensive tracking and analytics ( still in progress)
Backstack has opened up doors for deeper personalisation, recommendation engine and machine learning. Lets see how useful it proves. For now the focus will again be on improving use experiences, bug fixes and better Facebook integration and making rest of the unresponsive pages into responsive.
Keep writing ... writing is fun ... :)
