News Aggregation: It Should Be Processed Not Presented

Since the initial days of the smartphone/tablet revolution, reading news has been a popular activity via apps, mobile websites, SMS, social media mentions and instant message networks.

Most news organizations – both print and broadcast – have built their own apps, mobile websites and a host of aggregators have tried to provide their own abstraction and interpretation of news. Pulse, Flipboard were one of the earliest popular news aggregation apps.

Most news aggregators have systems to track and classify news into sources, categories and stories. Users are provided an option to browse across this classification. Some basic cross source, cross category access is provided through “related” article tags. But for most users, the workflow consists of selecting a source, category and story or category, source, story and getting to the detailed news.

I have always wondered why the aggegators present news with the source, category and story hierarchy when clearly there is merit in presenting news based on importance of the story/theme that the article covers. When something happens (reason for news story to be created) each news organization puts up its own version of the news story. Depending on the importance of that event, personality or issue the volume of stories covering that news  changes. An aggregator system should be able to look at the volume and momentum and present a ranking of news stories to the user – so that the user can also get a sense of the importance of the news story.

So As a user, I should be getting the top-most story, with all the related stories into a single view – irrespective of the source, so that as a reader, I can determine which all perspectives (sources) I want to read in detail. This will a) give me insight into the relative importance of the news and b) save me time from moving in/out of the source/category/story hierarchy.

This has been bothering me and so we decided to build an app to test this idea. Instead of an aggregator that “presents” you the news via sources, categories and stories, we built a solution that “processes” the news and shows you the top stories at any time. You can choose any category and hopefully in a future version, any keyword and the application will show you the top news for that selection at this time.

Here’s how this works:

1)      First we track news feeds, ie sources and cache it in our database. RSS feeds are the best source. We currently track about 1000 RSS feeds.

2)      Then we categories them according to the source or according to the Feeds.

3)      In our first step of “processing” we pick all the Articles of each category and make keywords from Title and Description of that article .

4)      Then we pick all the clusters of that category (In starting of that category if there is no cluster then we add those keywords to make a new cluster) and find similarity between old Cluster keywords and our new article keywords then we calculate Score’s for each matching ( Using Lucene tool )  & according to the Score’s we decide in which Cluster we have to add this article or we have to add a new. So, in effect we sort each story into a cluster. The cluster with the highest number of stories is most likely the hottest news event.

5)      Finally, the clustering is filtered according to time. We process stories from morning till afternoon; AND hourly updates thereafter to determine the top cluster at ay time. All the related stories from a cluster are shown with the latest article in descending order of time.

Essentially, our news application is telling the reader – here is the top story at this time and there are X related articles on this story, in the decreasing order of time. Now the reader can decide to read about the story either from the initial source that first reported it or the most popular source or the latest source or any other source that has the better worded title.

You can try this news reader on the Android Play Store. We have called it “inNews”. We are working hard to populate more feeds and build relationships with the news publishers so we can continue to innovate. Do download once and give it a spin – see how “processed” news changes how you think of news when it is “presented”.


Get it on Google Play

2 thoughts on “News Aggregation: It Should Be Processed Not Presented”

Leave a Reply

Your email address will not be published. Required fields are marked *