#LINKEDIN TIMEFIBERS FOLLOWER FULL#
#LINKEDIN TIMEFIBERS FOLLOWER CODE#

#LINKEDIN TIMEFIBERS FOLLOWER CODE#

Code complexity: Sensei was composed of smaller components- Bobo, Norbert and Zoie.

The incremental addition of hardware made the system that much harder to operate and also increased the cost to operate it. As the number of content records in our index grew over time, it became necessary to add more hardware to keep the overall system performing well.

#LINKEDIN TIMEFIBERS FOLLOWER FULL#

Operational concerns: It was essential to keep the full inverted index in memory to achieve latency requirements required for feed applications.

As the relevance algorithm is embedded in the query itself, clients could easily A/B test different relevance algorithms.Īlthough Sensei could scale horizontally, supported relevance based feeds, and supported easy A/B testing of relevance algorithms, it needed to be replaced for the following reasons:

Top N records from this sorted list were then returned to the client. Records from all these searcher nodes were aggregated on broker and then sorted according to their relevance scores. After applying the relevance algorithm, each node returned top N records. Each searcher node applied some filtering logic to prepare a candidate set for the relevance algorithm. Requests were received by a broker service which fanned out requests to all searcher nodes. Queries to Sensei were specified in BQL language, which consisted of both filtering and relevance logic. Built-in support for complex relevance algorithms to rank content records in feed.Ĭontent records were ingested into Sensei’s searcher nodes via Kafka.This system could be horizontally sharded to cater to an increase in traffic. A distributed, elastic, real-time searchable database built on top of Lucene.Sensei, the generic search system built by LinkedIn, which previously powered our feed infrastructure, provided the following features:

Improved relevance: Because of increased content inventory, we needed algorithms that produce better recommendations in the feed, thereby requiring computationally complex scoring and ranking that can impact real-time performance. At the same time, LinkedIn's network graph has become denser, which means that content shared by a member gets distributed to a wider audience. Wider distribution: LinkedIn's member base has grown from 200 million to more than 400 million in the last three years. We created FollowFeed to meet two primary requirements: Compared to Sensei, our previous feed infrastructure, FollowFeed has driven an increase in query performance (a fivefold reduction in feed query’s 99th percentile latency) and data retention (20 times larger index) with a smaller hardware footprint (the number of servers was reduced by half). Twenty months ago, we set about building FollowFeed, which today powers all of LinkedIn’s feed experiences. We needed an infrastructure that could dynamically sieve through the content shared by a member’s network to deliver a list of relevant, ordered, and personalized content while satisfying all requirements of a highly available low-latency application. Our existing feed infrastructure, Sensei, suffered from increasing operational costs and it could not perform well with LinkedIn's growth curve in terms of content and members.

To support and continue to drive engagement, this infrastructure needed to transform. Dozens of LinkedIn products feature custom feed experiences which are all powered by a singular infrastructure. It helps them discover the most relevant content, read original views from people that matter to them, and take part in unique conversations about things they care about. The feed is the default landing experience for LinkedIn’s members.

Linkedin timefibers follower