Gone are the days when anonymous shoppers browsed generic stock for an elusive item. Today, anyone can serve customers unlimited, personalized offers tailored to their interests.
All you need is data, right?
With all the options online, you need a system that narrows down the possibilities. Something that learns what people like.
That’s where recommender systems come into play.
Implementing a recommender system allows you to turn raw data into personalized offers. And personalized offers result in higher customer satisfaction, engagement, and sales.
Sounds good, right?
That’s why the following article will tell you:
Already know that you need a recommender system for your project? We can help! At Iterators, we design, build, and maintain custom software and apps for startups and enterprise businesses.
Schedule a free consultation with Iterators. We’re happy to help you find the right solution.
So, let’s start with the basics:
What is a recommender system?
A recommender system is a type of information filtering system. By drawing from huge data sets, the system’s algorithm can pinpoint accurate user preferences. Once you know what your users like, you can recommend them new, relevant content. And that’s true for everything from movies and music, to romantic partners.
Netflix, YouTube, Tinder, and Amazon are all examples of recommender systems in use. The systems entice users with relevant suggestions based on the choices they make.
Recommender systems can also enhance experiences for:
And the list is not exhaustive. Bottom line? If you want to provide users with targeted choices, recommender systems are the answer.
Here’s an example of a recommender system in e-commerce. H&M served the following recommendations to users who clicked on “pleated skirt” as a potential buy:
So, what are the advantages of adding a recommender system to your website or software?
Here’s a list of just a few:
A recent study by Epsilon found that 90% of consumers find personalization appealing. Plus, a further 80% claim they are more likely to do business with a company when offered personalized experiences.
The study also found that these consumers are 10x more likely to become VIP customers, who make more than 15 purchases per year.
The moral of the story? If you’re interested in cross-selling or serving personalized offers, a recommendation system is right for you.
Another positive benefit of using a recommender system is that you solve the long tail problem of online shopping.
When you go into a brick-and-mortar store, you only see a limited number of items to buy. There’s only so much space, right?
So, recommending customers things to buy is easy. At the front of the store, you put your newest, most popular goods on display. Walk into a bookstore and there are three tables:
Now, go online. Boom! you’re presented with millions of options. And if you don’t know what you’re looking for, you might find the sheer number of options overwhelming.
The problem is known as the long tail problem.
So, how does an online retailer help customers suffering from information overload?
Recommender systems.
So, let’s say you want to buy a book. You go online to Amazon and the first thing you see:
It’s the same as the display tables in the brick-and-mortar stores. But once you start making choices on the platform, Amazon’s recommender system takes over. Let’s say you search for The Great Gatsby. Amazon recommends:
Here the system served you Fahrenheit 451. That’s because past Fitzgerald customers must have also bought Bradbury. As an alternative, your recommender system could offer other Fitzgerald books.
Using machine learning, recommender systems provide you with suggestions in a few ways:
For starters, popular examples of collaborative filtering systems include Spotify, Netflix, and YouTube. But what does a collaborative filtering recommender system do?
A collaborative filtering recommender system analyzes similarities between users and/or item interactions. Once the system identifies similarities, it serves users recommendations. In general, users see items that similar users liked.
There are different types of collaborative filtering systems including:
An item-item filtering algorithm analyzes product associations taken from user ratings. Users then see recommendations based on how they rate individual products.
For example, you rate a book or movie as a 10/10. Now, you will see the top-rated books or movies with similar attributes. Below is an example from Goodreads.
I created a special list for books that I gave five-star ratings. Goodreads then recommends me the highest-ranked books from similar readers’ lists.
It’s not always easy to get users to give items ratings. That’s why item-item filtering can be as simple as clicking on a dress and seeing more dresses.
Ever come across the “people who viewed this item also bought” copy under a product?
That’s right. That’s also an item-item filtering system.
Amazon invented item-item filtering for their recommender system. Item filtering works best when you have more users on your platform than items.
The other kind of collaborative filtering takes the similarity of user tastes into consideration.
So, user-user collaborative filtering doesn’t serve you items with the best ratings. Instead, you join a cluster of other people with similar tastes and you see content based on historic choices.
Let’s say you use YouTube for the first time. You play a Beyonce song. The system clusters you with other users who also like Beyonce. Then the YouTube recommendation system shows you other videos chosen by users in your cluster. The more choices you make, the more relevant the results.
recommendation system youtube example" width="1404" height="409" />
Ever wonder why your YouTube playlist gets messed up after a party?
It’s because Brenda decided to karaoke Disney songs for two hours. Brenda chose Disney songs, now you’re clustered with the Disney kids.
Content based filtering uses the characteristics or properties of an item to serve recommendations. Characteristic information includes:
Let’s use a movie recommendation system as an example. Characteristics for the item Harry Potter and the Sorcerer’s Stone might include:
A content based recommender system can now serve the user:
Of course, the list is not exhaustive. Once the user makes choices, the recommender system can serve more targeted results.
Let’s say they choose Swiss Army Man next. It’s safe to assume the user likes movies starring Daniel Radcliffe. The system tracks these choices and begins to recommend films starring Daniel Radcliffe.
The system may also show the user more Harry Potter movies. The hypothesis is that if a user liked an item in the past, they might like similar items in the future.
Content based filtering systems can also serve users items based on users’ profiles. You can create user profiles based on historical actions. You can also ask users upfront about their interests and preferences.
Pro Tip: Using a hybrid recommender system allows you to combine elements of both systems. In general, that means elements of one system can remedy the pitfalls of the other.
And now for the bad news. Each type of recommender system has its own set of problems. Let’s take a look.
Collaborative filtering needs a lot of data to create relevant suggestions. So, when you start using a platform with a collaborative filtering system, you start cold.
The cold start problem in recommender systems is common for collaborative filtering systems.
For example, when John visits YouTube for the first time, the system has to wait for him to watch several videos. Only then can it serve him relevant recommendations for other videos.
A solution to the cold start problem in recommender systems is clustering data with attribute similarities. Let’s go back to our YouTube example.
John visits YouTube for the first time. The first video he selects is a Beyonce video. As mentioned before, the platform will cluster John with other users who watched the same video.
It could also add him to other clusters. Let’s say the video belongs to the “pop song” cluster. Needless to say, the pop song cluster is populated with pop songs, such as “Hit Me Baby One More Time” by Britney Spears.
Now, the system can recommend other songs based on the following criteria:
Another solution is to start by recommending users popular items.
YouTube and Amazon’s homepages both show users trending or popular items. IMDb has a similar strategy, showing new visitors the top-rated 250 film titles.
A final strategy is to request new users to provide the platform with information.
The problem with content-based recommender systems is that they are restrictive. You click on a dress and you see more dresses. The system is incapable of knowing that your interests go beyond liking dresses.
Again, a common solution is to ask users upfront about what kind of things they like. And as users interact with your site, you can use historical data to recommend them more tailored choices.
The customer buys a dress and some shoes. Now, you know that she likes both.
There are a couple of ways to go about adding a recommender system to your software or website. But first, you need to find the right people.
Usually, implementing a recommender system comprises three activities done by different people. These activities include:
Here’s a basic flowchart of a recommender system:
To oversee these activities, you would need to hire (respectively):
Let’s say you want to hire in-house people who will implement and manage your recommender system. Expensive? Yes. Here are the average salaries for each according to Glassdoor:
Not to worry. You don’t have to create a recommender system from scratch. There are ready-made SaaS solutions for recommender systems in e-commerce.
And for movie or music recommendation systems, there are off-the-shelf solutions. For example, it is possible to get an algorithm similar to the one that runs Netflix’s recommendation system.
Depending on your needs, you could also consider outsourcing. Outsourcing is beneficial because it enables flexibility and can be cheaper. Plus, you’re sure that you’ve hired people who know what they’re doing, and you don’t have to worry about employee turnover or recruitment.
Pro Tip: You don’t want to hire a data scientist before you have data. They won’t have any work and they will leave or get stuck doing the work of a data engineer. So, it’s best if you hire a data engineer first.
Need to hire a programmer but don’t know the first thing about programming? Not sure how to get started when you’re looking for a data specialist? Then read our article: How to Hire a Programmer for a Startup in 6 Easy Steps
First, you need to have your data engineer gather data for your recommender system.
As you can see in the chart, there are different datasets you might want to consider like:
User data is necessary for serving user-item recommendations via a collaborative filtering recommender system. It does require you to have access to a large number of user interactions.
To get user data, you can either ask for ratings or draw conclusions from user behavior. Asking for ratings is problematic because most users don’t bother to give items ratings. Drawing conclusions from user behavior is problematic because of the cold start problem.
As mentioned before, some solutions include:
Second, you can design an attribute ontology to gather item attributes for item-item recommendations. If you have limited metadata from users, you can use item attributes to create a content-based recommender system.
Either way you’ll need your web developer to integrate both front and back-end features into a supported system. Once that’s in place, the process runs continuously.
Remember, the more data you have and the fresher it is, the better your recommender engine.
Once you have data, your data scientists design a way to use it to build recommendations.
Building a utility matrix using matrix factorization techniques for recommender systems is a popular way to get started.
Do keep in mind that there are many ways to build recommendation systems. The following example is a highly simplified version of what a system looks like in reality.
The technique allows you to complete unknowns on a matrix based on user-item interactions.
Here’s a very basic example of a utility matrix for recommender systems:
When using a utility matrix, you should assume that the data will be sparse with more blank spaces than rankings. The goal of the matrix is to predict how John and Jane will rank the remaining films. Utility matrices organize rankings users assign to different items. In the example above, John gave Monty Python and the Holy Grail the highest score, while assigning The Meaning of Life a two. He didn’t rank Life of Brian.
The example above doesn’t give much insight into whether Jane will like either of the two films she did not rank. But in a real-life scenario, you would have much more data.
A real recommender system would compare her score with hundreds of similar scores. Patterns begin to emerge. For example, let’s say people who gave Life of Brian a 3 gave Holy Grail a 5 while giving The Meaning of Life a lower score.
The logical conclusion? Jane will like Holy Grail more than The Meaning of Life as the others did. So, let’s recommend her Monty Python and the Holy Grail.
Regardless, building a utility matrix requires large amounts of data. And that data is always going to be sparse, so your recommendation system algorithms will need to account for that.
Once the system is in place, data engineers flood the system with vast amounts of data. The model then runs in real time, preparing batches of recommendations for users.
Pro Tip: To populate a utility matrix with behavioral data, you use “1” for “like” and “0” for no action taken. Using 0 doesn’t mean the user dislikes that thing, only that she took no action.
Finally, you’ll serve users recommendations through a user interface or email campaigns. Your web developer integrates your recommender system with your website for that purpose.
Once users interact with those recommendations, guess what?
Your data engineers have even more data to feed your recommender system. Your data scientists can start collecting performance data to test your recommender system.
You’ll need to test for effectiveness so you can continue to improve recommendations.
How do you know if your new recommender system is up to snuff? Well, there are a few ways to measure the effectiveness of your recommender engine:
But first, you might want to take a look at the different features of recommender systems. You can use features as metrics for benchmarking.
For example, coverage is the degree to which you cover all available items and actions with your system. It’s a useful metric for making sure your system is thorough. Keep in mind that not every feature is suitable for every type of recommender system.
Once you’ve decided which features you need, you’ll want to make sure they’re working. You can do that by testing your system.
In the beginning, you will have no real knowledge about your users or their interests. So, how do you create personalized offers for people you don’t know?
The easiest way to test what will be effective is to create user personas. There are various assumptions you can make about your future users before they start using your platform.
Let’s say you sell books.
You can create personas based on choice prediction. Michael selects a book. With some statistical accuracy, your recommender system can predict whether Michael will like another book. You could base it on the genre, author, and other content attributes.
You can create personas along these interest paths to serve users initial recommendations.
A good way to assign user personas is to have new users answer a few brief onboarding questions. And that’s especially true for mobile applications.
Here’s an example of Foursquare’s onboarding process:
Good onboarding questions might also include requests for demographic data about your users. Knowing simple information like age and location targets your recommendations. If your reader is 15, she is going to be much more tempted by a Twilight recommendation than if she’s 80.
One free and easy way to get some basic insight into user demographics is to use Google Analytics. Most of you already have Google Analytics hooked up to your website. So, why not use it to give you insight into user demographics?
Google Analytics shows you the age, gender, and interests of the people visiting your website. You could also use social media analytics to get ideas for user personas based.
To use Google, you click on “Audience” and then “Demographics” to view the age, gender, and interests tabs. Enabling the advertising features can give you a deeper look into user demographics.
It is important to note that offline recommender system evaluations are often treated with skepticism. That’s because you use offline methods before the launch of a recommender engine. So, instead of involving users, you run simulations.
And the results of these simulations may not correlate with real user satisfaction. The responsibility of running a simulation will fall to those who created your recommender system.
Once you launch your recommender system, you run online evaluations. These evaluations comprise various A/B tests that you serve to users live.
If you’re unfamiliar with A/B testing, it involves serving different options to users who arrive at the same point. One user sees option “A” while the other sees option “B.” Whichever option does better, that’s the one you continue to serve all users.
By running A/B tests, you can see which recommendation inspires more clicks and conversions. While ideal, in the long run, this form of testing can have a negative impact on revenue and user experience if users don’t like either option.
Internet users and online shoppers have come to expect personalized experiences. At the very least, they want websites to make recommendations so they don’t waste time sifting through things they don’t like.
Recommender systems are a great way for any business to personalize their offers. While implementing a system may be costly, you’re sure to benefit from targeting your content.
Whether your goal is to keep people on your platform (YouTube) or show users exciting, new offers (Amazon) – a recommender system is the answer.
Natalie Severt says:
Thank you for reading! I hope you enjoyed the article. Let’s start the discussion:
1. Has anyone experienced an increase in user satisfaction and engagement after implementing a recommender system?
2. What’s your go-to strategy for overcoming the cold start problem in recommender systems?
We’d love to hear from you! Let us know if you have questions or a great recommender system success story! Leave us a comment and we’ll be happy to respond 🙂
Reply By Post AuthorDapi Sibeko says:
The article was easy to read and yet very informative
Natalie Severt says:
Thanks, Dapi! I’m glad you found the article enjoyable 🙂
Reply By Post AuthorDavid says:
Well written and informative article, thanks
Sharvari says:
Good article and sums up all the basic information @ recommender system.
Natalie Severt says:
Thanks, Sharvari! Glad you enjoyed the article.
Reply By Post AuthorLuisa Sequiro says:
Very interesting, thank you. I’ve started learning Python for data science and have an interest in building an online recommendation engine, so this is right up my alley.
Michał Kowalewski says:
Thank you, and good luck with your endeavors!
Luisa Sequiro says:
amir mortezaie says:
well written.
Thanks 🙂
Dennis says:
thank you very much for informative article😀😀😀🙏🙏
