For 24 hours between Monday and Tuesday morning, we ran a test on the paginated homepage that appears after clicking the load “More Stories” at the bottom of the homepage. The curation module has always appeared at the top of these pages, but with the introduction of new curation module styles a few months ago, seeing this module multiple times in a row became a less ideal user experience because of how much larger the new designs are on each homepage.
The current behavior for pressing that button is that the homepage reloads with the next stream of articles and lands on the curation module, which the reader already saw on the previous page. We wanted to test whether removing the curation module from paginated homepages affects reader behavior compared to the current set up.
Branch A: Curation module present on all homepages (Current Behavior)
Branch B: Curation module absent on pages after clicking “More Stories” on homepage and next page loads
Overall, this test was a success — CTR increased on the curation module across all page sizes even though the number of clicks remained consistent between the branches because the number of pageviews with the curation module decreased. The number of curation module clicks remaining consistent across the branches indicates that few people were clicking on it after pressing the “More Stories” button. As such, we wouldn’t be losing a significant number of clicks by removing the module after the first home page.
Looking at the overall CTR on the page though brings some mixed results.
On mobile-sized screens (<600 pixels), we’re seeing an increase in overall CTR driven by an increase in stream CTR, which makes sense as the curation modules on mobile take up a large amount of screen real estate. Mobile homepages without them show users the stream of articles immediately.
On 1024+ sized screens though, we’re seeing a slight, but statistically significant decrease in CTR on the test branch. This is strange because after diving into the data, we saw this decrease in CTR was on the first page, which had the curation module in both branches. And after checking the data by browser and website, and QAing the click tracking, nothing stood out as an obvious bug that might cause this difference. Also the decrease between the two was only about 1 percent and it’s possible that this is just a random result, so we aren’t too worried about this result.
Ad viewability for the top slot did decrease slightly (but statistically significantly) on desktop. This also makes sense though because generally the first thing a user does when a page loads is scroll down toward the content. On pages without the curation module, the top slot ad is higher up on the page, so it can be scrolled out of view quicker, leaving fewer opportunities for ads in the top slot to be viewable.
Finally, user behavior on the session level remained consistent across the control and test branches. There was a small decrease in bounce rate overall, but it’s not clear that this is related to our test. On both branches, users who see a “More Stories” homepage, by definition, did not bounce (because that page that loads has to be their second page view). Either way this change is so minor, that it shouldn’t turn heads either way.
This change is a win for user experience and the number of clicks on the curation module remained consistent between the control and test branch. The change in overall CTR on 1024+ browser sizes is weird, but the decrease is minor and we didn’t see any obvious bugs that were causing this. Feel free to also do some QA and see if anything stands out.
The one trade off would be a very slight decrease in view ability in the top slot, but that should only affect top slot ads on “More Stories” homepages.
- Tyler Alicea