Eyetools Eyetracking Research

Small decreases in viewing can decrease the probability of being clicked by more than 50%

For a long time, proponents of eye-tracking have said that the amount of viewing a link receives directly effects its likelihood of being clicked.  Well, it turns out that this is particularly true in the case of search results.  Recently, Eyetools and Sendtec released a study showing that a decreased viewing on a search results page has a huge impact on click-through conversions. This is true for both paid and organic listings. 

For example, the data shows that if overall viewing of a link falls from 80% to 60%, the initial probability that the user will select the link decreases by over 50%! This helps to explain why even small drops in page position for organic search links has such a significant impact on a site's conversion rate. 

In addition to emphasizing the need for ads and organic listings to have high page positions, this result also means optimizing link performance is critical since not everyone can be a top organic or adwords listing. 

These results may seem obvious in hind-sight, but I admit that I was very surprised to see just how big the price a page element can pay if it's not seen. I've included the excerpt below in the hopes that it will open a dialogue about the effect of increased viewing for other kinds of web pages. 

Click here to read the full report on Sentec's site.

----------------------- REPORT EXCERPT -------------------------

The graph shows an important correlation between increased link viewing and the probability of being clicked. [...] The page elements which received the most clicks were those seen by the most users.  As user viewing decreased, so did the number of clicks.  [... Our data shows that] decreases in viewing will have a disproportionately large negative impact on click-through.

A number of other studies have explored various factors which influence a user’s decision to click including link text, description, and perceived link relevance. Our study shows that percent user viewing can be an excellent predictor of user click behavior for online searches.  Our study supports the logic that users must first see a link before they can make selections based on text descriptions or relevance. There are many factors which contribute to whether or not a user decides to select a link, but all things considered, eye tracking data is a reliable and efficient way of understanding why people do and do not click.

It is important to note that this data is specific to search results and should not be used in conversation about standard web pages. Eyetracking has proven to be very helpful in directing redesigns of web pages, landing pages, and emails with great increases in conversion rates, but this data presented here is specific to search result pages and should not be generalized to other pages.

Article by Teresa Hernandez - Eyetools, Inc.

But what does it all mean? Understanding eye-tracking results (Part 5)

Part V:  Time and the Heatmap

Series topics:

Part 1: Misinterpreting the data (intro article)
Part 2: What can you learn from eye tracking data?
Part 3: What is a heatmap... really?
Part 4: What is a scan path?
Part 5: Time and the heatmap

In my previous post, I mentioned that heatmaps do not have a time component. Several people have asked me to discuss this topic in a little more detail, so here we go.

Important point #1:

A heatmap represents which content was seen by a group of participants. A heatmap helps to answer the questions “Where did users look?” and “Where didn’t users look?”

Important point #2:

If you are analyzing a heatmap, you are momentarily saying “Ok, in this instant we don’t care how long people looked at things, we just want to know what they saw.”

A heatmap does not answer the questions:

•    How long did someone look at my page or page element?
•    Did users look away and look back?

To further clarify these points, here’s a general overview of how a heatmap is created (click illustrations for a larger view):

Step 1:  Collect data A single participant views a web page. We record her eye movements as she browses. This gives us data which can be represented in a gaze replay animation. Now imagine that the animation is composed of a stack of stills, just like a flip book. Each page of our eye tracking “flipbook” contains the X,Y, and Time coordinates of a single fixation.

Step 2: Collapse individual data sets along the t-dimension Keep imaging our eyetracking data as a flipbook (3D data structure).  Now we will collapse the stack along the time axis.  In its most basic form, the calculation will take the area of a web page and note every location where the participant fixated.  By collapsing the time dimension, we remove it from all further calculations (poof!)  This means that someone who fixated the center of a page for 100ms will end up with the same fixation summary as someone who stared at the center of the page for an extended period of time.

Of course, the algorithm for doing this computation is more complex than the example I just presented. Any heatmapping algorithm should take into account peripheral vision, microsaccades, blinks, fixation duration, “bad data”, ocular drift, dynamic page behavior, etc.   For now let’s continue with the simplified example. Steps #1 and #2 are completed for, oh, say 15 people total. This gives us 15 individual fixation summary plots.

Step #3: Compute an average viewing value for each pixel of the webpage Again the algorithm which handles this step is more complex than what is presented here, but the basic idea is… For every pixel of a web page, the system asks “how many people saw this pixel?”  If 10 people saw the pixel out of a group of 15, then the algorithm says “66% percent of people saw this pixel… color it yellow on the plot.”

This averaging algorithm outputs a heatmap showing what percentage of participants saw each page element.  Handling the data in this way keeps any single individual from biasing heatmap percentages.  For example, imagine that 14 people only looked at the center of a page, and 1 person looked at the entire area of the page.  Even though 1 participant viewed a much larger area than anyone else (and probably spend a goodly amount of time doing it), she still only represents 6.7% of the 15 person group.  The resulting heatmap would then show that 100% of people fixated the center of the screen, and less than 10% looked elsewhere.

Other questions that have come up:

If you make a heatmap from only the first 10 seconds of viewing, doesn’t that add a time component to the heatmap?

The answer is no.  The method for computing a heatmap is the same no matter how large the time sample used. When you slice for specific time intervals, you are just selecting a specific group of fixations to include in the calculation. The resulting heatmap still has no time component.

Are heatmaps created from time slices more valid or informative than those created from full experiment sessions?

I suppose that depends on the kind of information you want to get from the plot.  If you want to see where people looked in the first 10 seconds, a time slice heatmap is appropriate. If you are trying to understand an order for page element viewing, a heatmap is probably the wrong analysis tool. If you want to see where people are looking over their entire experience with the page, a full session heatmap is the way to go.

This is the last planned installment of our “But what does it all mean?” series. However, the discussion is still open, so if anyone is interested in other topics, just let me know.

Article by Teresa Hernandez - Eyetools, Inc.
Illustrations by Boyd Richard - Eyetools, Inc.

But what does it all mean? Understanding eye-tracking results (Part 4)

Part IV:  What is a scan path?

Series topics:

Part 1: Misinterpreting the data (intro article)
Part 2: What can you learn from eye tracking data?
Part 3: What is a heatmap... really?
Part 4: What is a scan path?
Part 5: Time and the heatmap

I believe I referred to “scan paths” or “fixation patterns” in earlier entries.  Understanding scan paths are critical to interpreting eye movement results.

A scan path is a sequence of eye fixations made when someone samples a visual scene. Put simply, a scan path is the path our eyes follow when looking at an image.

One scan path image shows the sequence of fixations made by a single user.  Please note that individual fixation patterns are very noisy. However, if you look at a number of these traces, patterns can emerge. What we look for in individual scan paths are “orderly viewing patterns”.  In other words, we try to determine whether or not a person’s order of fixation follows a logical flow through a page.

Sample email heatmap (15 participants) and scan path.

At Eyetools, we produce scan path images which look like the image to the right.  A key to this can be found on our website. 

Qualitative analysis of scan path patterns is critical to any (usability targeted) eye-tracking study. Scan paths can be especially useful when presenting to design, marketing, and usability groups.  They very effectively illustrate how individual users are navigating the page.  However, when doing your basic analysis, experience and a working knowledge of eye movement/scene search literature will get you the most reliable read from these results.

There are many ways to discuss gaze traces.

Here are just a few examples of the imagery I've found works well when explaining traces:

• Think of view flow as a stream.  Are users flowing smoothly and unhindered through page content? Are things like ads or section breaks helping to guide the eye, or is the design placing a big boulder in the stream forcing users to detour around a section?

• Is there a "spaghetti effect"? Do many individual scan path images look as though someone dropped a handful of spaghetti on the page? (splat!) If users are searching a page at random, they are probably lost and aren't internalizing the message your page is trying to convey (at least not in the most efficient manner). 

• Does viewing suffer from a "pendulum effect"?  Do viewers use a page element as an anchor point and oscillate back and forth in multiple directions?  (Think Foucault's Pendulum) This pattern can result in decreased read-though of content, and has a number of other implications.  However, the severity of this problem really depends on the individual page content, design/client priorities, etc.

From this point on, any number of other metrics can be considered for data analysis.  The data you choose to examine more closely all depends on what you want to learn from your study.   

Continuing the Discussion: Methods and More

A note on software:

At Eyetools, we use in-house software to analyze our data. This has allowed us to develop tools that many commercial software systems don't readily provide. I'm not entirely familiar with the limitations of commercial and open source tools, but I assume that a large portion of stage 1 (heatmap analysis) and stage 2 (qualitative scan path analysis) is very common place.

But regardless of what kind of analysis algorithms you use (commercial, in-house, open source, etc.) it's important for the experimenter to have a solid understanding of eye movement terminology, experimental methodologies, and eye tracking algorithms.   After all, it helps to know (1) what you are actually measuring with an eyetracker, and (2) what the measurements mean, and (3) what the limitations of your hardware and software are. (Well, that's true in any field isn't it?)

A note on methodology:

Task instructions and participant preparation (priming) are important concerns in any behavioral or survey style study.  Eye-tracking is no different.  Rather than discuss methods at length, I’ll just pop in some pointers to other discussions and resources. 

•    [pdf] Granka, L and Rodden, K (2006). Incorporating Eyetracking into User Studies at Google. ACM Conference on Human Factors in Computing Systems.
•    [blog] Matteo Penzo's  Introduction to Eyetracking: Seeing Through Your Users’ Eyes
•    Suman, S and Bradburn, NM (1983). Asking Questions. San Francisco: Jossey-Bass.

Article by Teresa Hernandez - Eyetools, Inc.

But what does it all mean? Understanding eye-tracking results (Part 3)

Part III:  What is a heatmap… really?

Series topics:

Part 1: Misinterpreting the data (intro article)
Part 2: What can you learn from eye tracking data?
Part 3: What is a heatmap... really?
Part 4: What is a scan path?
Part 5: Time and the heatmap

A heatmap can be defined several different ways, but as they say, all roads lead to Rome. I find the 2 most useful ways of describing a heatmap are:

1 – A heatmap is an aggregate representation showing what areas a group of people fixated when viewing a webpage or email. The image shows where the sample group looked or didn’t look.  Well produced heatmaps should also take into account a reasonable percentage of peripheral vision. 

And

2 – A heatmap is a probability map which tells you what page content users are most likely to see. This is only true when the number of individual sessions used to create the heatmap is large enough.  At Eyetools, we recommend 15 people in most cases.  This number is based on what we have found gets our clients reliable results and the best cost-benefit ratio (bang for their buck).

Brand modified sample email. 17 participants.

Heatmaps are striking images, and very informative analysis tools.  However, they are just the tip of the iceberg when performing a proper analysis of eye movement data.  This is why companies who specialize in eye-tracking research have begun popping up. 

What can’t you learn from a single heatmap?

•    View order. Are users scanning the page in the most efficient way possible?  Are they forming a coherent message or are they lost? Sometimes a heatmap can be covered in "hot spots" but read through was very low.  More analysis is needed.

•    Copy vs. Layout Problems.  Heatmaps can help experimenters determine whether a page’s layout or copy is responsible for decreased viewing.  However, further analysis is needed to make a complete diagnosis.

•    How to improve page performance.  This is really knowledge that can only be gained through experience.  Viewing patterns can be affected by a number of different factors.  Heatmaps can help you to understand if there is a problem, but not how to specifically go about making changes.  (That’s why we're here. Please excuse the shameless plug.)

•    Variability.  By definition heatmaps do not have a measure of variability (error bar) associated with them.  I have a few thoughts, and (some rather interesting findings) about the variability in heatmaps and other eye-tracking metrics, but that is another discussion all together.  If anyone is particularly interested in discussing analysis at this level, let me know.

•    Time Components. Heatmaps do not have a time component. Time spent looking at a page element, or the page as a whole, is not represented at the heatmap level. 

Article by Teresa Hernandez - Eyetools, Inc.

But what does it all mean? Understanding eye-tracking results (Part 2)

Part II:  What can you learn from eye-tracking data?

Series topics:

Part 1: Misinterpreting the data (intro article)
Part 2: What can you learn from eye tracking data?
Part 3: What is a heatmap... really?
Part 4: What is a scan path?
Part 5: Time and the heatmap

People often ask me what exactly they can learn from eye tracking.  I have a short version answer which is:

We track:

•    Where people look
•    Where people click
•    What people ignore
•    And we discover why they decide to click ...or not click.

Why does it work?

•    Your eyes are hardwired into your brain and the eyes cannot lie.
•    Eyes can’t be "put down" like a mouse between clicks.
•    Eyes + clicks + subjective questions give a comprehensive view into the user experience.

This explanation is pretty common, but in reality, you can gain much more insight into the user experience from these kinds of studies.  Some advantages of eye tracking studies include:

•    Biometric measurements are more accurate than user feedback.  User feedback is invaluable, but notoriously unreliable.  Feedback regarding feelings, opinions, etc. must be acquired through survey methods.  However, measuring natural behavior gives a much more accurate picture of a user’s immediate experience than asking them after the task is complete.

•    Eye-tracking data can be used to accurately predict user feedback. This is true for questions regarding ease of use, ease of navigation, etc. All of these items also affect user confidence in the site and company.

•    Site navigation patterns can be mapped.  For instance, we can map out common navigation paths from a homepage to an interior page, and can begin to understand why some are being used more than others.

•    Results are representative of a “natural work environment”.  In other words, eye tracking does not require that a moderator be in the room with the user. Obviously the studies are done in a controlled environment, but not having other people in the room makes the experience very comfortable for users.

•    Viewing order of pages and page elements can be established.  We begin to understand what information is most likely to be seen/missed by users, and in what order.  For example, are users viewing a “Free Trial” offer 1st or 15th when seeing the page? Now you can know.

•    DHTML elements can be tracked separately.  The visual effectiveness and frequency of use of DHTML elements can be studied.

•    Short iterative testing can also be implemented. Because we can test prototypes (yes, even jpeg mock-ups), short eye tracking tests can be used to modify designs quickly.  This kind of testing is not a standard use of eye-tracking, but is proving to be very effective.

•    Page element placement, copy, etc effects can be ranked.  Should I change my header text? Move a menu to the right rail? Change the icon size or background? Certain layout changes can be ranked as to how much effect they will have on viewing patterns.

One point to keep in mind is that eye-tracking, like all other usability tests, is not going to provide all of the answers by itself.  Eye-tracking is a valuable and powerful tool when implemented alone.   But if you want the best over all picture, use it in conjunction with other usability tests, and as a part of an iterative process.

Does eye tracking measure visual attention?

Yes, eye tracking can estimate the areas of an interface which receive visual attention.  The “bouncing around” of the eye trace shown in the video is created by a series of fast eye movements called saccades.  When your brain is planning an eye movement, it shifts covert attention to the eye’s destination [ 1].  The attentional shift and saccade movement have been shown to be inseparable [ 2 ].  And, of course, once you fixate something, your visual system starts processing the image.

Does this mean that we remember 100% of everything we fixated on a screen? Definitely not.  Our brains can suppress images, or use the visual signal to inform any number of low level cognitive functions. Does it mean that what we fixated has had an opportunity to directly affect our experience with an interface? Absolutely.

Gaze trace information helps us to understand the areas of a page which most helped to form a user's visual experience with the site or email.

References:

Here's an EXELLENT introduction to eye tracking and usability:
Matteo Penzo's Introduction to Eyetracking: Seeing Through Your Users’ Eyes

[1]   Shipp S. (2004) The brain circuitry of attention. Trends Cogn Sci. 8, 223-30.
[2]   Peterson MS, Kramer AF, Irwin DE. (2004) Covert shifts of attention precede involuntary eye movements. Percept Psychophys. 66, 398-405.
[ref]   Liversedge SP, Findlay JM. (2000) Saccadic eye movements and cognition. Trends Cogn Sci. 4, 6-14.

Article by Teresa Hernandez - Eyetools, Inc.

But what does it all mean? Understanding eye-tracking results (Part 1)

Part I:  Misinterpreting the data

Series topics:

Part 1: Misinterpreting the data (intro article)
Part 2: What can you learn from eye tracking data?
Part 3: What is a heatmap... really?
Part 4: What is a scan path?
Part 5: Time and the heatmap

In 2000, the Poynter Institute released their first study analyzing how users view online news websites. Yet, 7 years after eye-tracking made this first major impression on the usability and marketing industries, there still seems to be a lot of confusion over what eye-tracking data can actually tell you about how users interact with your site.

Vague descriptions of methodologies and misinterpretation of eye-tracking data has lead to skepticism about the validity of eye-tracking in usability and marketing research.

Getting answers to common questions

In this next series of blog entries, I thought I’d take a shot at dispelling some small fraction of the confusion surrounding eye-tracking research.  Over the next few weeks I’ll address some recurring questions I get about our research, and the optimal use of eye-tracking studies.

Questions like:

•    What is a heatmap… really?
•    How to read a scan path… and what is a scan path?
•    How do you get the most out of eye-tracking analysis? (What many commercial software packages won’t tell you)
•    Basic eye movement terminology and why is it important when interpreting results?

Bad web design is not a good thing.

Just to start us off, I thought I’d share one of my favorite misinterpretations of eye-tracking data.  This originally appeared in a blog entry last year:

I think web surfing is a hunting activity. The eye is looking for anamolies, for things that don't belong. (That might be why the word anomaly, spelled wrong in the previous sentence, got your focus). […] One of the takeaways is that bad web design might actually be a good thing! Slightly bad design isn't familiar. It's off. It demands attention. (Very bad design demands the 'back' button, of course).

What I learned from eyetracking -- Seth Godin

I have a love/hate relationship with Seth Godin’s article. I love it because it is a perfect cautionary tale about why we should take the time to stop and understand data.  Quick assumptions (especially based on eye-movement recordings) can lead to some surprising conclusions.  This erroneous interpretation has gotten quite a bit of attention, and has even been mentioned in meetings I’ve had with several designers.

So just a few notes to get us started:

•    Bad web design does not encourage viewer attention. It discourages the user from making an effort to understand web content, and only succeeds in getting users lost and frustrated. 

•    Novel interface design does change looking patterns.  However, as long as a website or email is well designed and intuitive, users will learn to navigate it quickly. 

•    Individual search patterns should almost never be considered alone. The video is interesting and fun to watch, but cannot by itself give useful information about how a broad range of people view the site.

•    Individual gaze plot data is always noisy.  This is because we normally move our eye 3 times a second.  A group of gaze plots must be examined to find patterns in page viewing.

Useful References:

Poynter Studies
2005 Enquiro, Eyetools, Did-It Study -- Google

Article by Teresa Hernandez - Eyetools, Inc.

Eyetracking on blog entries (improving blog writing with eye tracking)

Eyetools Heatmap of people reading this blog
blog.eyetools.net

The "red" indicates my titles are read, but the "green" and "blue" shows that not so much of my content is read.

(How to interpret an Eyetools heatmap)

I'm a smart technical guy -- analysis and visualizations come easy to me -- and I'm even successful at public speaking, but trying to figure out HOW to write this weblog eats at me!

But I can tell you one thing... All of the blog entries earlier than this one sure aren't written the right way! I can't speak to the success of this entry you're reading right now -- you can leave a comment -- but I guarrantee you I'm going to find out in next week's eyetracking test!

So, my writing can be optimized and here's why I know...

The lack of the color red on that eyetracking heatmap over there shows that people aren't hanging on my every word, despite the fact that they should be since this eyetracking stuff is fascinating and valuable (in my unbiased opinion).

But is it my writing style or lack of value that explains the non-reading?

Well, given that I believe that knowing whether people read what you write is valuable, I'm stuck with the realization that my writing must not be all that great...

Alas.

But I'm learning. I'm going to be a great writer yet! I'm going to iterate, try different things and test the effect until I get it right. Now, in retrospect, I wish I had gotten this feedback a long time ago in college -- maybe I could have gotten a better grade on all those essays.

Blog Analysis and Optimization with Eyetracking -- or "Oh, oh, that blog's writing needs fixing"

Eyetools testing showed that the original "Navigation Bar" blog wasn't written as well as I'd thought... here's what I found, and what I did about it.

This is what happens when you have easy access to eyetracking testing: you start wondering how well you're doing writing about it... and after wondering very briefly, you test it and find that it needs improvement. Alas.

An Eyetools Heatmap of people reading one of our blogs

Our first blog: All that red at the top shows that everyone reads our title and the first 1-line paragraph, but that they start dropping off in the middle of the "story" paragraph (guess the writing wasn't as good as I thought :-).

The "statistics" section held good information, but only 60% of people read it. Can I do better?

And the "punch line" certainly is getting wasted down there (50% and mostly skimming). No big surprise. Shouldn't have been so egotistical to believe that I could actually hold people's attention ALL the way to the end, but that final point was important, so I'm moving it.

So, I'm going to tweak the text of the original blog and post it as a new one. I've still got a bunch of people who will be coming through for this study so I'll test that as well! Crossing fingers... I'm going to go write the new version now...