Reconfiguring Page Layout: SEO Split Testing Lessons from SearchPilot
- August 26, 2020
- Content
AI Summary
SearchPilot ran a controlled split test moving a flight search widget from the bottom of the hero image to the top, a change the team anticipated would have minimal impact and tested on a better safe than sorry basis. The variant produced a 7 percent drop in organic sessions, which the source estimates would have translated to a loss of roughly 10,000 organic sessions a month, so the change was never rolled out.
- The only change was the vertical position of a search widget within the hero area. No content was added or removed.
- Result: a 7 percent decline in organic sessions on the variant pages, negative and large enough to stop the rollout.
- The team's prior expectation was that the change would be neutral, which is precisely why it was worth testing.
- The source's stated lessons are that UX-owned layout decisions can move SEO performance, and that testing needs to measure traffic and conversion together.
Most published SEO case studies show you something that worked. This one shows you something that did not, which is more useful, because the changes that quietly cost you traffic are never the ones you were watching. Nobody flagged this widget move as an SEO risk. It went through the process a design improvement normally goes through, and only because someone chose to test a change they were confident about did anyone discover it was worth an estimated 10,000 organic sessions a month.
Consider how a change like this normally ships. A product team decides the search widget should be more prominent. It is an obvious improvement: the widget is the primary conversion action, moving it up puts it in front of more people, and the reasoning is sound enough that nobody argues. It goes out with eleven other changes in the next release. Organic traffic softens over the following weeks and gets attributed to seasonality, or to an algorithm update, or to nothing at all. The individual cause is never identified because it was never isolated.
The test
The change was narrow: the flight search widget was relocated from the bottom of the hero image to the top. The hypothesis was that impact would be minimal, and the source describes the motivation as a better safe than sorry approach, validating the change before implementing it.
That framing is worth pausing on, because it inverts the usual reason for running a test. Most tests are run to prove a change works. This one was run to confirm a change was harmless. The teams that catch expensive mistakes are the ones that test the second category, and the reason most organisations do not is that testing a change you are confident about feels like a waste of a testing slot.
The methodology was SearchPilot's standard controlled experiment. Comparable pages are randomly split into control and variant groups, the change ships only to the variant group, and a model forecasts what those pages would have done unchanged, using the control group as the baseline. The measured deviation from that forecast is the result. Seasonality, algorithm updates and competitor activity hit both groups equally, which is what makes the number attributable to the change rather than to the calendar.
The result
The variant produced a 7 percent drop in organic sessions, which the case study states would have translated to a whopping estimated loss of 10,000 organic sessions a month. The article does not publish a confidence interval figure, though the decision not to roll the change out indicates the result was considered reliable enough to act on.
| What the study reports | What it does not report | How to cite it responsibly |
|---|---|---|
| A 7 percent decline in organic sessions on the variant | A confidence interval or p value | Quote the 7 percent as a reported result, not as a measured effect with stated precision |
| An estimated loss of around 10,000 organic sessions a month | The absolute traffic of the site, so the base is unknown | Use the percentage when generalising. The absolute figure only describes this one site |
| That the change was widget position within the hero | Exactly what the two layouts looked like, or what was displaced | Describe the change as reported. Do not reconstruct the design from imagination |
| That the team expected minimal impact | A tested explanation for why the impact was not minimal | Any mechanism you offer is a hypothesis. Label it as one |
Reporting boundaries assessed by SEO ProCheck against the source case study.
Why it might have happened
The case study reports the outcome without establishing a cause, which is the honest position for a test of this design: a split test tells you whether an effect exists, not why. What follows is hypothesis, offered because you will need working theories to decide what to test next, and clearly labelled as such.
| Hypothesis | Reasoning | How you would check it on your own site |
|---|---|---|
| Content displacement | A widget moved to the top pushes headings and body copy further down, reducing the textual signal near the top of the page | Diff the rendered text order between versions and check whether the H1 or intro copy moved below the fold |
| Above the fold composition changed | Visitors landing from search saw a form first rather than context confirming they were in the right place | Compare bounce or engagement rate between the two groups if you have the segmentation to do it |
| Source order shifted | If the widget markup moved earlier in the document, it now precedes the primary content a crawler reads | View source on both versions and measure how many bytes precede the main content block |
| Layout shift or interaction cost | A heavy interactive component at the top of the page can worsen Largest Contentful Paint or Cumulative Layout Shift | Compare field Core Web Vitals between the groups, which is the easiest of these to test and the most often skipped |
Hypotheses are SEO ProCheck commentary. The source case study reports the negative result without establishing a mechanism.
The fourth row is the one to check first because it is cheap and it is measurable. A large interactive widget promoted to the top of the hero is a classic cause of a worsened Largest Contentful Paint, and if it is inserted after initial paint it will also generate layout shift. Our guide to diagnosing high Cumulative Layout Shift covers how to attribute a shift to a specific element, and the Core Web Vitals overview covers where the field data lives.
The lesson the source actually draws
The case study's own conclusions are organisational rather than tactical, and they are the durable part. Three points are made explicitly.
First, layout modifications traditionally owned by UX teams can significantly influence SEO performance. This is the point that changes how a company works rather than how a page looks. The backlog of changes capable of moving organic traffic is far larger than the backlog SEO teams normally review, and most of it never gets shown to them.
Second, full funnel testing matters, measuring organic traffic and conversion simultaneously, because a change that optimises the user experience may harm search visibility. The failure mode here is structural rather than anyone's fault: the conversion team measures conversion rate, the SEO team measures organic sessions, and a change that improves the first while reducing the second looks like a success to whoever presents it. Only a shared measurement catches the trade.
Third, cross-functional collaboration between SEO and product teams is increasingly critical. The practical version of that is not a sign off gate, which design teams will route around within two sprints. It is a shared definition of which templates are high risk, and an agreement that changes to those templates get tested rather than reasoned about.
Building the process this implies
The generalisable takeaway is a triage rule for layout changes. Not everything needs a split test, and pretending otherwise guarantees the process gets abandoned. Sort by traffic exposure and by how much the change moves content relative to the fold.
| Change type | Example | Risk | Recommended handling |
|---|---|---|---|
| Repositioning a major element on a high traffic template | Moving a search widget, form or primary CTA within the hero | High | Split test it. This case study is exactly this row |
| Adding a new element above existing content | A promotional banner or trust bar inserted at the top | High | Split test, and measure Core Web Vitals in both groups |
| Restyling without moving anything | Colour, typography, border radius changes | Low | Ship it. Monitor at the aggregate level only |
| Changes below the primary content | Reordering footer modules or related content blocks | Low to medium | Staged rollout with monitoring is proportionate |
| Full template redesign | Everything changes at once | High and unattributable | Split test the whole template against the old one, because you will never separate the individual effects afterwards |
Triage framework is SEO ProCheck commentary, derived from the organisational lessons stated in the source case study.
The bottom row is where most real damage occurs. Redesigns bundle dozens of changes, ship them together, and make attribution impossible afterwards. If you can only test one thing a year, test the redesign, because it is the only change where a negative result is otherwise permanently invisible. Our guide to site speed beyond Core Web Vitals covers the performance side of what redesigns tend to break.
If you cannot run split tests
Most sites lack the page volume for this methodology, which needs hundreds of comparable pages carrying real organic traffic before an effect is distinguishable from noise. That does not mean the lesson is unavailable, it means you substitute a weaker but honest alternative.
- Stage the rollout by page group. Ship the layout change to one category or one region first and hold the rest constant for four to six weeks. You get a comparison group, just an unrandomised one.
- Annotate every deploy. Most sites cannot answer what changed on the day traffic moved. A dated log of template changes is not a test, but it converts an unanswerable question into an answerable one.
- Never bundle a layout change with a content change. If both ship together and traffic moves, you have learned nothing and you will argue about it for a quarter.
- Watch impressions and average position, not clicks. Clicks move with seasonality and with SERP feature changes you do not control. Impressions respond more directly to how the page is being assessed.
- Keep the old template deployable. The value of this case study is that the change was reversible before it reached the whole site. Preserve that ability.
For a companion result where a structural change went the other way, our write up of the split test on whether unique content earns its cost covers what happens when the expected effect fails to materialise at all, which is the third and most common outcome.
FAQ
Yes. This split test moved a flight search widget from the bottom of the hero image to the top, with no change to the content, and organic sessions fell 7 percent. The team expected minimal impact. Layout determines what a visitor and a crawler encounter first, and altering that sequence changes the page even when every word on it is identical.
The case study does not establish a mechanism, so any explanation is inference. The most plausible candidates are that the widget displaced textual content that was contributing relevance signals, or that it changed what was visible above the fold in a way that altered how quickly visitors found what they wanted. Treat both as hypotheses rather than findings.
Sign off is the wrong framing, because it turns into a bottleneck that design teams route around. The workable version is that layout changes to high traffic templates get tested rather than assumed, and that organic traffic sits alongside conversion in the success criteria. This case study is the argument for that process, since the change looked harmless to everyone involved.
In this case a reported 7 percent decline in organic sessions, which the source estimates would have translated to a loss of around 10,000 organic sessions a month at that site's scale. The percentage is the transferable figure. The absolute number depends entirely on the size of the site and should not be quoted as a general expectation.
Measuring the effect of a change on organic traffic and on conversion at the same time, rather than one team measuring each in isolation. It matters because the two can move in opposite directions: a layout change that improves conversion rate can reduce the traffic arriving to convert, and neither team looking at their own metric alone would see the trade.
No, it is the cheapest possible outcome. A negative result means the change was identified as harmful on a fraction of pages and never reached the rest of the site. In this case the test prevented an estimated 10,000 monthly session loss, which is a considerably better return than most positive results deliver.
How many of your layout decisions have been tested rather than assumed?
An audit identifies the templates where a layout change carries real organic risk, and the measurement you would need to catch it.
Source: https://www.searchpilot.com/resources/case-studies/seo-split-test-lessons-reconfiguring-page-layout/
Claude Vincent is a technical SEO consultant focused on crawlability, rendering, and AI-search visibility. He writes the field guides and case studies at SEO ProCheck, with a bias toward the durable, unglamorous work that decides whether search engines and AI answer engines can actually read and cite a site.
About SEO ProCheck
Technical SEO consulting and GEO strategy with 20 years of enterprise experience. Case studies, resources, and tools for search and AI visibility.
Work With Me
Technical SEO audits, GEO strategy, site migrations, and international SEO. Hourly consulting for teams who need hands-on support, not just reports.
Subscribe to our newsletter!
