Google recently announced a change in the format of the referring URL for visitors finding your site through keyword searches.
The new format promises to reveal much more information about the search result rankings that was previously available, and this post will show you how to take advantage of that information to tune your SEO efforts using Google Analytics.
Starting this week, you may start seeing a new referring URL format for visitors coming from Google search result pages. Up to now, the usual referrer for clicks on search results for the term “flowers”, for example, would be something like this:
http://www.google.com/search?hl=en&q=flowers&btnG=Google+SearchNow you will start seeing some referrer strings that look like this:
http://www.google.com/url?sa=t&source=web&ct=res&cd=7&url=http%3A%2F%2Fwww.example.com%2Fmypage.htm&ei=0SjdSa-1N5O8M_qW8dQN&rct=j&q=flowers&usg=AFQjCNHJXSUh7Vw7oubPaO3tZOzz-F-u_w&sig2=X8uCFh6IoPtnwmvGMULQfw
What does that mean for me?
Common speculation is that the cd parameter in the referral URL indicates the position of your page in the search results, and this has been confirmed by Matt Cutts.
Knowing what position your pages hold in the SERPS for a given keyword can be enormously helpful to your SEO efforts, so it makes sense to start tracking this new data. Prior to the change in referer, it was possible to track your SEO rankings with Google Analytics, but you could only get information about the page on which you were ranking for a keyword search, not your overall position.
Create a New Profile
You are definitely going to want to create a new profile within Google Analytics so that you do not interfere with the data that is already being collected for your site (The profile that we are about to set up will only capture data for organic search results from Google that use the new referer string).
Sign in to your Google Analytics account and “Add a Profile for an Existing domain”. You can give it any name you like, I chose to use “Google SERP Ranking”.
Create the Filters
The first thing that we need to do is create a couple of filters to make sure that only organic google searches are included in this profile. Order is important, so make sure that you either create your filters in the order that they are described here, or that you go back when you are finished and reorder them appropriately.
1. Include Organic Search Results
Create your first custom include filter using the Campaign Medium filter field and a filter pattern of organic.
2. Include Google Search Results
Create another custom include filter using the Campaign Source filter field and a filter pattern of google.
3. Include New Google Referral URL
Create the final custom include filter using the Referral filter field and a filter pattern of google.com/(search|url).*\bcd=\d*.
4. Extract Keyword and Ranking Data
The last filter we need to create is an advanced filter that will extract data from the referal string and create custom output that will be viewable in our reports. Create an advanced custom filter, and set the pulldowns, text areas and radio buttons as shown in the screenshot. This will extract the keywords and SERP position from the referal string and replace the output with our own custom string in the format <keyword> (Rank: <position>)
For clarity, here are the three regular expressions from the previous screenshot:
\bq=([^&]*) \bcd=(\d*) $A1 (Rank: $B1)
Visualizing Results
Once you have created the new profile and filters, and have started receiving traffic, you should be able to see the keyword/ranking data in any of your reports when you change the dimension to User Defined Value.
Please share your experiences with this or any other helpful Google Analytics tricks that you use!
Update: You can also replace the %20 with a space in your reports by adding some additional filters
Thank you Chris!
I followed this tutorial step-by-step and we have received our first day of keywords with their respective ranks. I forgot to set up goals for the new profile, so I guess we’ll have to wait until tomorrow if any of these keywords are creating successful goal conversions.
Great Brian, I’m glad you found it helpful!
Awesome post. help me a lot. :) cheers.
Hi Chris,
I had set up these series of filters for the Newfangled website. It is listing out the keyword rankings perfectly. However, I set up several goals which I had duplicated from another profile which listed all organic traffic from Google. The organic traffic profile showed several goal conversions whereas the profile set up to show keyword ranking registered zero goal conversions.
After further analysis, I noticed that the keyword ranking profile does not show pages after the landing page, unless the landing page was refreshed. Meaning, the pages/visits is hovering around 1 page per 1 visit.
If the above is the case, then it explains why no goal conversions were being registered.
Have you heard of anybody else encountering this issue?
(I might just be completely missing something though)
Hi Brian, my initial thoughts are that the landing page is the only one included in this profile because one of the filtering requirements is the presence of the google-specific string in the referring URL (which is only present with the first page visit).
I’d be very interested to hear if you are able to work around this somehow, as being able to track conversions based on the keywords and SERP position would be very handy.
Hi Chris,
Thanks for the quick response. Phew. So it wasn’t me just thinking I was going crazy. I’ll let you know what I find as well as confirming step #3 being the filter that is only including the landing pages in the profiles.
Thanks again for your help!
Chris,
I removed filter #3, and the profile now tracks pages deeper than just the landing pages. Some keywords and their respective ranks are also being tracked, but the majority of the organic traffic is encompassed under one keyword (not set). I hope this confirmation helps you in some way.
Great idea! I am implementing this, but apparently had a few typos in my filters because the screen shot was a bit hard to read.
Would it be possible for you to post the regular expressions (for the filters) in plain text so that we could simply copy and paste them?
Thanks again.
Barbara
@Chris: Thanks for taking the time to do this tutorial. My question is–does GA aggregate the “average” click position with keywords which resulted in multiple clicks?
I’m going to test this on a couple of sites tomorrow. As you know, ranking reports are way dead and SEOs have been measuring success by looking at monthly average search frequency (Google AdWords API) & traffic to site.
This allows marketers to determine what “percentage of search market share” the site is achieving. It gets more granular when you mash in data from the Trellian API to improve one’s read on “inventory,” in the form of how many global searches are levied.
With a little work, adjustments can be made for seasonality as well. Anyway, thanks for the post and pleased to meet you. Cheers.
In comparing my organic Google traffic in one of my main profiles vs. what I’m seeing in this new custom profile, I’m seeing only a fraction in the latter vs the former.
Is this because of the third “include” filter? Only some of the referral URLs from Google’s organic results contain the cd= parameter, and we’re only seeing those in the new profile?
I did double-check my transcription of the regex’s from the screenshots.
Marc, you’re right about the ‘cd’ parameter. Google has not completely rolled out the new referral URL format yet, so only some of the referrals from Google SERPs will be using the new format.
Great article-i found this really useful. A person needs to be aware of these changes on google.
Hi Cris,
I have doubled checked my profile configuration, but all I get is a blank page statement like this: keyword(page:)
Any thoughts?
Thx Chris but when I drill down to user defined value for traffic sources>search engines and I got a (not set)result. So for its been about a day and I got less than ten visits. I followed the directions to a tee but should I make any adjustments?-Eddie
HI
fantastic.. if this can do what i think, then there is no need to pay any of the expensive software tools as IBP.. do you agree?
can you please type in the text for filter 4 in plain text so we can type it in?
i cant see some of them on my screen! :(
thanks again!
Step 3 is including information from the organic results from google.com. Has anyone had experienced with implementing this for other locations as well (ie google.fr, google.de)–maybe like this:
google.(com|de|fr|co.uk)/(search|url).*\bcd=\d*
or maybe this would adulterate the results by mixing positions from different data-centers. Any thoughts?
Thanks for this article. Please let me know if you figure out the Google Base/Shopping test from the post above.
Great article. This will help us evaluate keywords so much!
Great post! I’m a little late to the party. Just so I’m clear, the order of filters are:
1. Include Organic SERPs
2. Include Google SERPs
3. New Google Referral URL
4. Construct Ranking String
Thank you! Just when I think I know what I’m doing in analytics, I find something like this that takes it to a whole new level! Thank you for such clear instructions and photos. I can’t wait to start seeing the traffic!!!
Now, if only what I write about (real estate appraisal, granite) were a little bit more interesting!!! :-)
This is a great post. I had just previously customized my page position script variables so they read cd= and q= rather than the original q= and start=. Going to try what you are suggesting.
What I would really like to do is also capture the source= as well which captures whether the click is coming from web, news, images, video, or maps. Anyone capturing this source along with keyword and position?
Hi there, many thanks for this great tip! I have setup a new profile and filters as you’ve said, but I’m not receiving any traffic on them? The main website profile is receiving traffic, my keyword ranking profile shows the clock (tracking installed) and says ‘Waiting for data’ – it’s been like this for a week. Given the main profile is receiving data, what could the problem be?
AHA, Ben
Thanks for the tutorial Chris. I also echo SwitzerBaden and other’s comments – it would be great to further segment the ranking data into google TLD’s (.co.uk, .fr) and also the verticals (images, news, maps). Would you recomend setting up separate profiles or can it be done by amending the filters? Once I gather some data and experiment I will comment back, but would be interested to hear from anyone who had tried this already (@SwitzerBaden, @StephenRacano?)
I just spent an hour building a script using PHP and cURL to find my position in SERPS for various terms, then I found your post! Mine took a lot longer and is less accurate… Thanks for your help!
Chris, this is very helpful. Finally a keyword tracking system that is free as a service and it’s made by google.
Hope I’ll get this right on my google analytics account for my company website.
Keep writing cause you got great stuff here!
Awesome stuff Chris, thanks for the tip, should come in pretty handy!
Hi Chris
Thanks a lot for this, it is very helpful & insightful.
However for the 20% thing, couldn’t remove it no matter what i did. Even the alt thing didn’t work.
Kept on trying until i found this article http://seoontap.com/analytics/serp-report-google-analytics.html, which does what you posted here but for pages not for ranks & for more than one search engine.
The part I found useful was the extract of the keyword.
No need to extract the keyword after the q & then spend lots of time formatting it.
You can simple choose the Field A-> Campaign Term and enter (.*)
GA is obviously saving a clean version of the term over there.
Again thanks for a great post
Awesome stuff Chris, thank you so much for the important tip, please post other tips also
Great tips about keyword ranking tool and Google analytics. These are very important tools regarding search engine optimization. Thanks for the tips
Hi Chris. These are great news, well, at least to me. If i search on google thru firefox and chrome, i get different URLs. Did they change the parameters? Does this solution work today? Would love to read from you.
Hi
I am using your definitions to see keyword ranking in google analytics. But I also want to compare the keyword ranking according to dates. For example, I want to compare where I rank for “hotel” word on Nov 6 and Nov 30. When I apply your filtering and choose two dates, I get results like,
hotel – position 6 – 5 visits nov 6
hotel – position 8 – 1 visit nov 6
how can I fix this?
thanks
Nazli
Hello, I just set up the filters step by step and put the filters in this order but I am not getting anything.. my main account shows traffic for the last few days but this new filter shows 0. What have I done wrong? Does this still work with the new version of analytics?
Thank you Chris for a very helpful piece.
Hi! I know this is kind of off topic but I was wondering if you
knew where I could find a captcha plugin for my comment form?
I’m using the same blog platform as yours and I’m having trouble finding one?
Thanks a lot!