Archive for the ‘Search Engine Optimization’

Recently Google did a post on their Webmaster Tools blog about “infinite” crawling:

To infinity and beyond? No!

This gives some background on what “infinite” crawling is and possible causes.  It also tells you how to check for messages in your Google Webmaster Tools dashboard that indicate Google has found a problem.

Wether or not Google flags an issue here, it is a good idea to examine your website for “infinite”, or “excess”, crawling issues.  By “excess” I mean the same issue of crawlers having to follow lots of different URLs that lead to the same or similar pages, but just not infinite. 

Common examples are search/sort/selection functions that generate URLs with parameters like “page=x”, “type=x”, “price=x-y”, etc, especially when links are given on the page leading to various combinations of these.  As spiders follow these, they often return very similar results, and lead to yet more permutations of URL parameters.  Spiders can spend a lot of time following these and not find any new content, and will thus stop crawling your site or ignore pages you really want them to find since they are too busy following these other pages.

To find out if you have problems with infinite or excess crawling, a good place to start is your website analytics.  Do a report on one month’s worth of traffic, sort by the URL, and look through for areas where you see lots and lots of similar URLs.  It is usually pretty easy to spot these — you will see page after page of similar URLs.  You can also use website crawling tools such as Xenu to get a list of crawlable URLs.

Fortunately, you can address issues with infinite or excess crawling.  What you want to do is identify the set of pages that have unique content or lead to unique pages (e.g., the default sort/selection for your product catalog, listings, etc), and use robots exclusion protocols to keep spiders from following and/or indexing the rest.  Thus the spiders will follow a limited set of URLs, index your good content, and not waste time on URLs that serve redundant content.

The main tools to use here are as follows:

1. Robots.txt:  With robots.txt, you can exclude entire sets of URLs with one simple command.  Used carefully, this can be safe and powerful.  For example, this command would stop spiders from following any non-default sort lists in your product catalog:
          Disallow: /mycatalog/search?*&sort=
2. Robots meta tag:  With robots meta tag, you can tell spiders not to index certain pages, or to not follow links on pages.  For example, you could use this to tell spiders not to index “next” pages in your catalog search, or to not follow links on pages that use non-default sorting, etc. 
3. rel=”nofollow” tag: Using the rel=”nofollow” tag on selected links in a page will tell the spiders to not follow those links.  You could use these on all links within a page that lead to duplicate content or excess crawling.  For example, if you have links in your catalog search page for different sort, price selection, etc options, you could use rel=”nofollow” on all links except those that lead to default settings.

By doing some trimming on what links spiders will follow, you can avoid infinite/excess crawling issues, which will focus the search engine spiders and your internal page rank on those page that have unique, useful content.  This gives your unique content a much better chance of being crawled and ranking higher.

John Erickson
LeadQual 

If you are planning to change your website domain, or change the path for file names, you will want to carefully consider the impact to your SEO rankings. The “history” and age of a page are important for SEO.  Web sites and web pages that are well established have more trust, have had time to be fully crawled and indexed, and also collect inbound links that significantly help rankings. If you change the website or page name, then you will likely take a big hit in your SEO rankings, and it will take time to recover (often lots of time!).

So does this mean you have to keep the same domain or path/page names forever? Fortunately, you don’t. There are ways to change your URLs and transfer your SEO history to the new names. The secret here is the effective use of HTTP 301 redirects.

A 301 redirect tells the search engine that a page (or website) has been permanently moved to another location. If you put in place 301 redirects for all of your old URLs to the new equivalent, it will tell search engines the old URL is being retired, and they should treat the new URL as the new name. Thus the search engines will start treating the new URL as the actual page, and work to replace instances of the old URL with the new one in their index. They will also treat links to the old URL as being links to the new one, thus preserving your hard-earned link popularity.

When you change a URL and use 301 redirects, you will typically recover most if not all of the SEO rankings you had previously, though it will still take a little time. Even with 301 redirects, it can take a few weeks for the search engine to discover all the new URLs and work them into the index, replacing the old. This could be longer for sites with low trust (new site or low page rank), or for sites with thousands of pages. If you have improved your URLs, such as a domain name that now contains keywords, or path/file names that don’t have parameters (all that ?name=value stuff), or added keywords to your path/file names, then you will likely improve rankings over the long term.

When you change URLs (domain name, path name or file name), make sure you do the following:

1. Change all instances of the old URLs in your site to the new ones. Don’t rely on the 301 redirects for your internal migration — that will look sloppy to the search engines.
2. Use an HTTP 301 redirect from all old URLs to all new ones.
3. Review your external links and ask prominent sites to use the new URL. Even though the 301 redirects will do most of the work for you, you still want to be collecting those direct links. You can find most of your external links by using the linkdomain:www.mydomain.com command at search.yahoo.com.
4. Always use the same capitalization for a URL for all references (new or old). For example, don’t use www.mydomain.com/ThisPage.html mixed with www.mydomain.com/thispage.html. To a search engine, those are two different pages.

When you do your redirects, make sure to use an HTTP 301 redirect. There are a number “wrong” types of redirects that will prevent your migration from succeeding, and you will lose all your rankings for a long time. Do not use an HTTP 302 redirect, JavaScript redirects, meta redirects or any other such technique.

If you have hundreds of URLs to migrate, you will want to automate the migration somehow. One technique is to use a 404 not-found page handler, which is a page or script that the web server will load when a page is not found. If you can do scripting within your 404 not-found page, you can look up the inbound URL in a table and find the new URL, and then issue a 301 redirect from the page script. This is easy to do with PHP, ASP, JSP and virtually any scripting language. With this approach, you just need a table of old/new URLs that can be read by the script. If you can create an algorithm for mapping old to new URLs (e.g., just replacing parts of the old URL), then the script could be coded to do that as well.

The bottom line here is that with a little work, you can migrate to a new domain or page names, and still preserve most or all of your SEO successes. You will see a dip during the migration, but with the proper techniques it should be temporary.

John Erickson
LeadQual

Search Engine Marketing

Are you running SEM in house or perhaps thinking about what it would entail? Probably a good idea to get familiar with some of the common mistakes SEM PPC mistakes small business users make. You should also be aware that any sudden spikes in impression levels, click levels or anything else for that matter should be thoroughly investigated.Is Google’s Pay Per Action in Flux? Will it Survive or become of the hundreds of Google “beta” products that slowly disappears into obscurity. Cross Tracking is also long gone.Have you been in the Google & Yahoo SEM forums? I am sure you’ve seen people talk about their Google & Yahoo reps and how they said this and that and how they will do this and that for you. Well how does one obtain a Google or Yahoo rep? Good Question. Find some tidbits here on what it takes!

Social Media & Search Engine Optimization

How about all you’ve heard about search media and the potential to use them for SEO benefits? There are so many sites out there and many will allow you to post up a naked link right on your profile!

Industry News

What is going on in Yahoo? Ever since the MSN/Yahoo debacle, many seem to be discontent with the leadership, particular shareholders (i.e. Icahn) tried to forcibly try to appoint a new board and is nearing releasing some more info on his proxy fight, and there has been mass exodus and corporate reshuffling. Now comes the partnership with Google. It will be very interesting to see how Yahoo showing Google Ads will impact the Search Engine Marketing Industry. Will the bids rise or fall? Will the click volume change in both Google & Yahoo. How will Yahoo go about sorting through what gets shown? We will see in due time. Wishing them the best in getting through this difficult time.The latest execs to leave are Stewart Butterfield & Caterina Fake the founders of Flickr which is owned by Yahoo. A creative resignation letter?

Paul Lee
Sr. Manager of Web Marketing
LeadQual

So you are sure your site could get more traffic if you could do some Search Engine Optimization (SEO) on your website, but the boss asks you to justify the project with hard numbers and provide a business case. However, it can be tough to predict how much return an SEO project will have — how do you provide solid estimates when SEO is so unpredictable?  Fortunately, there are some simple techniques you can use to get an idea of traffic improvements and create an estimate of financial return.

At a high level, you need to do four things:

1. Assess your state of your site for SEO — what needs to be improved to gain rankings,
2. Estimate how much you could increase traffic if you were to do search engine optimization on your site,
3. Estimate the value of that traffic to your business (usually a monthly run rate), and
4. Estimate the cost of the SEO project(s) needed to gain that much traffic.

With this information, you can easily construct a cost/benefit model, which hopefully will justify the SEO project (and impress the boss with your analytical skills!).

Step one is to assess your current site. If you have some experience in SEO, you may be able to do this yourself. You could also ask an SEO firm to conduct a quick, low cost analysis of your site to identify key areas that could be improved.  Look at all factors that impact SEO, including site content, website structure and inbound link popularity.  You can also look at your competitors who show well for search terms — what are they doing right and could you do the same.  You might build a high level list of issues and opportunities.

Step two is to estimate how much traffic could be increased if you optimized your site for search.  There is no one technique to do this, and it is hard to estimate with much accuracy, but here are some things to factor into an estimate:

1. What are your current search positions for target keywords?  How do they compare to your competitors?
2. What is your current traffic from organic search, and specifically for target keywords?  Look at your web analytics data.
3. Assess how bad your current site is optimized for your target keywords.
4. Compare your current site to competing sites that rank well, and assess which sites you think you could beat.  Consider factors like page rank, inbound links, age of the site, website content, internal linking and all other factors that go into good SEO.  Ask yourself which sites you think you could beat, and that gives you an idea of where you could rank.
5. Based on where you are in current rankings (which may be nowhere), and where you think you could be, estimate traffic opportunity.  If you have some baseline data, you can do a rough estimate by figuring traffic will double from page 1 to page 2 of search results, double again to bottom of page 1, and increase 10% for each position you increase on page 1 (with larger increases as you go to the top).  If you don’t have a baseline with current traffic, you may have to rely on keyword research tools to estimate overall volume.

Step three is to estimate the value of the traffic to your business.  You could draw on a number of things to create this estimate.  Here are some ideas:

1.  Look at your PPC data.  Hopefully you are tracking costs, sales, revenue and margin on your paid search (if not, you may need help with SEM services, or a new agency if they are not measuring and optimizing this!).  Your PPC performance per visitor is a good proxy for SEO performance.
2. If you are doing brand advertising online, consider the value of an impression.  Having your site show up in SEO results is a brand impression, even if only 1% click on the search result.
3. While it is hard to assign a dollar amount, there is value in displacing competitors.  If you are doing other marketing spends that target competitors, consider the value of displacing a competitor in the search results.
4. You may be in an industry where you recruit partners, employees,distributors, affiliates, etc.  Consider the value of being prominent in the search engines, and how this can greatly improve your sales/registration/signup/etc rates.  Imagine talking to a prospect, and telling them to search a high value term on Google to see your site high in the results, and asking if they want to be a part of that success.  This can work really well.

The final step is to estimate the cost of projects to optimize your website.  This will be based on what you found in step one.

With the results of these four steps, you should be able to construct a model that shows cost/benefit for your SEO projects.  Give some factors will be rough estimates, you might assign a range to certain assumptions to get “best”, “worst” and “likely” scenarios.

Armed with your data and model (and assuming it shows a positive ROI!), you can confidently go to your boss and show how your SEO project can be justified.

John Erickson
LeadQual

The title tag of a web page has a lot of leverage for SEO. All search engines, especially Google, place a lot of weight on keywords in the tittle tag. So, that means you should cram as many keywords as possible in the title, right? The answer is no — you need to limit the length of the title tag.

First a little background. The title tag for a web page specifies a short phrase that is to be used as the title for that page. This is specified using the <title> HTML tag. For example:

<title>This is the title of this page</title>

The title tag does not appear in the actual body of the web page. Most all browsers show the title in the window bar, which is the bar across the top of the window on your screen. The title tag is also shown in most search engine results, such as in the following Google result:

The title tag is very important for SEO. Since the length of the title tag is limited and it represents the content of the page, search engines place a lot of importance on this tag. Notice that words used in a search are shown in bold in the search results, helping your entry to stand out and get more clicks from users. Also, only the first 66 characters or so of the title is displayed in Google search results (Yahoo shows more — about 100), and depending on the size of the window displaying your page, the title may be truncated in the browser title bar. This is one reason you want to keep your title length limited.

The first few words of the title get the most “weight” for SEO rankings, and the more words included in the title, the less weight is given to other words (so adding too many words to a title dilutes the benefit). Thus if you want to emphasize a keyword or phrase for a page, put it first in the title tag, and limit the overall length of the title.

So how long should a title tag be, and how do you decide what to include? Here are some guidelines:

1) It is best to limit the title to 66 characters, or about 8 words. It is OK to make it longer (up to about 100 characters, or 12 words), but you will get some dilution and not all of the tag will show in search results.

2) Make the title tag on each page unique, and use keywords specific to that page. Make sure any words used in the title tag are also in the the page body.

3) Put your most important words first in the title. Don’t repeat any given word more than once (repeating too often could be tagged as spam).

4) Phrase your tag as a title if you can, not just a list of keywords. It is OK to list several short phrases and use commas, vertical bars or dashes as separators. Use proper case (where each word is capitalized). Use phrasing that is attractive to users, since the title tag is shown in search results.

5) If you want to include your brand name or domain in your title, it is usually best (from an SEO perspective) to put it last, since you probably want to emphasize other keywords by putting them first. You will usually rank well for your brand or domain anyway.

6) Avoid using special characters in the title tag if you can, since not all search engines will display them correctly. In general, ©, ® and ™ (for copyright, registered trademark and trademark symbols) will work OK, but many other symbols and special ascii characters will show funny in the results, so avoid using them.

The bottom line is to keep your title tag short (8-12 words), focus on the important keywords for the page, and make it unique for each page on your website.

John Erickson
LeadQual