Home Home Redirection And Duplicate Content In Websites

Redirection And Duplicate Content In Websites

redirect-and-duplicate-seo

In this post we will see the different ways to redirect URL’s and how those techniques will effect our website’s search engine ranking. We will also have a look at the different ways of handling duplicate content in our websites and how search engines handle duplicate content if we don’t specify anything. Having a good knowledge on redirection and duplicate content handling will help you to optimize your website for search engine ranking.

What is URL Redirection?

URL redirection is sending users and search engines to another URL from the one they originally requested.

URL redirection is done due to the following reasons:

  1. Website has been moved to new domain: Suppose qscutter.com is now moved to plus.qscutter.com. Then we need to redirect all users requesting qscutter.com to plus.qscutter.com
  2. Multiple domain names: qnimate.com and www.qnimate.com point to the same website so we need to redirect all requests to qnimate.com
  3. Logging outgoing links: Instead of qnimate.com sending the visitor straight to the other sites, links on the qnimate.com can direct to a URL on the qnimate.com domain that automatically redirects to the real target. And then qnimate.com server can log outgoing link and also HTTP referer.
  4. Shortening: Sometimes we need to create short URLs for our long ones because of word limits in blogs, comments and other platforms. For example qnimate.com/this-is-my-category/this-is-my-page/ can be shortened to qnimate.com/?page=1. And when user visits qnimate.com/?page=1, the qnimate web server will redirect user to qnimate.com/this-is-my-category/this-is-my-page/.
  5. Search Engine Spamming: Redirects can be used to show different results to search engine and users. And also redirects are used to steal pagerank. Getting deep into these topics is out of scope of this post.
  6. Removing referer header: If user clicks on qscutter.com on the page of qnimate.com than browser sends the referer header assigned to qnimate.com. This header is used to log traffic sources and also used to prevent many website attacks like CSRF. This header can be removed using redirection. For example qnimate.com has a anchor link to qscutter.com. We don’t want qscutter.com to have access the referer header. So instead of sending visitors straight to qscutter.com, qnimate.com can direct the URL to itself than remove the referer header and redirect the user to qscutter.com
  7. Handling duplicate content: redirection is also used to discard multiple pages and URLs having same content. This post explains more on handling duplicate content using redirection. We will get in depth into this topic.

What are the different types of redirection?

There are many ways to redirect URLs. All these methods are categorized into two categories, HTTP redirection and HTML redirection.

HTTP Redirection

Redirection done using HTTP Location header and HTTP status code 3XX is called HTTP Redirection. There are many types of HTTP redirection.

  1. 300 Redirect or Multiple Choices
  2. 301 Redirect or permanent redirect
  3. 302 Redirect or Found or Temporary Redirect
  4. 303 Redirect or See Other
  5. 307 Redirect or Temporary Redirect
  6. 308 Redirect or Permanent Redirect
  7. HTTP refresh header

Note: Remember that all HTTP status code beginning with 3 are not redirects. For example 304, 305 and 306 status codes are not redirects.

HTML Redirection

Redirection done using HTML and Javascript code called HTML redirection. Browsers recognize HTML redirection.

There are mainly 2 types of HTML redirection:

  1. Refresh Meta tag
  2. JavaScript redirects

Search engines don’t execute HTML code, they just parse it. Therefore search engine will not recognize Javascript redirects. Search engines always parse meta tags, so search engines will know about the refresh meta tags.

301 Redirect or permanent redirect

This is used when we completely move our web pages to new domain or new URL. Search engines will never visit this URL again instead directly visit the URL specified in the Location header.

Note that when we use 301 redirect the redirection request is made GET regardless of the original request.

Let’s see an example of 301 redirect:

qscutter.com

<?php
    header('HTTP/1.1 301 Moved Permanently');
    header('Location: http://plus.qscutter.com/');
    exit();
?>

302 Redirect or Found or Temporary Redirect

This is used when we have temporarily moved our web pages to new domain or new URL. Search engines will index the new URLs temporarily and disable the original URLs. Search engine in this case will crawl both original and temporary URLs at constant interval. Once it finds the original URLs not redirecting anymore they will enable the original ones in the index and remove the temporary URLs from index.

Note that if a web page temporarily redirects for more than few days than the rank of the page in search result will start going down. That means keeping a website down for long time will decease its ranking in search engine result.

In HTTP 1.0 its called as Found and in HTTP 1.1 its called as Temporary Redirect.

All HTTP clients don’t implement 302 redirect in the same way. Some clients make the redirection request type to be same as original request type. While some clients make the redirection request type to be GET regardless of the original request type.

Let’s see an example of 302 redirect:

qnimate.com

<?php
    header('HTTP/1.1 302 Found');
    header('Location: http://labs.qnimate.com/');
    exit();
?>

qnimate.com

<?php
    header('HTTP/1.1 302 Moved Temporarily');
    header('Location: http://labs.qnimate.com/');
    exit();
?>

Many websites using HTTP 1.1 respond status as “HTTP/1.1 301 Moved Temporarily” which is actually invalid. But still browsers and search engines ignore it and take it as a temporary redirect due to massive usage of this status.

HTTP 303 Redirect and 307 Redirect

303 and 307 are also used for temporary redirection of URLs. 303 and 307 are recognized by HTTP 1.1 clients only.

302 redirect implementation differs according to client. So W3C introduced 303 and 307 to specify proper implementation.

When we send 303 status code, we are explicitly asking the client to make the redirect request as GET regardless of the type of original request.

If the original request is GET and we want redirection request to also be GET than its better to use 302 instead of 303 because many HTTP 1.1 client don’t recognize 303 status code.

qnimate.com

<?php
    header('HTTP/1.1 303 See Other');
    header('Location: http://labs.qnimate.com/');
    exit();
?>

If we want the redirection request type to be same as the original request type than we use 307 redirect. Suppose original request type is POST than redirection request type will remain as POST.

qnimate.com/search.php

<?php
    header('HTTP/1.1 307 Temporary Redirect');
    header('Location: http://qnimate.com/?search=true');
    exit();
?>

308 Redirect or Permanent Redirect

308 is used for permanent redirection. Its recognized HTTP 1.1 clients only.

When we use 301 the redirection request types is GET regardless of the original request type. But what if we want the redirect request type to remain same as original request type, in this case we use 308 redirect.

qnimate.com/search.php

<?php
    header('HTTP/1.1 308 Permanent Redirect');
    header('Location: http://qnimate.com/?search=true');
    exit();
?>

HTTP refresh header

Using Refresh HTTP header we can redirect the user to an another URL after a specified amount of time.

This is basically used when a page content is updated after every constant interval of time. So the page can load itself and make it feel like a automatic updating page.

qnimate.com

<?php
//here the time is 0 seconds. So as soon as browser encounter this HTTP response it will redirect to specified URL.
header("Refresh: 0; url=http://qscutter.com/");
?>

Refresh Meta tag

Using Refresh meta tag we can redirect the user to an another URL after a specified amount of time.

qnimate.com

<html>
<head>
<!-- here the time is 0 seconds. So as soon as browser encounter this meta tag it will redirect to specified URL. -->
<meta http-equiv="Refresh" content="0; url=http://qscutter.com/" />
</head>
<body>
<p>Please follow <a href="http://www.example.com/">this link</a>.</p>
</body>
</html>

JavaScript redirects

We can also redirect using Javascript. This kind of redirection is done usually due to some user desired action on the webpage.

qnimate.com

<script>
window.location='http://labs.qnimate.com/'
</script>

What is duplicate content in websites?

Duplicate content is when more than one URL point to the same web page. This causes problem for search engines to decide which URL to consider as the most relevant. This also causes problem to websites as the pagerank gets divided into multiple pages and therefore loose search ranking.

Few example URLs causing duplicate content

  1. Tracking URLs: Suppose qnimate.com/index.html is the home page. If you want to monitor the sources from where your users come from and how long they stay, for that you can pass GET parameters to the URL and monitor the time using AJAX. For example: qnimate.com/index.html?src=facebook. For search engine it seems like qnimate.com/index.html and qnimate.com/index.html?src=facebook are two different URL. It also sees that they both generate same content so they consider these URLs to have duplicate content. These two URLs now have different search ranking.
  2. Multiple URLs: A page can have different URLs. qnimate.com/24-august displays the post published on 24th august. And qnimate.com/24-august/18:00 displays the post published on 18:00 in the same date. And it is possible that these two URLs generate the same page.
  3. Domain names: www.qnimate.com and qnimate.com point to the same webpage. But search engines take them as two different URLs, which causes duplicate content.

Problems caused due to duplicate content

  1. Search engines don’t know which page to be displayed in search result.
  2. Search engines don’t know whether to pass all pagerank to one URL or divide among all those URLs.

Solution to duplicate content?

There are basically two solutions for handling duplicate content.

  1. Canonical link element
  2. Redirection
  3. Robots meta tag or X-Robots-Tag HTTP header field or robots.txt

Canonical link element as an solution to duplicate content

Let’s learn canonical link element implementation with an example.

Suppose qnimate.com/index.html is the home page. If you want to monitor the sources from where your users come from and how long they stay, for that you can pass GET parameters to the URL and monitor the time using AJAX. For example: qnimate.com/index.html?src=facebook. For search engine it seems like qnimate.com/index.html and qnimate.com/index.html?src=facebook are two different URL. It also sees that they both generate same content so they consider these URLs to have duplicate content. These two URLs now have different search ranking.

We can use rel=canonical meta tag to instruct search engine that they both are same and all ranking power of qnimate.com/index.html?src=facebook should be passed to qnimate.com/index.html.

Let’s the the code implementation:

qnimate.com/index.html

<!doctype html>
<html>
<head>
<title>A programming blog</title>
</head>
<body>
body content!!!!
</body>
</html>

qnimate.com/index.html?src=facebook

<!doctype html>
<html>
<head>
<title>A programming blog</title>
<link href="http://qnimate.com" rel="canonical" />
</head>
<body>
body content!!!!
</body>
</html>

When search engine will visit the second URL it will encounter rel=canonical meta tag so it will pass all ranking power of this URL to the first one. And also search engine will display first URL in search result.

HTTP Redirection as an solution to duplicate content

Permanent redirection passes all ranking power of original URL to the redirect URL. But temporary redirection doesn’t pass any ranking power.

Let’s look at an example of using HTTP permanent redirection.

http://www.qnimate.com and http://qnimate.com are two different URLs pointing to same web page. So we can redirect all requests from http://www.qnimate.com to http://qnimate.com permanently.

For example:

http://www.qnimate.com

<?php
    header('HTTP/1.1 301 Moved Temporarily');
    header('Location: http://qnimate.com/');
    exit();
?>

Robots meta tag as an solution to duplicate content

By using robots meta tag we can instruct the search engine to not crawl specific pages or to crawl specific pages. Using robots meta tag we cannot pass ranking power of one page to another.

Let’s take an example:

Suppose we have two pages with same content. Let those pages be http://qnimate.com/1.html and http://qnimate.com/2.html. We want search engine to index and display only http://qnimate.com/1.html and ignore http://qnimate.com/2.html. So we can add robots meta tag in http://qnimate.com/2.html and instruct the search engine to ignore it.

http://qnimate.com/2.html

<html>
<head>
<meta name="robots" content="noindex">
</head>
</html>

This same thing can be achieved using X-Robots-Tag HTTP header field and robots.txt file.
To learn more about these follow these links link1link2

Conclusion

In this post we saw the different redirection techniques. We saw how and where to use them for optimizing your website for better search result. We saw how redirection can be used for solving content duplicate problems. There is always more to learn. This post will be updated as new techniques arrives. Thanks for reading this. Please Like and Share.

Narayan Prusty
Displaying And Highlighting Source code in HTML PageCreating A Page Load Indicator

Leave a Reply

To create code blocks or other preformatted text, indent by four spaces:

    This will be displayed in a monospaced font. The first four
    spaces will be stripped off, but all other whitespace
    will be preserved.
    
    Markdown is turned off in code blocks:
     [This is not a link](http://example.com)

To create not a block, but an inline code span, use backticks:

Here is some inline `code`.

Narayan Prusty

I am a full-stack web developer. I specialize in Blockchain and JavaScript. This is my personal blog where I write about programming and technologies that I learn and feel interesting to share.

Image4 years ago SEO, Web Development, , , , , , ,
Share this
Related Articles