I didnt have a robots.txt file in my root directory and when I run google's robot.txt analysis it gives me a 404 error which means it cant find the robots.txt file . I did use <meta name="robots" content="all"> which I thought did the same thing . Could this be one of the reason I am getting such a poor PR - the site is nearly 5 years old with over 5000 hits per month.

PageRank and robots.txt are mutually exclusive. PageRank is merely a VERY rough indicator of the quality and quantity of your inbound links, based on the same formula for them in turn.

PageRank also means very little and has been blown way out of proportion.

Let's make sure you see that again...it's important:

PageRank also means very little and has been blown way out of proportion.

It's one of over 3 digits' worth of ranking factors (personally, I believe it's between 150-200) that big G uses to analyze a site.

The biggest thing you need to concern yourself with is: "is Google sending me traffic, and if so, how much and for what?" Everything else is just steps to a greater goal.

By "hits" do you mean page views? "Hits" in and of themselves are meaningless since they can be any file on a page.

The robots tag will do the same thing, but what you can do is put a very basic robots.txt file into the root directory of your site just so SOMETHING is there.

User-agent: *
Disallow:

Just copy and paste those two lines of code, put that in the root directory of your site, and you're set.

Thanks for replying so quickly .For 80% of my keywords I dont rank in googles top 100 list .Most of the traffic comes from traditional meduims that have "highlighted" the url . Another possible reason is that I have a database running that cant be indexed by the search engines and this is where most of the content lies. Any other suggestions welcomed.

PAtrick

First, robots.txt must be saved in unix format. I think that's a cr with no lf, but your text editor will know for sure.

What is it you're trying to achieve with robots.txt, though? That file is used to tell well-behaved, polite and friendly robots not to look at some of your content. Most people have an empty file to reduce 404 traffic, but telling Google not to look at some of your pages won't help get them indexed.

Google doesn't even know you have a database, so they aren't punishing you for using one. All they care about are pages, the ones that reflect what's in the database. How are these set up? If you give us one of the URLs that might be involved, you'll get better advice.

But in general I think Google can handle two query string parameters and still index you in a normal way. More than that might land you in the supplemental index, which means (1) they don't check back very often, and (2) you only come up in searches where they don't have many other results. Both of these are very bad. Both of these are reasons people use URL rewriters.

Thanks for the advice . The url is www.staysa.co.za .It is an accommodation listing site that allows people to quickly choose what type of accommodation they want in a particular town including what sort of price they are prepared to pay . The inital posting started out as me maybe thinking that 1 of the reasons that the site wasnt in the top 100 (google) for a fairly specific keyword search was that the search engines wernt indexing the site because there wasnt a robots.txt file .YOu are now maybe giving me an idea with the url rewriting .Please feel free to add any additional comments .

Okay, I see one problem right off the bat. We'll need somebody like Adam, who's more skillful than I am, to confirm this. But let's have a look at your page title, which is also the first paragraph on your page ( after all the navigations ):

Accommodation Western Cape | B&B | BnB | Western Cape Holiday | Bed & Breakfast, Guest House | Eastern Cape | Western Cape | South Africa StaySA | Bed and Breakfast - StaySA

We see the B&B abbreviation twice in each place, and Bed and Breakfast twice, in both places. Western Cape shows up three times, per. This is known as keyword-stuffing, and Google doesn't like it. They're not punishing you for using a database, but they may well be punishing you for this.

You seem to have a little more than 2,300 pages indexed by Google, and surprisingly none of them are in the supplemental index. But looking at the SERPS I can't really tell one from the other. You may also be running up against the duplicate content filters. They don't punish you for this, but they only return one of the duplicates.

Also, "Link Partners" don't help.

Hope this helps, but I also hope we get someone more knowledgable to come in and give you a second opinion.

John, you've hit on some of the details quite nicely! The mistake that you made, and it's a common one so please don't feel bad, is that the details themselves aren't necessarily the major issue. The major issue is the big picture.

pgee: I don't know if you built this site yourself or not, but whoever did clearly had just enough knowledge of SEO to be really, really dangerous. Things such as the keyword stuffing in the title tag (by the way, you really only want 1-2 keyphrases there) and down at the bottom with all of those links doesn't help you OR your users.

To see what I mean, bookmark your home page. What happens? You see all that crap in the title tag, right? Your users will see the same thing.

Use one, MAYBE two phrases to describe what the page is about for your title and your header tag. I just realized that John saw the H1 tag whereas I saw the same text in the title.
Link partners: same thing. What good do those link partners do your users? They may be "related", but they don't seem to be useful for your users (especially anything outside of South Africa.)

The bottom links on your home page...bigtime messy. You could easily group these by the major regions of South Africa (as you've done in your map) and then get to the cities via a second page. This would be much easier from a user standpoint to grasp, and you shouldn't hurt any SE positioning (which you don't seem to have anyway.)

Basically, what you need to do is stand back and build the site for your users, and then add the SEO in without interfering with the user experience.

Doesn't make sense? Consider the things I pointed out to you above. What would happen if you managed to gain a decent SERP position for some phrase? Say that user ended up on your home page. What would happen, based on the issues I pointed out above? They'd probably leave, wouldn't they? (I know I would.)

Finally, this really doesn't affect usability that much, but it does the site owner absolutely NO good whatsoever: the designer link at the bottom right of the page. Why should they get the right to take away visitors to your site? You paid them to build it, that's the end of the story. Designers who do this should have their Internet access cut off, and I mean that.

pgee there's already a lot of good info for you above. I want to comment on the robots.txt question. Using the meta tag does have the same affect as the robots.txt file. Google is just checking your robots.txt and not looking at all your meta tags when you do that check. Also the way you're using the meta tag is meaningless. That's the default behavior. It won't make search spiders visit any more than they normally would. Generally you use robots.txt or the meta robots tag to tell them not to do something.

Originally Posted by ADAM Web Design

John, you've hit on some of the details quite nicely! The mistake that you made, and it's a common one so please don't feel bad, is that the details themselves aren't necessarily the major issue. The major issue is the big picture.

Grasshopper is learning, one day will get the stone from hand.