[Resolved] [Resolved] [Resolved] [Resolved] [Resolved] [Resolved] [Resolved] [Resolved] [Resolved] [

[Paisano]

Hello there. I just wanted to get some views and oppinions about the fine art of search engine submissions. What do you think is the best way to promote your own site or a client's site that you've developed?

There are two schools of thought floating around:

Do It Yourself Method:
You handpick those search engines you want to be listed on and submit your site one at a time.

The ShotGun Method:
You use a third party service to submit your site to hundreds or even thousands of search engines.

A variation on this theme is using a software package such as webposition gold to submit your site(s) yourself to hundreds or thousands of search engines. It's still a shotgun approach.

Pro's and cons?
For quality rankings, the manual approach seems best,
But who has the time?

What do you guys/gals think?

[vsecades]

Paisano,

This is a touchy subject. I would you go with the webposition gold software, or a viable alternative for the following reasons...

1-Manual tracking of your info on several search engines is boring and long. That's what the software is for.

2-You can submit your website with the software, saving yourself a lot of time.

3-Search engines modify their algorithms just slightly, and tax you on lower ratings if you mess up every once in a while

So the manual version is totally out of the question. Just thought I'ld let you know.

[l0ungeb0y]

Well, if you are serious about search engine submission
Then the items brought up are the last things to be considered... so I'll touch on those last.

First and most important: The pages themselves.
Start with the header. Do you have a good title that contains a few keywords that concisely describe the page.
And don't hesisitate to be lengthy, you have quite a bit of room for a title.

Keywords, back to web design 101... do you have plenty of them and are they relevent?
Page description... gotta have a good page description that nicely mirrors the actual body contents and the keywords as well.

My favorite and the most overlooked meta tag is ROBOTS
<META NAME="robots" CONTENT="all | none | index | noindex | follow | nofollow">

a good use for the sites homepage is:
<META NAME="robots" CONTENT="index, follow">
...that clearly informs the spider that it can index this page and follow all it's links.
This is redundant if you use a robots.txt file in your $DOC_ROOT, but I feel it's best to be as robust as possible since some spiders such as those from the National Directory Index (NationalDirectory-SuperSpider/1.2) do not read from robots.txt (atleast acording to my database, which logs each spider that reads from robots.txt)

There are many spiders out there on the web that will visit your site. Here is a quick sample from my database of those who have visited mine:

| Lycos_Spider_(T-Rex)
|
| Mercator-Scrub-1.1
|
| Mozilla/2.0 (compatible; EZResult -- Internet Search Engine acurtis@directhit.com)
|
| Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
|
| Mozilla/4.0 (compatible; MuscatFerret/2.0; http://www.webtop.com/)
|
| Mozilla/4.0 (www.first-search.com)
|
| NationalDirectory-SuperSpider/1.2
|
| ru-robot/0.1 [hseo@cs.rutgers.edu]
|
| Scooter-Jellyfish1
|
| Scooter/2.0 G.R.A.B. V1.1.0
|
| Unlost Web Crawler 2.0.1.6
|
| VILLspider betaversion 0.2
|
| WebReaper - http://www.otway.com/webreaper
|
| WebSnarf/0.0
|
| ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com


I can go on and on about webspiders because I've researched them extensively (via scripts and tricks and talking to those who write some of the interesting ones like the Alexa crawler) as well as writing my own.

For more info on controlling how webspiders use your site visit this link: http://www.w3.org/Search/9605-Indexi.../Spidering.txt


Now onto the page itself.
The first consideration for a page is whether r not a spider can actively follow links. This may sound like a stupid question, but many sites that use Flash forget to include a means to support spidering by link following.
Since a spider cannot understand Javascript nor can it understand Flash, a Flash or DHTML site will have to take this into consideration. Advanced server-side programming can help you create pages for spiders and lowend browsers that can provide an alternitive means of viewing your site.

Also, note that http://www.google.com will cache your pages.
Just do a search and you'll see each result has a 'cached' link, so keep this in mind.

After the links, what about the text?
Can the spider read it or is it written by DHTML or contained in Flash or Java applets?
Also, how is the text formatted and set up? Do you use Headers (<H1-5>) effectively? Spiders like headers, as they interpret a header to be just that, a title to a section of a page, preferably a following paragraph denoted by a <p>. Do you use <p>,<dd> and other text formatting/description attributes well? And does the body contain a good proportion of keywords and words that appear in the description and title?

These are all very important as to how well your page will be indexed.

Also, how well is your filesystem structured?
Do you create a new directory for each new page or do you let all pages sit in one directory?

Optimally, you should create a new directory for each page and entitle it index.html.
Certain engines such as Lycos and Infoseek will view each index.html as a new directory index and index it in their database accordingly (counting as a brand new site), and add to your homepages ranking by link relevance (the more indexes that link to your home page the better)

There's a lot to be considered here.
So, I suggest you visit this page:
http://www.w3.org/TR/1999/REC-html40...otes.html#recs
and read up on other considerations as well.

Then I suggest you do submissions to all the major engines and check back every couple weeks and make some minor modifications to your META-TAGS as needed and resubmit as applicable.

And yes, having a means of tracking webspider site traversal is very important. I roll my own, but the one listed in the previous post above might be of use.

So get reading and good luck.
[Edited by l0ungeb0y on 07-14-2002 at 03:50 PM]

[OddDog]

now that is the longest post i have ever seen !!!!!!

[l0ungeb0y]

Originally posted by OddDog
now that is the longest post i have ever seen !!!!!!

Ya, I just chopped it down some by removing some of the database dump I inserted.

It is a very imformative post though, hence the length as there is much to creating pages for optimal performance with spiders in mind.

...also fixed some of the links that were mangled by the UBB auto-linking for the webspiders. I suggest visiting the spiders homepage to gain more insight into the different spiders. The inktomi bot home page is one I recommend highly as the slurpbot is used by engines such as google.

[l0ungeb0y]

another good link is the registery of internet robots:
http://www.robotstxt.org/wc/active/html/index.html

[Paisano]

Thanks so much for an excellent and informative post!
I've taken the liberty of copying and pasting it to a text file for many future references. I've learned a lot from it already! I appreciate the info! I hope I can return the favor someday!

Originally posted by l0ungeb0y
Well, if you are serious about search engine submission
Then the items brought up are the last things to be considered... so I'll touch on those last.

First and most important: The pages themselves.
Start with the header. Do you have a good title that contains a few keywords that concisely describe the page.
And don't hesisitate to be lengthy, you have quite a bit of room for a title.

Keywords, back to web design 101... do you have plenty of them and are they relevent?
Page description... gotta have a good page description that nicely mirrors the actual body contents and the keywords as well.

My favorite and the most overlooked meta tag is ROBOTS
<META NAME="robots" CONTENT="all | none | index | noindex | follow | nofollow">

a good use for the sites homepage is:
<META NAME="robots" CONTENT="index, follow">
...that clearly informs the spider that it can index this page and follow all it's links.
This is redundant if you use a robots.txt file in your $DOC_ROOT, but I feel it's best to be as robust as possible since some spiders such as those from the National Directory Index (NationalDirectory-SuperSpider/1.2) do not read from robots.txt (atleast acording to my database, which logs each spider that reads from robots.txt)

There are many spiders out there on the web that will visit your site. Here is a quick sample from my database of those who have visited mine:

| Lycos_Spider_(T-Rex)
|
| Mercator-Scrub-1.1
|
| Mozilla/2.0 (compatible; EZResult -- Internet Search Engine acurtis@directhit.com)
|
| Mozilla/3.0 (Slurp/si; slurp@inktomi.com; http://www.inktomi.com/slurp.html)
|
| Mozilla/4.0 (compatible; MuscatFerret/2.0; http://www.webtop.com/)
|
| Mozilla/4.0 (www.first-search.com)
|
| NationalDirectory-SuperSpider/1.2
|
| ru-robot/0.1 [hseo@cs.rutgers.edu]
|
| Scooter-Jellyfish1
|
| Scooter/2.0 G.R.A.B. V1.1.0
|
| Unlost Web Crawler 2.0.1.6
|
| VILLspider betaversion 0.2
|
| WebReaper - http://www.otway.com/webreaper
|
| WebSnarf/0.0
|
| ZyBorg/1.0 (ZyBorg@WISEnutbot.com; http://www.WISEnutbot.com


I can go on and on about webspiders because I've researched them extensively (via scripts and tricks and talking to those who write some of the interesting ones like the Alexa crawler) as well as writing my own.

For more info on controlling how webspiders use your site visit this link: http://www.w3.org/Search/9605-Indexi.../Spidering.txt


Now onto the page itself.
The first consideration for a page is whether r not a spider can actively follow links. This may sound like a stupid question, but many sites that use Flash forget to include a means to support spidering by link following.
Since a spider cannot understand Javascript nor can it understand Flash, a Flash or DHTML site will have to take this into consideration. Advanced server-side programming can help you create pages for spiders and lowend browsers that can provide an alternitive means of viewing your site.

Also, note that http://www.google.com will cache your pages.
Just do a search and you'll see each result has a 'cached' link, so keep this in mind.

After the links, what about the text?
Can the spider read it or is it written by DHTML or contained in Flash or Java applets?
Also, how is the text formatted and set up? Do you use Headers (<H1-5>) effectively? Spiders like headers, as they interpret a header to be just that, a title to a section of a page, preferably a following paragraph denoted by a <p>. Do you use <p>,<dd> and other text formatting/description attributes well? And does the body contain a good proportion of keywords and words that appear in the description and title?

These are all very important as to how well your page will be indexed.

Also, how well is your filesystem structured?
Do you create a new directory for each new page or do you let all pages sit in one directory?

Optimally, you should create a new directory for each page and entitle it index.html.
Certain engines such as Lycos and Infoseek will view each index.html as a new directory index and index it in their database accordingly (counting as a brand new site), and add to your homepages ranking by link relevance (the more indexes that link to your home page the better)

There's a lot to be considered here.
So, I suggest you visit this page:
http://www.w3.org/TR/1999/REC-html40...otes.html#recs
and read up on other considerations as well.

Then I suggest you do submissions to all the major engines and check back every couple weeks and make some minor modifications to your META-TAGS as needed and resubmit as applicable.

And yes, having a means of tracking webspider site traversal is very important. I roll my own, but the one listed in the previous post above might be of use.

So get reading and good luck.
[Edited by l0ungeb0y on 07-14-2002 at 03:50 PM]

[RazoRmedia]

The manual approach is always the best way as search engines do not like automated submissions but it can be very time consuming.

You cannot beat web position gold for its reporting capabilities and its submission procedures although I found it too be much better if I submitted manually too.

The main thing to remember is that you should alway think about search engine optimisation when your designing and developing a site as the main thing that will increase your rankings is content.

It is a broad misconception that meta tags are the be all and end all of search engines, they're not, concentrate on content and link popularity and you can't go far wrong.

A good site is http://www.spider-food.net

Good luck

[hurricaneone]

As much as it grinds my teeth up, that's an excellent post l0ungeb0y. You factor in Razor's addition and there's some top quality info there.

[serpent star]

All Very good stuff.

Loungeboy, I would love to pick your brain for a couple of days, it's obvious to me that you are extremely knowledgable and are a true asset to the Flashkit community. (Really, I'm not just sucking up either.)

Anyway, I just thought I'd share my ace in the hole with you all.
Very good info and lots to be learned from this forum. Search Engine Submission and Optimaization can be a full time opportunity in and of itself.

Hope y'all learn a thing or two from the link provided.

http://www.webmasterworld.com/

Check out the search engine world forum as well as the SEO forum.

[pheck]

great info, thanks guys. myFK'd.

[oftenusa]

I recently tried to do manual submissions to search engines and had a very frustrating experience. I read something that said that a vast majority of internet users find websites through only 10 or so search engines, which makes sense. When I tried to go to those search engines I found they cost money or don't even take submissions! I found a lot of search engines get their content from a handful of distributors. In all cases I tried to submit to the search engine rather than the content provider but this didn't happen very often.

Here is a list of the ones I successfully submitted to:
Google
Lycos
Yahoo
Altavista

AOLsearch - via Open Directory (dmoz.org)
About - I had to send an email to the guide
Ah-ha

Search.com/Snap
AllTheWeb
National Directory

I'm interested if other people have had the same experience and if there is something I'm missing! My feeling after the experience was pretty close to what people have said already... Build your site to be favorable to search engines, get other websites to link to yours and let the content do the work in spreading the word. Sound good?!

[l0ungeb0y]

Originally posted by oftenusa
My feeling after the experience was pretty close to what people have said already... Build your site to be favorable to search engines, get other websites to link to yours and let the content do the work in spreading the word. Sound good?!

Sounds good to me.

And I'm glad to see that people got something usefull out of my earlier post.

[angiedarrah]

I am so excited this post is avilable for total newies in the search engine submission world. But I must admit, it is all a little overwhelming. Could anyone recommend a good place to start learning? (Other than this great post, of course.) Are there any links designed for beginners?

I had no idea there was so much to all this.

Thanks to everyone for making this info available.

[darkdog9]

again i ask, WHAT WOULD I DO WITHOUT FK? this is something that i have kind of avoided really looking into cause it seems like a pain in the neck. the sites i have done have had a built in audience who were referred to the site via an outside source. that is not the case with 3 of my current projects. this will be extremely helpful. thanks to all, again.

[Paisano]

I feel your pain. It is a lot more complicated that most people will ever imagine, but that's all right. It's what separates the men from the boys (the women from the girls?)!
Those that are good at this fine art, will reap many rewards from it personally and professionally.

The best part of it all is that you can never master it...much like the game of golf (unless your name happens to be Tiger Woods, of course)! You can always improve your skills in this very mystical and obscure and ever-evolving and perpetually changing artform.

The best place that I've ever found regarding search engines is http://www.spider-food.net, which was mentioned earlier in this thread too. I would spend lots of time there. http://www.searchengines.com is good too.

good luck and please share any tricks and tips you encounter...ok?

Originally posted by angiedarrah
I am so excited this post is avilable for total newies in the search engine submission world. But I must admit, it is all a little overwhelming. Could anyone recommend a good place to start learning? (Other than this great post, of course.) Are there any links designed for beginners?

I had no idea there was so much to all this.

Thanks to everyone for making this info available.

[Paisano]

I am so glad that I decided to make this post. I hesitated at first because I was kind of embarrassed that I didn't know more about what seems to be a very simplistic concept...search engines! I mean, everyone knows how yahoo, google and the like work...what's the big deal, right?
Well, it is a big deal when it comes to search engine rankings and quality of placement! I know that I hardly ever go to a second or third page of search results!
What are the odds of others doing so? Thus, my strong curiosity with search engine submissions. I knew that there was much more to it than most of us think. The more I learned about it, the more confused and perplexed I became!
It was the same feeling I got when I first became interested in girls and learned how beguiling and complex women were! Mysteries wrapped up in a riddle!

Anyway, glad you like this thread. I hope you will contribute to the knowledgebase with your findings.
Good luck!

Originally posted by darkdog9
again i ask, WHAT WOULD I DO WITHOUT FK? this is something that i have kind of avoided really looking into cause it seems like a pain in the neck. the sites i have done have had a built in audience who were referred to the site via an outside source. that is not the case with 3 of my current projects. this will be extremely helpful. thanks to all, again.

[benwalsh]

Does flash sites and search engines go together
I love flash and like to create my sites solely in this program how do I go getting listed on search engines

[gilmore1974]

What about data driven web sites? I was curious as to whether or not spiders trawl through databases as well as static content. Depending on the job, you may wish your site to be fully data driven in order to give your client control of their data and content. This obviously means that the web pages will be constructed on the server side, so how would this content be read by a spider?

[RazoRmedia]

Originally posted by gilmore1974
What about data driven web sites? I was curious as to whether or not spiders trawl through databases as well as static content. Depending on the job, you may wish your site to be fully data driven in order to give your client control of their data and content. This obviously means that the web pages will be constructed on the server side, so how would this content be read by a spider?

spiders will not follow anything after a '?' in a URL (most of the time);

therefore

http://www.whatever.com/default.php

is fine whereas

http://www.whatever.com/default.php?id=6&name=bruce

won't be followed by a spider