There are now java utilities on the Internet to help create site maps, and the output will be an .xml file. This can then be given to Google for them to scan your site. While the XML is useful, they are not the easiest files to read, so there is another way and other search engines like this .txt file as well.

The other way which is quite easy to do is to create a text file in DOS using the DIR command:

dir /b/s >c:sitemap.txt

what you need to do is create a directory structure on your local PC exactly the same as on the web server.

Once you have created the directories are correct then run the above command from the root of that directory. This will create a local directory of all the files within the directory and subdirectories. The internet uses different slashes from the ones used locally, but this is easily remedied.

Once youve created the file its just a matter of doing a global replace of c: in the text file for your URL plus changing the slashes to the / slash. This can be done in notepad.

It is very easy.

The same command was run on http://www.town-directory.net which has thousands of pages.