There are many files that power your website that you may not want available to the public.
Depending on how your website is built, this may or may not be an issue. We see this often with Magento in particular. For example, the /skin/ folder.
If I type in the direct path for the /skin/ folder, I do not want to see it in the web browser and I definitely do not want search engines indexing it.
The easiest way to fix this (and have control over this specific folders you want to protect) is to add the following to your .htaccess file in that specific folder.
If you do not have an .htaccess file in the folder, create a new file, name it, add this line, SAVE FILE.