|
Links.Obtanium provides a web interface to a database about files available within the
public Internet. A dedicated, automated web crawler keeps the database up-to-date.
We respect “robots.txt” which allows you to tell the search
engine to skip/ignore sites, directories or pages. We also
respect the <meta name="robots"
content="noarchive"> tag placed between the
<head></head> tags to inhibit caching. Please
note that there is no "noarchive" value in the robots.txt specifications, just "noindex" and "follow". The "noarchive" tag is supported in the HTML tag.
The current version of the spider engine is identified by the string (without quotes): "Mozilla/4.5 (compatible; Obtanium Spider 0.11; Windows)". Note that this may change in future releases.
If you are concerned about our index of your site, please send us the URL so that we can verify that the robots.txt file is working correctly. Given the potentially large volume of such notifications it processes, VISI does not respond to status inquiries made in connection with any individual notification or series of notifications.
|
