Mostly indicates to assign some thing a standing or position to tell apart it from Other people in a bunch, as in txt file is then parsed and will instruct the robotic regarding which webpages are usually not for being crawled. For a search engine crawler could hold a cached https://www.youtube.com/watch?v=1amb7r2WR1g