Robots.txt is a text file website owners create to give instructions about which URLs the crawler can access on your site.

[{"selector":"#anim-47051728-850e-469f-8068-f267ba20a754","keyframes":{"transform":["translate3d(-111.74603%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-0d89d6ef-cc98-4456-970c-aae939c598cc","keyframes":{"transform":["rotateZ(-180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}] [{"selector":"#anim-11774554-5cc9-4fa6-bd7a-bb7e4456386b","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-d134f13b-1e0f-4cf4-9295-dac506450cb4","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-5c3f027e-1bea-4aa6-9fb1-84830d295669 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-31.28680067153245%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}]

Learn Important facts About Robots.txt

[{"selector":"#anim-663ac98e-eccf-430d-83a6-4f77eb898e1e","keyframes":{"transform":["translate3d(-114.24333%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-788094ed-36e9-43c6-89cf-770305a3e584","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-10df7479-5470-41e1-81a1-34957ed4404c","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}] [{"selector":"#anim-b387fab6-2c79-4abf-ba80-8c0f4ccf1da2 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-31.285244247341844%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-8ae58200-a45d-45de-917b-89362035fce4","keyframes":{"transform":["translate3d(-601.56251%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-40bd6e5d-abee-4c62-b463-4b77c1e6a5f5","keyframes":{"opacity":[0,1]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"both"}] [{"selector":"#anim-64c6b8b4-3a5a-47ac-bf9b-c1665d5938c5","keyframes":{"transform":["scale(0.15)","scale(1)"]},"delay":0,"duration":600,"easing":"cubic-bezier(0.4, 0.4, 0.0, 1)","fill":"forwards"}]

Webmasters use robots.txt to manage the crawling behavior of search engine

[{"selector":"#anim-67c2786f-296b-4a56-b0b2-ead307d0bcac [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-14.843749786673241%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-23b7061f-d789-4c5e-95df-8c192b3e6e78","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-e5510fc3-3034-47c6-8696-345751edf4e2","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

"robot.txt" file is placed at the root directory of your websiteExample: www.example.com/robots.txt

[{"selector":"#anim-68de3492-fe8c-40c7-a104-ba7571450113 [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-14.843749786673241%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-4f7a79d0-d0cf-4f88-a7b8-f6d4e25dc0cc","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-942b379e-62f2-46b3-87aa-ff5d310fd787","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

Rbots.txt file also helping in SEO and give you controlling on your website

[{"selector":"#anim-52b3a62f-3d07-4006-b727-d89bc88b827f [data-leaf-element=\"true\"]","keyframes":{"transform":["translate3d(-14.843749786673241%, 0, 0) translate(-25%, 0%) scale(1.5)","translate3d(0%, 0, 0) translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"fill":"forwards"}] [{"selector":"#anim-36ba9b53-5c6f-4a11-9732-0f80f055bca3","keyframes":{"opacity":[1,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-c8ca802d-ed40-47df-abdc-c0331794c16a","keyframes":{"transform":["scale(3)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}]

NEXT⏭

[{"selector":"#anim-43b2a3c5-f2c3-4790-b723-7d41ffb074a6","keyframes":{"transform":["translate3d(-125.53192%, 0px, 0)","translate3d(0px, 0px, 0)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .8, 1)","fill":"both"}] [{"selector":"#anim-d23858c2-84aa-4622-9ce4-d46a8ea0307e","keyframes":{"transform":["rotateZ(-180deg)","rotateZ(0deg)"]},"delay":0,"duration":1000,"easing":"cubic-bezier(.2, 0, .5, 1)","fill":"forwards"}] [{"selector":"#anim-18c475ca-faf1-40e8-85be-ef5d717d270b [data-leaf-element=\"true\"]","keyframes":{"transform":["translate(0%, 0%) scale(1.5)","translate(0%, 0%) scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] [{"selector":"#anim-8f0dd01a-469e-4eae-9034-a801994610ab","keyframes":{"opacity":[0,1]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"both"}] [{"selector":"#anim-32b7caf9-c89b-4473-ba87-b9a2642896a0","keyframes":{"transform":["scale(0.3333333333333333)","scale(1)"]},"delay":0,"duration":2000,"easing":"cubic-bezier(.3,0,.55,1)","fill":"forwards"}] Learn more

Robots.txt is a text file website owners create to give instructions about which URLs the crawler can access on your site.

What is robots.txt File?

Learn Important facts About Robots.txt

Webmasters use robots.txt to manage the crawling behavior of search engine

Fact 1

"robot.txt" file is placed at the root directory of your websiteExample: www.example.com/robots.txt

Fact 2

Rbots.txt file also helping in SEO and give you controlling on your website

Fact 3

NEXT⏭