|
Txt. Directives no longer supporte below are some commands that google no longer supports - partly for technical reasons they have never been supporte. Crawl-delay directive previously you could use this directive to specify the fetch interval in seconds. For example. crawl. Then you nee to set the crawl-delay directive to 5: user-agent: googlebot crawl-delay: 5 google no longer supports this command. But bing and yandex still support it. That being said. You nee to be careful when setting this directive. Especially if you have a large website. If you set the crawl-delay directive to 5. The spider can only crawl a maximum of 17.
280 urls per day. If you have millions of pages. This crawl volume is very small. On the other France WhatsApp Number Data hand. If you have a small website. It can help you save bandwidth. Noindex directive this directive has never been supporte by google. But until recently. It was thought that google had some "Code for handling unsupporte and unpublishe rules (such as noindex). " so if you wish to prevent google from indexing all your blog pages. Then you can use this directive: user-agent: googlebot noindex: /blog/ but at the same time. On september 1. 2019. Google made it clear that this directive is not supporte . If you want to exclude a page from search engines.
Use the meta robots tag. Or the x-robots http header directive. Nofollow directive this command has never been officially supporte by google. It use to be use to prevent search engines from following a certain link or a special path. For example. If you want to block google from following all blog links. You can set the command like this: user-agent: googlebot nofollow: /blog/ google state on september 1. 2019 that this directive would not be supporte. If you want to prevent search engines from following all links on a page. Then you should use the meta robots tag. Or the x-robots http header directive. If you want to specify that a link should not be followe by google.
| 
發表於 17:46:30