Join the Webstudio community

Updated 3 months ago

Locate the robots.txt file

At a glance

The community members are discussing how to find and edit the robots.txt file. Some key points:

- The community members currently do not have a robots.txt file, but they can set it up on Cloudflare.

- Redirects may also solve the problem of disallowing old URL paths.

- The robots.txt file is generated by the webstudio site, but there is no UI to edit it directly. The community members can follow an issue on GitHub to track this feature request.

The community members suggest a few options to manage the robots.txt file, such as serving their own file from another host or using Cloudflare to redirect the robots.txt path.

Useful resources
hey guys where do i find the robots.txt. got some old url paths i need to disallow
O
A
10 comments
currently we don't have robots.txt, so depending on what you need you can do it on cloudflare

also, wouldn't redirects settings solve you problem?
from what i understand you would need to redirect to something that makes sense. those urls are old and pointing to something i dont have. plus there are around 10k left
also i see robot.txt is in your seo details on the webstudio site. is it there just not accessible.
where do you see it on webstudio site?
that's something we generate but we don't have a UI to edit from the builder, follow this issue https://github.com/webstudio-is/webstudio/issues/4257
another option for you right now is to serve your own robots.txt and host it somewhere else
or host the site via cli yourself, then you can edit robots.txt as usual
easierst way is to sign into the cloudflare and redirect robots path to any static file hosting, even github would work fine
ok will do that. ty πŸ˜„
Add a reply
Sign up and join the conversation on Discord