The community members are discussing how to find and edit the robots.txt file. Some key points:
- The community members currently do not have a robots.txt file, but they can set it up on Cloudflare.
- Redirects may also solve the problem of disallowing old URL paths.
- The robots.txt file is generated by the webstudio site, but there is no UI to edit it directly. The community members can follow an issue on GitHub to track this feature request.
The community members suggest a few options to manage the robots.txt file, such as serving their own file from another host or using Cloudflare to redirect the robots.txt path.
from what i understand you would need to redirect to something that makes sense. those urls are old and pointing to something i dont have. plus there are around 10k left