In this help article, we will guide you through the process of using the website crawler feature in Ask71 to import content from your website.

Log in to your Ask71 account and go to the Content Tab.

Click on the "Create Folder" button to create a new folder for storing the imported content from your website.

Select the newly created folder and click on the "Import from Website" button.

The website crawler will crawl your website for all textual content that Ask71 can learn from. It works even better if you import with a sitemap link. The crawler can crawl up to 5,000 pages.

If you want to import only a certain category of webpages, insert the URL for that category, and it will be considered as the seed URL.

For Example:

https://dundermifflin.com/employees has multiple pages such as:

https://dundermifflin.com/employees/Michael

https://dundermifflin.com/employees/Jim

https://dundermifflin.com/employees/Dwight

Ask71 will import all webpages under the "employees" pathname viz. Michael, Jim, and Dwight.

You can stop a website from being crawled by clicking on the 'stop' icon on the right side of the import progress-bar.

If you stop the crawler mid-way, you will be provided with two options:

"Stop but store indexed pages" allows you to save the pages that have already been crawled and indexed.

"Stop and don't store any pages" will discard any page that has been indexed.

If you're having issues with the website crawler, follow these recommendations:

By following these steps, you can successfully import content from your website into Ask71's knowledge base, allowing it to provide more accurate and relevant information based on your website's content.