# Crawler

**URL:** https://tosdr.community/c/development/crawler/27.md

[Latest](https://tosdr.community/latest.md) · [Categories](https://tosdr.community/categories.md) · [Tags](https://tosdr.community/tags.md)

---

## [About the Crawler category](https://tosdr.community/t/about-the-crawler-category/1722)

<div class="topic-metadata">

**Author:** [@justin](https://tosdr.community/u/justin)\
**Replies:** 0

</div>

(Replace this first paragraph with a brief description of your new category. This guidance will appear in the category selection area, so try to keep it below 200 characters.) Use the following paragraphs for a longer d…

---

## [Crawlers do not have JavaScript](https://tosdr.community/t/crawlers-do-not-have-javascript/3363)

<div class="topic-metadata">

**Author:** [@Deveroonie](https://tosdr.community/u/Deveroonie)\
**Replies:** 0\
**Last updated:** [April 4, 2025, 10:00pm UTC](https://tosdr.community/t/crawlers-do-not-have-javascript/3363 "2025-04-04T22:00:31Z")

</div>

\[04/04/2025\] Document Number Service \<–(OPTIONAL)–\> Observations The crawlers do not have JavaScript enabled, resulting in the document not crawling successfully (as you need JS to access the site).

---

## [Broken link on "Uncrawlable!" forum post](https://tosdr.community/t/broken-link-on-uncrawlable-forum-post/3068)

<div class="topic-metadata">

**Author:** [@cwg1231](https://tosdr.community/u/cwg1231)\
**Replies:** 0\
**Last updated:** [October 16, 2023, 11:32pm UTC](https://tosdr.community/t/broken-link-on-uncrawlable-forum-post/3068 "2023-10-16T23:32:10Z")

</div>

The forum post describing what people should do if a page is uncrawlable has a broken link. The “how to crawl documents” link appears to be dead. Additionally, I believe the post should contain more information on who it…

---

## [Crawl Failure + Site down (?): akshit's blog](https://tosdr.community/t/crawl-failure-site-down-akshits-blog/3004)

<div class="topic-metadata">

**Author:** [@Blazyrawr](https://tosdr.community/u/Blazyrawr)\
**Replies:** 1\
**Last updated:** [August 31, 2023, 12:21pm UTC](https://tosdr.community/t/crawl-failure-site-down-akshits-blog/3004 "2023-08-31T12:21:07Z")

</div>

\[08/31/2023\] Document Number Service Observations Site seems to no longer exist on Google when searched for

---

## [Crawl failure: Consumidor.gov.br](https://tosdr.community/t/crawl-failure-consumidor-gov-br/1931)

<div class="topic-metadata">

**Author:** [@KnossosDomovoi](https://tosdr.community/u/KnossosDomovoi)\
**Replies:** 7\
**Last updated:** [July 23, 2021, 8:11pm UTC](https://tosdr.community/t/crawl-failure-consumidor-gov-br/1931 "2021-07-23T20:11:10Z")

</div>

I tried to add the service Consumidor.gov.br, however the crawler can’t read any documents there. I tried several different crawlers. Are they all being blocked by the service? The error message: the bot gets the sta…

---

## [Crawl failure: Sympla](https://tosdr.community/t/crawl-failure-sympla/1956)

<div class="topic-metadata">

**Author:** [@KnossosDomovoi](https://tosdr.community/u/KnossosDomovoi)\
**Replies:** 1\
**Last updated:** [July 19, 2021, 8:44am UTC](https://tosdr.community/t/crawl-failure-sympla/1956 "2021-07-19T08:44:39Z")

</div>

\[06/10/2021\] Documents: 6927 6928 6929 6930 6931 6932 6933 6934 6935 6936 \[07/18/2021\] Documents: 8564 8565 8566 8567 8568 8569 8570 8571 8572 8573 8574 Wheh! :sweat\_smile: Added them all. All belong to the servic…

---

## [Crawl failure: Apply.me, People by name, McAfee](https://tosdr.community/t/crawl-failure-apply-me-people-by-name-mcafee/1723)

<div class="topic-metadata">

**Author:** [@Agnes\_de\_Lion](https://tosdr.community/u/Agnes_de_Lion)\
**Replies:** 0\
**Last updated:** [March 31, 2021, 1:14pm UTC](https://tosdr.community/t/crawl-failure-apply-me-people-by-name-mcafee/1723 "2021-03-31T13:14:08Z")

</div>

\[03/31/2021\] Tunisia-sat forums Cookie Policy Reason: Reason: WebDriverError Stacktrace: unknown error: session deleted because of page crash from unknown error: cannot determine loading status from tab crash…

---

## [Crawl failure: Mailo, CDKeys](https://tosdr.community/t/crawl-failure-mailo-cdkeys/1724)

<div class="topic-metadata">

**Author:** [@Agnes\_de\_Lion](https://tosdr.community/u/Agnes_de_Lion)\
**Replies:** 1\
**Last updated:** [March 22, 2021, 9:58pm UTC](https://tosdr.community/t/crawl-failure-mailo-cdkeys/1724 "2021-03-22T21:58:52Z")

</div>

\[03/22/2021\] Charter Mailo 4162 Mailo Reason: Error Stacktrace: write EPROTO 140022019606400:error:141A318A:SSL routines:tls\_process\_ske\_dhe:dh key too small:../deps/openssl/openssl/ssl/statem/statem\_clnt.c:2157…