2️⃣ I fixed the situation when crawler is redirected to a large file or to an endless stream when searching for nodeinfo api.
This situation used to block the crawler until the memmory run out. Then it crashed and process was restarted. So it also slowed down the crawling.
Štěpán Škorpil's personal Mastodon server - instance of federated social network