1️⃣ I finally added a method to the crawler, that removes users from index if they disappear. For example when their accounts on instances are deleted or marked as not public. ⚰️ 🗑️ ♻️

2️⃣ I fixed the situation when crawler is redirected to a large file or to an endless stream when searching for nodeinfo api.
This situation used to block the crawler until the memmory run out. Then it crashed and process was restarted. So it also slowed down the crawling.
🐛 ♾️📈

Current statistics of reported by :
222903 public users on 3551 instances
634 users on 1315 instances (only current beta has needed api)
277613 channels on 1195 instances
23931 users on 402 instances

Sign in to participate in the conversation
Škorpil's Mastodon

Štěpán Škorpil's personal Mastodon server - instance of federated social network