Commit graph

38 commits

Author SHA1 Message Date
pezcurrel
d1f088a026 Command for subprocesses gets now built on the fly using cmd function; logfile doesn’t get opened if logminmsglev < 4; other minor changes 2022-12-17 17:31:24 +01:00
pezcurrel
0b9e892aef Splitted old crawler.php in 2; this is the part that coordinates 2022-12-17 15:02:20 +01:00
pezcurrel
3804171253 Renamed getfc to gurl 2022-12-16 21:59:26 +01:00
pezcurrel
4522bc3ea8 Added mysqli_query error management for older php versions to function myq 2022-12-16 19:02:41 +01:00
pezcurrel
3a720a90ac Added a warning when nodeinfo specs couldn’t be fetched; made it set New=1 even when host doesn’t respond and is not in the db 2022-12-15 12:45:20 +01:00
pezcurrel
9360bdc481 Made false positives for “IsMastodon” less likely (impossible?) 2022-12-12 22:40:17 +01:00
pezcurrel
e07fba673d Fixed call to non-existent function ”mysq” to “myq” 2022-12-12 08:36:18 +01:00
pezcurrel
ec6324fb4f Made langs() shorten to a maximum of 5 elements the $languages array 2022-12-12 08:29:18 +01:00
pezcurrel
d80ba5ddc4 Fixed double “,” in a query inside langs() 2022-12-12 08:24:26 +01:00
pezcurrel
6f9260e08e myq() did not return results, now it does 2022-12-12 08:17:01 +01:00
pezcurrel
f6752a34bc Added function “myq” as a wrapper for mysqli_query managing exceptions; used it throughout the whole script 2022-12-12 08:12:29 +01:00
pezcurrel
2649e7d137 Info from nodeinfo didn’t end up into $info, now they do 2022-12-12 00:47:06 +01:00
pezcurrel
b16515f4e8 Lots of changes :-)) 2022-12-11 23:29:51 +01:00
pezcurrel
61ad655a62 Disabled fetching profile’s page when “noindex” is not set in account because it takes too long; disabled featured tags fetching fro the same reason; other minor changes 2022-12-10 23:32:58 +01:00
pezcurrel
f343cb702e Changed some eecho messages importance 2022-12-10 13:57:30 +01:00
pezcurrel
4b7f6a199c Added truncs where needed; added code to check for “noindex” on user’s profile page when “noindex” is not set in accounts info 2022-12-10 12:35:22 +01:00
pezcurrel
18ce06871b Added ckratelimit() where useful; made it more flexible with lowercasing every header key; more work on fetching users from users directories 2022-12-09 22:53:18 +01:00
pezcurrel
8341f0e209 Fixed a cosmetic bug; some more work into users directories fetching 2022-12-09 19:25:44 +01:00
pezcurrel
09765e566e Added “normalization” of language codes (dash to underscore) and “pt_BR” language 2022-12-08 13:53:43 +01:00
pezcurrel
c3a90ba6b8 Adding “fetchuser” option; changed “timeout” default from 5 to 10 seconds 2022-12-08 00:03:10 +01:00
pezcurrel
a07e59d52c Added option “--moreclauses” 2022-12-05 21:18:58 +01:00
pezcurrel
a9b5e72373 Added fedibird, ecko and hometown to the regex deciding whether an instance is mastodon or not 2022-12-03 14:44:59 +01:00
pezcurrel
c4b1a53439 Added German language (de) 2022-12-02 16:29:24 +01:00
pezcurrel
632202b69c Added “Done” eecho in the end of run 2022-12-02 16:07:05 +01:00
pezcurrel
e194bd597f Changed notifications “levels”, removed “eecho”es when “notify” is used since it “eecho”es by itself 2022-12-01 17:42:48 +01:00
pezcurrel
92cee555c7 Removed support to Blacklist table 2022-12-01 05:41:54 +01:00
pezcurrel
1eadd2f3ce Removed “StartNodes” loading and “Noxious” table loading 2022-11-30 07:19:14 +01:00
pezcurrel
1a6446e82b Don’t use “black” as derogatory (“blacklist” -> “list of noxious instances”, etc.) 2022-11-29 17:36:49 +01:00
pezcurrel
9b87447da0 Changed “langs” function so that, if “api/v1/instance” returned a language different from the default en, it assumes it is right, because it has been explicitly set, and avoid doing autodection of languages based on last toots 2022-11-23 20:35:53 +01:00
pezcurrel
86c26009e7 Added “NameGL” to the query in “langs” function (forgot it in previous commit) 2022-11-23 19:32:50 +01:00
pezcurrel
5421041a7c Added gl, uk support into “langs” function 2022-11-23 19:30:32 +01:00
pezcurrel
87ff532831 Use gmmktime instead of mktime in pgdatetomy 2022-11-01 07:44:55 +01:00
pezcurrel
dcbd52a04b Fixed a little flaw in get_toot_languages function 2022-05-06 09:21:56 +02:00
pezcurrel
136f99ba24 Rimossa l’opzione «-r» 2022-05-06 06:29:19 +02:00
pezcurrel
371e4c9a2e ... 2020-10-22 17:54:05 +02:00
pezcurrel
049c1d839a ... 2020-10-21 15:26:31 +02:00
pezcurrel
06f702ce71 ... 2020-10-20 16:31:16 +02:00
pezcurrel
5ec1354546 ... 2020-10-18 06:53:27 +02:00
Renamed from web/mustard/clitools/crawler.php (Browse further)