Jul 17, 2025 Current Resources

Re: Re[2]: Correctly dealing with bots and scrapers.

nanog mailing list archives

From: "Compton, Rich via NANOG" <nanog () lists nanog org>
Date: Wed, 16 Jul 2025 21:01:53 +0000

I don’t have experience with it, but I was pointed to this project recently which may be of interest to you.
“Anubis is a Web AI Firewall Utility that weighs the soul of your 
connection<https://en.wikipedia.org/wiki/Weighing_of_souls> using one or more challenges in order to protect upstream 
resources from scraper bots. “
https://github.com/TecharoHQ/anubis

From: Ryland Kremeier via NANOG <nanog () lists nanog org>
Date: Wednesday, July 16, 2025 at 2:08 PM
To: North American Network Operators Group <nanog () lists nanog org>
Cc: Ryland Kremeier <rkremeier () barryelectric com>
Subject: Re[2]: Correctly dealing with bots and scrapers.
I've thought about using client-side tokens with behavioral analysis and
a points system, but haven't implemented it in one of my sites to test
it's reliability just yet.

But basically issue a signed token to each client and watch that client
using js instantiated on each page of the site. Give actions (or lack
thereof) scores based on likelyhood to be a bot and fail2ban them past a
certain threshold.

look at mouse movement, scroll depth, keypress timings, page
transitions, dwell time, js events, start/end routes, etc.

- so if a client is on a page for under 100ms, it accumulates 'bot'
points, say +10 pts
- no EventListener JS events +40 pts
- client joins site at a nested page and drills down +5 pts (low because
people could bookmark a specific page that triggers this)
- etc.

then decay these points based on the opposite effects happening.

There's no guaranteed solution and this would require careful tweaking
of the ban threshold and whatnot with real testing to not accidentally
block your users, but I feel it could mitigate a lot of bot situations.
For banned users just redirect them to a page with a little unban form
they can quickly fill out or something.

-- Ryland

------ Original Message ------
From "Tom Beecher via NANOG" <nanog () lists nanog org>
To "North American Network Operators Group" <nanog () lists nanog org>
Cc "Tom Beecher" <beecher () beecher cc>
Date 7/16/2025 2:43:51 PM
Subject Re: Correctly dealing with bots and scrapers.

As Chris states, broad IP based blocking is unlikely to be very effective ,
and likely more problematic down the line anyway.

For the slightly more 'honorable' crawlers, they'll respect robots.txt,
and you can block their UAs there.

Fail2ban is a very good option right now. It will be even better if
nepenthes eventually integrates with it. Then you can have some real fun.


On Wed, Jul 16, 2025 at 3:39 PM Andrew Latham via NANOG <
nanog () lists nanog org> wrote:

 Chris

 Spot on, and I am getting the feeling this is where the value to a
 geo-ip service comes to play that offers defined "eyeball networks" to
 allow.

 On Wed, Jul 16, 2025 at 12:57 PM Chris Adams via NANOG
 <nanog () lists nanog org> wrote:
 >
 > Once upon a time, Marco Moock <mm () dorfdsl de> said:
 > > Place a link to a file that is hidden to normal people. Exclude the
 > > directory via robots.txt.
 > >
 > > Then use fail2ban to block all IP addresses that poll the file.
 >
 > The problem with a lot of the "AI" scrapers is that they're apparently
 > using botnets and will often only make a single request from a given IP
 > address, so reactive blocking doesn't work (and can cause other issues,
 > like trying to block 100,000 IPs, which fail2ban for example doesn't
 > really handle well).
 > --
 > Chris Adams <cma () cmadams net>
 > _______________________________________________
 > NANOG mailing list
 >
 https://urldefense.com/v3/__https://lists.nanog.org/archives/list/nanog () lists nanog 
org/message/AFJF4UQJZW6ALTY6SA7OHBN2AZC72SZQ/__;!!CQl3mcHX2A!HHKMxs_ccAQOJOwC_YKtrF7G1WneCnCa-eVa1fOtm7gtXNEQmCfoGSNzwbBbBHoF5dI02OqwcRW4-NgPkVX1Vg$<https://urldefense.com/v3/__https:/lists.nanog.org/archives/list/nanog
 () lists nanog 
org/message/AFJF4UQJZW6ALTY6SA7OHBN2AZC72SZQ/__;!!CQl3mcHX2A!HHKMxs_ccAQOJOwC_YKtrF7G1WneCnCa-eVa1fOtm7gtXNEQmCfoGSNzwbBbBHoF5dI02OqwcRW4-NgPkVX1Vg$>



 --
 - Andrew "lathama" Latham -
 _______________________________________________
 NANOG mailing list

 https://urldefense.com/v3/__https://lists.nanog.org/archives/list/nanog () lists nanog 
org/message/DHUYTBIXFMWE2KWC5NKCR7AJIWPYUL4E/__;!!CQl3mcHX2A!HHKMxs_ccAQOJOwC_YKtrF7G1WneCnCa-eVa1fOtm7gtXNEQmCfoGSNzwbBbBHoF5dI02OqwcRW4-NiX3zC-RA$<https://urldefense.com/v3/__https:/lists.nanog.org/archives/list/nanog
 () lists nanog 
org/message/DHUYTBIXFMWE2KWC5NKCR7AJIWPYUL4E/__;!!CQl3mcHX2A!HHKMxs_ccAQOJOwC_YKtrF7G1WneCnCa-eVa1fOtm7gtXNEQmCfoGSNzwbBbBHoF5dI02OqwcRW4-NiX3zC-RA$>

_______________________________________________
NANOG mailing list
https://urldefense.com/v3/__https://lists.nanog.org/archives/list/nanog () lists nanog 
org/message/ECB77Z6SVDZZ6SZ5YGWP4YJ5HVX6KRQE/__;!!CQl3mcHX2A!HHKMxs_ccAQOJOwC_YKtrF7G1WneCnCa-eVa1fOtm7gtXNEQmCfoGSNzwbBbBHoF5dI02OqwcRW4-Ng-8Wl2SQ$<https://urldefense.com/v3/__https:/lists.nanog.org/archives/list/nanog
() lists nanog 
org/message/ECB77Z6SVDZZ6SZ5YGWP4YJ5HVX6KRQE/__;!!CQl3mcHX2A!HHKMxs_ccAQOJOwC_YKtrF7G1WneCnCa-eVa1fOtm7gtXNEQmCfoGSNzwbBbBHoF5dI02OqwcRW4-Ng-8Wl2SQ$>

_______________________________________________
NANOG mailing list
https://urldefense.com/v3/__https://lists.nanog.org/archives/list/nanog () lists nanog 
org/message/P5QTTOVSM3J73OSNMDW2MALXA7BIXZQV/__;!!CQl3mcHX2A!HHKMxs_ccAQOJOwC_YKtrF7G1WneCnCa-eVa1fOtm7gtXNEQmCfoGSNzwbBbBHoF5dI02OqwcRW4-Ng4AKhkpw$<https://urldefense.com/v3/__https:/lists.nanog.org/archives/list/nanog
 () lists nanog 
org/message/P5QTTOVSM3J73OSNMDW2MALXA7BIXZQV/__;!!CQl3mcHX2A!HHKMxs_ccAQOJOwC_YKtrF7G1WneCnCa-eVa1fOtm7gtXNEQmCfoGSNzwbBbBHoF5dI02OqwcRW4-Ng4AKhkpw$>
_______________________________________________
NANOG mailing list 
https://lists.nanog.org/archives/list/nanog () lists nanog org/message/PNUTJVLETF4JYFM7ECJKKK5XFNQ5OCLA/