[ixpmanager] RADB WHOIS Rate Limiting
Barry O'Donovan (INEX)
barry.odonovan at inex.ie
Mon Jun 19 11:34:25 IST 2023
Just so I'm clear:
- works when run manually on the command line for all members?
- does not work when executed via artisan scheduler?
How do you know it's not working? You've said you have no logs, etc. so
just trying to understand what "not working" looks like.
You can lose the ' >/dev/null 2>&1' from the the artisan cron and see
what messages crop up?
You can change the schedule in app/Console/Kernel.php for testing purposes.
How many members are you running this for? There's IXPs using bgpq3
against radb for hundreds of members, including some really big ones.
Are you running any other queries against radb separate to this that may
cause them to limit your IP?
- Barry
> Richard Laager via ixpmanager <mailto:ixpmanager at inex.ie>
> 16 June 2023 at 05:33
> My IRRDB updates are definitely not working. They work when run
> manually, though, whether through the web interface or the CLI (for
> one AS or for everyone). When I looked before, I saw that the Laravel
> scheduler runs them with >/dev/null 2>&1, so I get nothing in the
> output there.
>
> Should there be logs somewhere? I have nothing in
> storage/logs/laravel.log.
>
> Thoughts on next troubleshooting steps?
>
> I backed out my runInBackground() change, so we'll see if that makes a
> difference.
> --
> Richard
>
>
> _______________________________________________
> INEX IXP Manager mailing list
> ixpmanager at inex.ie
> Unsubscribe or change options here:
> https://www.inex.ie/mailman/listinfo/ixpmanager
> Richard Laager via ixpmanager <mailto:ixpmanager at inex.ie>
> 12 June 2023 at 04:27
> For now, I am using the time honored hack of a sleep(). I'm not sure
> if the runInBackground() is necessary, but since the sleep()s make
> this take substantially longer, that seemed like a reasonable thing to
> try.
>
> Any thoughts on a better answer?
>
> diff --git a/app/Console/Commands/Irrdb/UpdateAsnDb.php
> b/app/Console/Commands/Irrdb/UpdateAsnDb.php
> index 5332a0ec4..a91e76ead 100644
> --- a/app/Console/Commands/Irrdb/UpdateAsnDb.php
> +++ b/app/Console/Commands/Irrdb/UpdateAsnDb.php
> @@ -67,7 +67,13 @@ class UpdateAsnDb extends UpdateDb
>
> $customers = $this->resolveCustomers();
>
> + $first = true;
> foreach( $customers as $c ) {
> + if ($first) {
> + $first = false;
> + } else {
> + sleep(1);
> + }
> $task = new UpdateAsnDbTask( $c );
> $this->printResults( $c, $task->update(), 'asn' );
> }
> diff --git a/app/Console/Commands/Irrdb/UpdatePrefixDb.php
> b/app/Console/Commands/Irrdb/UpdatePrefixDb.php
> index d75a892c4..dc5fbd847 100644
> --- a/app/Console/Commands/Irrdb/UpdatePrefixDb.php
> +++ b/app/Console/Commands/Irrdb/UpdatePrefixDb.php
> @@ -67,7 +67,13 @@ class UpdatePrefixDb extends UpdateDb
>
> $customers = $this->resolveCustomers();
>
> + $first = true;
> foreach( $customers as $c ) {
> + if ($first) {
> + $first = false;
> + } else {
> + sleep(1);
> + }
> $task = new UpdatePrefixDbTask( $c );
> $this->printResults( $c, $task->update(), 'prefix' );
> }
> diff --git a/app/Console/Kernel.php b/app/Console/Kernel.php
> index f00333073..b7ca9b90c 100644
> --- a/app/Console/Kernel.php
> +++ b/app/Console/Kernel.php
> @@ -57,9 +57,11 @@ class Kernel extends ConsoleKernel
> // IRRDB - https://docs.ixpmanager.org/features/irrdb/
> if( config( 'ixp.irrdb.bgpq3.path' ) && is_executable(
> config( 'ixp.irrdb.bgpq3.path' ) ) ) {
> $schedule->command( 'irrdb:update-prefix-db' )->cron( '7
> */6 * * *' )
> + ->runInBackground()
> ->skip( function() { return env(
> 'TASK_SCHEDULER_SKIP_IRRDB_UPDATE_PREFIX_DB', false ); } );
>
> $schedule->command( 'irrdb:update-asn-db' )->cron( '37
> */6 * * *' )
> + ->runInBackground()
> ->skip( function() { return env(
> 'TASK_SCHEDULER_SKIP_IRRDB_UPDATE_ASN_DB', false ); } );
> }
>
>
>
> On 2023-06-07 00:20, Richard Laager via ixpmanager wrote:
> --
> Richard
>
>
> _______________________________________________
> INEX IXP Manager mailing list
> ixpmanager at inex.ie
> Unsubscribe or change options here:
> https://www.inex.ie/mailman/listinfo/ixpmanager
> Richard Laager via ixpmanager <mailto:ixpmanager at inex.ie>
> 7 June 2023 at 06:20
>
> I seem to be getting rate limited by RADB with update-asn-db /
> update-prefix-db. It will get partway through and then die. Doubly
> unfortunate is the fact that IXP Manager doesn't keep track of where
> it was in the cache run, so the next time it starts over again at the
> top. As a result, networks farther down probably never get updated.
>
> Are other people running into this?
>
>
> https://www.radb.net/support/informational/query.html says:
>
>> Please note: The RADb WHOIS service is rate-limited. For large
>> queries, we recommend using the RADb API
>>
>> Programmatic access is considered abuse of the service.
>
> Isn't that exactly what IXP Manager / bgpq3 / bgpq4 are doing???
>
>
> Later, it says:
>
>> If you plan on making a large number of queries please invoke a
>> persistent TCP/IP session. This is done by telnetting directly to
>> whois.radb.net and issuing the !! command. This will spare our server
>> having to establish and teardown connections for every query.
>
> bgpq4, at least, does that. But IXP Manager is still separately
> invoking bgpq4 for each AS when running update-asn-db / update-prefix-db.
> --
> Richard
>
>
> _______________________________________________
> INEX IXP Manager mailing list
> ixpmanager at inex.ie
> Unsubscribe or change options here: https://www.inex.ie/mailman/listinfo/ixpmanager
--
Kind regards,
Barry O'Donovan
INEX Operations
https://www.inex.ie/support/
+353 1 531 3339
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://www.inex.ie/pipermail/ixpmanager/attachments/20230619/eb5e06f6/attachment.htm>
More information about the ixpmanager
mailing list