Heads up! - Restriction for some API functions - if you pull data, read this.

Fanko
Offline
New Member

1 day 5 hours ago - 1 day 5 hours ago #7605 by Fanko

Replied by Fanko on topic Heads up! - Restriction for some API functions - if you pull data, read this.

Hi Simomax, thanks for the reply! Regarding the IP-based rate limit, I think we don't need to save the request timestamps to the database; keeping them in memory should be enough. The data used is a few dozen bytes per request and does not take up much memory. I'm not entirely familiar with the server architecture, but as far as I know, it runs on PHP on a Raspberry Pi.
It seems there are some PHP packages, such as the Symfony Rate Limiter with APCu, which allow for this kind of in-memory lookup table to ensure fast processing.
Here is a sample code snippet generated by AI. I can't guarantee that it works out of the box, but it illustrates my idea: reducing the number of requests per IP address to 1 every 10 seconds. The old records are automatically cleared, so there's no risk of running out of memory. I suppose another solution could be used if this one isn't a good fit.

<?php
require 'vendor/autoload.php';

use Symfony\Component\RateLimiter\Storage\ApcuStorage;
use Symfony\Component\RateLimiter\RateLimiterFactory;

// Get the client's IP address
$ip = $_SERVER['REMOTE_ADDR'];

// 1. Initialize the ultra-fast APCu in-memory storage
$storage = new ApcuStorage();

// 2. Configure the rate limit settings
$factory = new RateLimiterFactory([
    'id' => 'api_rate_limiter',
    'policy' => 'token_bucket',
    'limit' => 1,                           // Maximum number of allowed requests in the interval
    'rate' => ['interval' => '10 seconds'], // Time window before a new request is allowed
], $storage);

$limiter = $factory->create($ip);

// 3. Check if the current IP has exceeded the limit
if (false === $limiter->consume()->isAccepted()) {
    // Return a 429 Too Many Requests status code
    http_response_code(429);
    header('Content-Type: text/plain');
    die("Too Many Requests! Please wait 10 seconds before trying again.");
}

// -----------------------------------------------------------------
// Your data processing / reading logic continues below this line...
// -----------------------------------------------------------------
echo "Request allowed. Fetching your data...";

Another option that can also help is caching the data for 30-60 seconds, and while it is in the cache, returning it instead of calling the server, but this will require more memory, as it will store the entire response, not just the timestamp. Thanks

Last edit: 1 day 5 hours ago by Fanko.

The following user(s) said Thank You: Simomax

Please Log in or Create an account to join the conversation.

Simomax
Topic Author
Offline
Moderator
Nuts about radioactive everything!

1 day 1 hour ago - 1 day 1 hour ago #7606 by Simomax

Replied by Simomax on topic Heads up! - Restriction for some API functions - if you pull data, read this.

Thanks for that Fanko. We may not need it yet as I have thought about this and slept on it a bit

There is already basic rate limiting in the radmon core, and a lot of the API requests are already pulling cached data, or at least data that isn't being re-written each time. This applies to the downloadable CSV's and graphs etc. The user list/station list is pulled frequently and that is dynamic, so is rebuilt on every request, and radmon uses that to populate the station list, with the station's latest CPM, so has to be up to date. And there are some other bits that work dynamically. The already cached/saved files I'm not concerned about, it's things such as the user list being pulled to get everyone's CPM over and over, or some outside entity pulling a bunch of station pages, over and over. I have seen data being pulled every few seconds for stations that update every minute, or 5 minutes. It's crazy. Some of these people mustn't have much a clue about the data they are pulling, and IMHO, I don't really think they are looking at the data much either.

What I am considering (and Dan hasn't said a blunt 'no' to this either) is gatekeeping instead of rate limiting. The idea is anyone can have at it at all the data, as with the spirit of radmon.org, but they must be a registered user. Whether with a station or not. This is my idea - anyone that is a registered user can 'apply' for an API token. It would be very simple and instant. Once registered, click an API button on the navigation (this will also include API info), from there click a button to generate a token. Once clicked a new row is created in a table and a random token is generated, stored in the DB and printed on screen. The user would also have the option to regenerate a new one or remove it completely, to disable, for what purpose, I don't know, but it should be an option. Then every public API request function would have to contain the username and the token (no joomla password) and the function/parameters itself. And the existing functions that require the station name and data sending password can stay the same.

Doing it this way means several things. First, it doesn't put any more pressure on the server, except for one lookup to the DB, for certain functions. It's no different than what is already implemented on some functions so the impact would be negligible. I am going to start logging the API requests as separate log files, with a bit more info on them that is useful to Dan and I. And also in the new API token table one row will be called 'is_enabled' or something similar. Meaning we can simply switch it off if we see anyone abusing it.

Doing it that way cuts slack for anyone that want's to use it for legitimate purposes and those abusing it will be seen very easily, and cut off even easier. And I think that most will fall at the hurdle of actually having to create a user account in the first place!

I will certainly keep your idea under my hat though.

Last edit: 1 day 1 hour ago by Simomax.

Please Log in or Create an account to join the conversation.

Fanko
Offline
New Member

8 hours 5 minutes ago #7607 by Fanko

Replied by Fanko on topic Heads up! - Restriction for some API functions - if you pull data, read this.

Hi Simomax, I agree with the suggestion, but even with registered users, an aggressive bot could make 100 requests per second, which will overload the server because it has to check the database every single time to see if the API token is active. Also, manually disabling tokens is reactive - it happens after the server is already overloaded. On the plus side, it will definitely stop casual and lazy bots that won't bother to register.
Rate limiting is a way to keep the server up and running. It stops one person from slowing down the site for everyone else by running dozens of simultaneous requests that they might not even use. This allows everyone to access data from radmon.org, which is exactly in the spirit of the project.
A good solution could be combining both approaches, ensuring that even registered users are restricted from pulling data frequently enough to strain the server.
Cheers!

Please Log in or Create an account to join the conversation.

Moderators: Gamma-Man

Time to create page: 0.217 seconds

Solar powered Raspberry Pi 4 server stats: CPU 73% Memory 15% Swap 18% CPU temp=62.8'C Uptime 2 Days