Heads up! - Restriction for some API functions - if you pull data, read this.
4 days 22 hours ago #7588
by Simomax
Replied by Simomax on topic Heads up! - Restriction for some API functions - if you pull data, read this.
Agreed Fanko, rate limiting would work well for the most part, however after 5-6 hours of going through the logs with a fine tooth comb and arranging/cross referencing the data, it appears it is a minority that are taking the largest slice of the pie. At first I thought a few radmon users were pulling data for themselves, that's at least what I expected, but my findings show a more targeted, data harvesting thing going on. Here are my findings:
First of all I am not going to name and shame anyone on radmon.org, simply because I don't believe this to be coming from any radmon.org users. This appears to be bots/AI/data harvesting services for the main part. There are a good handful of radmon users that I can see are querying the site for their personal use. I can't say for sure if every query/request to the station details/cpm etc are from a radmon user, but for the most part it does look like regular radmon users. And they are just using it normally, like Fanko. So my apologies for abruptly changing the API without notice to those stations/users. All you need to do is update your code, or YAML, or python, whatever, to include your station password and then you can use the two functions for getting the cpm and usv.~
The rest of queries, and I am talking maybe two orders of magnitude more than the regular radmon users, are from just 5 IP addresses. None of the IP addresses submits to radmon.org, by the look of it. Dan is looking into getting me full access to the apache logs so I'll know for sure then.
These 5 IP addresses go into three categories - 1, monitoring within a certain area, 2, harvesting and serving to another web service, and 3, literally hoovering up every single station graph, over and over.
Three IP addresses are querying 5/6 stations for their current cpm/usv. This is category 1. These appear to be say 5 or 6 stations in a country, or area. My guess is they are simply gathering local records, for something. What that something is, I don't know. None of these three IP addresses appear to submit to a station at radmon.org
One IP address keep hoovering up the station list. Over and over and over, every 5-10 minutes. Not real bad really, except the data is being published on a website warlens.net and published as though it is their own, on the face of it. I haven't looked much into that yet. Their user agent is WarLens/1.0 (radmon-watch). So warlens is just raking up the whole station list, the cpm and locations and serving them up on it's own site. I can't see any attribution or anything, just radmon stations and their cpm. Building the station list each time has a small impact on the server as it is a large query. It takes less time direct using the API than re-writing it through joomla, thankfully!
Then we come down to the 5th IP address.
(I don't think I have ever used that emoji on this site before...) This is category 3, literally hoovering up tones of data/graphs and chewing up many cpu cycles and IOs. This IP is constantly getting the station list (station list = user list = https://radmon.org/radmon.php?function=showuserlist) and also getting station pages (such as https://radmon.org/radmon.php?function=showuserpage&user=radspod_one). It's not so much bandwidth that is an issue. The server sits on a gigabit fibre connection. I suspect it's upload is around 100mbps, so no issues with bandwidth. IO becomes a big part. Serving up the graphs over and over, querying the database over and over with queries that create large amounts of results uses a lot of IO on the RPi.
A couple that I forgot (from this 8 hours log) - there are a couple of IP's that are pulling the station page for a couple of stations. This could be users pulling their graphs to another location, I don't know. Hard to say right now. Less than ideal though as it is over and over every 5 and 10 minutes. What bothers me about these kinds of automated processes is, is anyone looking at them? Or are they just being pulled to some web server's drive somewhere, overwriting the last one, then the next is requested, overwrites the old one, and so on. It's a highly inefficient use, unless someone is actually looking at them, 24/7.... I doubt that. It's just a bit rude IMHO. I pull the graph images to my radmon page, but only when someone looks at it. It's less wasteful.
That's about it for what I have found so far, and this comes on top of restricting the CSV/graph generation. Now sadly, and again, my apologies for this, but category one has been fixed by implementing the station password with the request, so people using this will need to update their code to include the password. Categories two and three are still ongoing. So I do feel a bit shitty about changing the API and only stopping the cat 1 stuff. I will get on to the other bits soon, in one way or another. I'm (trying to) speaking with Dan, but seems he is offline right now. He's going to hate the barrage of messages I have left him.
I don't know if anyone else has noticed, I certainly have, that the website seems a lot snappier now. It still has it's hanging/waiting times, but generally I think it feels noticeably faster. Hopefully I can claw some more back soon, but I need to speak with Dan, and think of better ways for people to access the data that hopefully benefits everyone, except those data-sucking leaches.
First of all I am not going to name and shame anyone on radmon.org, simply because I don't believe this to be coming from any radmon.org users. This appears to be bots/AI/data harvesting services for the main part. There are a good handful of radmon users that I can see are querying the site for their personal use. I can't say for sure if every query/request to the station details/cpm etc are from a radmon user, but for the most part it does look like regular radmon users. And they are just using it normally, like Fanko. So my apologies for abruptly changing the API without notice to those stations/users. All you need to do is update your code, or YAML, or python, whatever, to include your station password and then you can use the two functions for getting the cpm and usv.~
The rest of queries, and I am talking maybe two orders of magnitude more than the regular radmon users, are from just 5 IP addresses. None of the IP addresses submits to radmon.org, by the look of it. Dan is looking into getting me full access to the apache logs so I'll know for sure then.
These 5 IP addresses go into three categories - 1, monitoring within a certain area, 2, harvesting and serving to another web service, and 3, literally hoovering up every single station graph, over and over.
Three IP addresses are querying 5/6 stations for their current cpm/usv. This is category 1. These appear to be say 5 or 6 stations in a country, or area. My guess is they are simply gathering local records, for something. What that something is, I don't know. None of these three IP addresses appear to submit to a station at radmon.org
One IP address keep hoovering up the station list. Over and over and over, every 5-10 minutes. Not real bad really, except the data is being published on a website warlens.net and published as though it is their own, on the face of it. I haven't looked much into that yet. Their user agent is WarLens/1.0 (radmon-watch). So warlens is just raking up the whole station list, the cpm and locations and serving them up on it's own site. I can't see any attribution or anything, just radmon stations and their cpm. Building the station list each time has a small impact on the server as it is a large query. It takes less time direct using the API than re-writing it through joomla, thankfully!
Then we come down to the 5th IP address.
A couple that I forgot (from this 8 hours log) - there are a couple of IP's that are pulling the station page for a couple of stations. This could be users pulling their graphs to another location, I don't know. Hard to say right now. Less than ideal though as it is over and over every 5 and 10 minutes. What bothers me about these kinds of automated processes is, is anyone looking at them? Or are they just being pulled to some web server's drive somewhere, overwriting the last one, then the next is requested, overwrites the old one, and so on. It's a highly inefficient use, unless someone is actually looking at them, 24/7.... I doubt that. It's just a bit rude IMHO. I pull the graph images to my radmon page, but only when someone looks at it. It's less wasteful.
That's about it for what I have found so far, and this comes on top of restricting the CSV/graph generation. Now sadly, and again, my apologies for this, but category one has been fixed by implementing the station password with the request, so people using this will need to update their code to include the password. Categories two and three are still ongoing. So I do feel a bit shitty about changing the API and only stopping the cat 1 stuff. I will get on to the other bits soon, in one way or another. I'm (trying to) speaking with Dan, but seems he is offline right now. He's going to hate the barrage of messages I have left him.
Please Log in or Create an account to join the conversation.
Moderators: Gamma-Man
Time to create page: 0.280 seconds