Radmon.org API rate limiting and mass upload feature
2 days 17 hours ago - 2 days 17 hours ago #7528
by Simomax
Replied by Simomax on topic Radmon.org API rate limiting and mass upload feature
I've thought of a couple of ideas, both have their pros/cons/caveats.
1 - Employ a mass upload feature. A CSV file or similar between two date/time points. Upload and stuff into the DB. Could be fairly easily done, unless Joomla gets in the way (she's a bitch). It's open to manipulation though - think 'creative' tik-tok influencer. Something happens close by to them, some power station goes into alarm, so for clicks and giggles they create a user/station and feed it a load of fake historical data, make video "OMG the power station 10 miles from me might be melting down!!! Look see my geiger counter here - I'm being irradiated!!" Far out, maybe, but someone if they had all of the bits (of info, knew about radmon.org etc) may actually do this. Some people are literally crazy these days. So I thought of some kind of 'trusted user/station' thing. But how would that work? There are several users/stations I could immediately mark as trusted, but also many that I have no clue about their stations and the validity of their readings. Maybe people could apply to become trusted, but how to validate? Look back at their graphs and if they look good then fine. That could work. The crux of it all though, is that all of that would probably only be for you, gigermantes, and would require manual intervention. I don't know of any other user that would use a manual mass upload from txt/csv.
2 - (this I think is clever, but would require more development). Employ a failover server, somewhere else on the internet. Maybe my server (it has 99+% uptime). This would require everyone to add to their code for submission, and I guess people could just opt out by not using it. Lets assume a counter with an ESP32/8266, add into the code a failover server url. If the ESP submits to radmon.org and doesn't get the OK, or a 200, anything else, then it simply fires the submission to another server. On this server it could, potentially, drag down the stations table (some mysql sync, or php, but dragging the data across in text/json form, I'm not too keen on. It would have to be encrypted or something) so the server has a copy of the user names and station passwords, that way the submissions could be verified against the user base. If good the readings are stored on a database. A cron or task schedule runs every so often and if radmon.org becomes available again, the failover server runs a script and hoofs all the data right into radmon's database, and can verify the user/station name and password for every record going back in. Then once done purge the failover data.
I actually really like #2, and it would work for every station if the user wants. It's quite a lot of work though and developing radmon.org is hard for me at best - I'm old/seasoned hardcore (back in the day) microsoft sysadmin with very little nix experience, and radmon is a nix box, hardened too, with Joomla guarding the web front end. It's like developing half blind, through a keyhole using chopsticks, where I'm used to romping around playground. But I do like the idea, a lot.
1 - Employ a mass upload feature. A CSV file or similar between two date/time points. Upload and stuff into the DB. Could be fairly easily done, unless Joomla gets in the way (she's a bitch). It's open to manipulation though - think 'creative' tik-tok influencer. Something happens close by to them, some power station goes into alarm, so for clicks and giggles they create a user/station and feed it a load of fake historical data, make video "OMG the power station 10 miles from me might be melting down!!! Look see my geiger counter here - I'm being irradiated!!" Far out, maybe, but someone if they had all of the bits (of info, knew about radmon.org etc) may actually do this. Some people are literally crazy these days. So I thought of some kind of 'trusted user/station' thing. But how would that work? There are several users/stations I could immediately mark as trusted, but also many that I have no clue about their stations and the validity of their readings. Maybe people could apply to become trusted, but how to validate? Look back at their graphs and if they look good then fine. That could work. The crux of it all though, is that all of that would probably only be for you, gigermantes, and would require manual intervention. I don't know of any other user that would use a manual mass upload from txt/csv.
2 - (this I think is clever, but would require more development). Employ a failover server, somewhere else on the internet. Maybe my server (it has 99+% uptime). This would require everyone to add to their code for submission, and I guess people could just opt out by not using it. Lets assume a counter with an ESP32/8266, add into the code a failover server url. If the ESP submits to radmon.org and doesn't get the OK, or a 200, anything else, then it simply fires the submission to another server. On this server it could, potentially, drag down the stations table (some mysql sync, or php, but dragging the data across in text/json form, I'm not too keen on. It would have to be encrypted or something) so the server has a copy of the user names and station passwords, that way the submissions could be verified against the user base. If good the readings are stored on a database. A cron or task schedule runs every so often and if radmon.org becomes available again, the failover server runs a script and hoofs all the data right into radmon's database, and can verify the user/station name and password for every record going back in. Then once done purge the failover data.
I actually really like #2, and it would work for every station if the user wants. It's quite a lot of work though and developing radmon.org is hard for me at best - I'm old/seasoned hardcore (back in the day) microsoft sysadmin with very little nix experience, and radmon is a nix box, hardened too, with Joomla guarding the web front end. It's like developing half blind, through a keyhole using chopsticks, where I'm used to romping around playground. But I do like the idea, a lot.
Last edit: 2 days 17 hours ago by Simomax.
Please Log in or Create an account to join the conversation.
- geigermantes
-
Topic Author
- Offline
- New Member
-
Less
More
- Posts: 5
- Thank you received: 0
14 hours 19 minutes ago #7529
by geigermantes
Replied by geigermantes on topic Radmon.org API rate limiting and mass upload feature
Hi all,
From your first post:
Well if the rule is 30 sec minimum, I'll follow that. So I changed to 30s; That should be enough for the queue to empty itself.
From your 2nd post:
We could open a topic for the bulk upload feature and register the interest (and the reason why).
However I can understand the concern: I think it is totally feasible (and it would happen) that 1 or more individuals would abuse the system just because they can.
As long as I can empty the backlog in case of API unavailability, then there is no need for this feature. This was more to save time and hundreds of API calls by grouping them into one.
As for data replication, what you are describing is easily achieved with PostgreSQL native bidirectional logical replication.
You can have 2 (or more actually) databases in separate locations: everything you write in the first database is automatically replicated to the other one. Same with the other direction.
The databases are always kept in sync.
If one goes down, the other one keeps all the changes in a queue and when it is back up again, the sync is automatically done.
The good thing is that this is a logical replication: you can replicate only some tables, not all, or a subset of the data. So the traffic between the databases is minimal.
The databases don't have to run on the same hardware, same OS or have the same version, as long as they have the replication feature.
I have seen that running with 3 databases in Europe, USA and ASIA. Works well and is not difficult to setup. The replication time is basically the latency between the locations.
What I would do is separate the API function from the rest: 1 or several servers hosting the DBs and APIs, DB replicated, and a separate server hosting the rest of the functions (forums, maps, graphs, etc ...) that you can manage without impacting the APIs
Well, food for thought, I think the saying is.
From your first post:
Carry on trying to fill in the gaps as you are every 13 seconds and see what happens.
Well if the rule is 30 sec minimum, I'll follow that. So I changed to 30s; That should be enough for the queue to empty itself.
From your 2nd post:
1 - Employ a mass upload feature.
We could open a topic for the bulk upload feature and register the interest (and the reason why).
However I can understand the concern: I think it is totally feasible (and it would happen) that 1 or more individuals would abuse the system just because they can.
As long as I can empty the backlog in case of API unavailability, then there is no need for this feature. This was more to save time and hundreds of API calls by grouping them into one.
Employ a failover server,
As for data replication, what you are describing is easily achieved with PostgreSQL native bidirectional logical replication.
You can have 2 (or more actually) databases in separate locations: everything you write in the first database is automatically replicated to the other one. Same with the other direction.
The databases are always kept in sync.
If one goes down, the other one keeps all the changes in a queue and when it is back up again, the sync is automatically done.
The good thing is that this is a logical replication: you can replicate only some tables, not all, or a subset of the data. So the traffic between the databases is minimal.
The databases don't have to run on the same hardware, same OS or have the same version, as long as they have the replication feature.
I have seen that running with 3 databases in Europe, USA and ASIA. Works well and is not difficult to setup. The replication time is basically the latency between the locations.
What I would do is separate the API function from the rest: 1 or several servers hosting the DBs and APIs, DB replicated, and a separate server hosting the rest of the functions (forums, maps, graphs, etc ...) that you can manage without impacting the APIs
Well, food for thought, I think the saying is.
Please Log in or Create an account to join the conversation.
Moderators: Gamma-Man
Time to create page: 0.127 seconds