API planned playerupdate changes

[1ApRiL]

I monitored the API usage lately and I see that most API requests happen in the following minutes of an hour:

• 0-1: 271% over average
• 30-31: 201% over average
• 15-16: 171% over average
• 45-46: 174% over average

This looks like typical cronjob usage. At these times the update queues get filled with hundreds of players.

To prevent this I plan a new parameter for the playerupdate:
Default would be: type=cronjob
If type is cronjob, the update request gets into a special low priority queue gets worked off slower.
If type is direct, the update request will get worked off normally.

You will be only allowed to use direct if the request was issued by a real person. Examples:

• Click on update button
• First access of the player on the app or site.
• date_lastseen is newer than date_check on a visit to the app or site.

I don't know yet when exactly I will introduce that system. But it will come and you should prepare your apps for it.
If your update requests are started by a real person, you can add type=direct to your signed request parameters, so your app is prepared.

[ZA Tony]

Hallo April,
kannst uns das auch mal in DE erklären, von mir aus auch im DE Forum.

Vielen Dank schon mal

[Tex-Twil]

Hi Dominik,

You will be only allowed to use direct if the request was issued by a real person. Examples:

• date_lastseen is newer than date_check on a visit to the app or site.

I don't know yet when exactly I will introduce that system. But it will come and you should prepare your apps for it.
If your update requests are started by a real person, you can add type=direct to your signed request parameters, so your app is prepared.

Some questions:

Will the API call fail or fallback to the cronjob update if the API client uses the direct update but should have used the cronjob ? An automatic fallback will be of course easier for API users

In another thread, it came up that the date_lastseen is not very reliable so it will be still difficult to detect that a direct update is possible.


Will API keys get more update tickets per hour ? Currently we have only 2 updates/h which is very little! Would it be possible to have e.g. 2 direct updates/h and let's say 10 cronjob/h ?

Thanks

[Grezvany13]

I guess the biggest problem is that most tools autmaticly (= cronjob) retrieve all players by a direct lookup/update, while this data doesn't have to be there instantly.

Although the API already uses a queue all lookups/updates are placed here, while most of the time (like leaderboards) it's OK to have it at the bottom of the queue, where as real updates (= user request) should be on top.


The solution by April is perfect, unless developers abuse it. Although Tex makes a valid point; useing the cronjob option should allow us to have more request (eg. 10 per hour), although the direct option should be lowered (max 1 per 2 hours).

[Tex-Twil]

although the direct option should be lowered (max 1 per 2 hours).

But there are cases where a user needs more than that. In my iPhone app, I can have a list of players and even now, with 2 updates per hour, it is impossible to update all players at once. The use case is simple: when I finish playing I want to check my and my teammates stats

http://f.cl.ly/items/0j3F222I440X2P1...enshot_01a.png

Btw, when do you plan to enable those options ? I've already integrated them in my app

[ezzemm]

Is this correct?

If a user triggers a update by hand, then is should use this "CURLOPT_POSTFIELDS":

$postdata = Array(
    'player' => 'playername',
    'type' => 'direct',
    'opt' => 'clear,all'
); 

[Tex-Twil]

Is this correct?

If a user triggers a update by hand, then is should use this "CURLOPT_POSTFIELDS":

$postdata = Array(
    'player' => 'playername',
    'type' => 'direct',
    'opt' => 'clear,all'
); 

This concerns the playerupdate function, not the playerlist one.

[ezzemm]

This concerns the playerupdate function, not the playerlist one.

I am not sure what your mean... in this topic it is about the playerupdate function.

[Tex-Twil]

I am not sure what your mean... in this topic it is about the playerupdate function.

In your PHP code you specify 'opt' => 'clear,all' so I suppose you are constructing a player request. This topic is related only to the signed playerupdate requests (which do not have any 'opt' parameter.

[xbenderCH]

I don't know yet when exactly I will introduce that system. But it will come and you should prepare your apps for it.
If your update requests are started by a real person, you can add type=direct to your signed request parameters, so your app is prepared.

2 questions:
- Is it possible to pass this parameter to the current api without having problems? (to be prepared when you introduce it to the system)
- Wouldn't it make sense to store this type on the API master id? You could avoid abuse of this parameter and the default should be 'cronjob'. Only approved 'client's should get the 'direct' priority and you have the majority over it.

Best
xbenderCH

[Tex-Twil]

2 questions:
- Is it possible to pass this parameter to the current api without having problems? (to be prepared when you introduce it to the system)
- Wouldn't it make sense to store this type on the API master id? You could avoid abuse of this parameter and the default should be 'cronjob'. Only approved 'client's should get the 'direct' priority and you have the majority over it.

Best
xbenderCH

1: yes ... it's working for me at least

2: but what if the client wants to use the cronjob as fall back. Try "direct" if no updates left, use cronjob.

Maybe a special "cronjob"-only keys could be delivered to people doing cronjob like tasks. The normal keys could still use both types of updates

[1ApRiL]

I believe in the honesty of the API users that they will not abuse it.

It would be right like this:

$postdata = Array( 
'time' => time(),
'ident' => $myAPIIdent,
    'player' => 'playername', 
    'type' => 'direct', 
); 

You can add this parameter now without problem or any effect since handling of this parameter has no effect yet.

I will announce it beforehand when it will have an effect.

Connecting this parameter with an API key is no good idea, because some apps/leaderboards offer direct and automatic/scheduled updates.

And sometimes you want direct i.e. if the user adds his player the first time. (This is a user action which is OK for direct call.

You can tell me here some examples of how your apps update players and I can make a list of examples what a direct and what a cronjob action is.

[ezzemm]

@Tex
This was a error in thinking from my side

@1April
In ezStats, I will use this method: A user can start a update from adminpanel, and a POST-parameter is sent via AJAX. The update visitors or a cronjob can start, comes without POST-parameter:

$type = $_POST['request'] == "admin" ? "direct" : "cronjob";

### player was pushed ###
if ($array['pushed'] == "1") {
    $result = $player->update($array['id'], $array['name'], $array['platform']);
}

### player is not in bf3stats-DB yet ###
else if ($array['checkstate'] == "0" OR $array['checkstate'] == "notfound") {
    $result = $player->push_sync($array['name'], $array['platform'], "direct");
}

### Last update dates back more than X hours ###
else if (($time - $array['date_check']) > $synctime) {
    $result = $player->push_sync($array['name'], $array['platform'], $type);
}

### Stats update ###
else {
    $result = $player->update($array['id'], $array['name'], $array['platform']);
}


public function update($playerid, $name, $platform) {
    $postdata = Array(
        'player' => $name,
        'opt' => 'clear,all'
    );
    { ... }
}

public function push_sync($name, $platform, $type) {
    $data = Array(
        'ident'  => $this->settings->get('ident'),
        'time'   => time(),
        'type'   => $type,
        'player' => $name
    );
    
    $syncdata = $this->getAPIDataSigned('playerupdate', $platform, $data);
    { ... }
} 

[ronaldmathies]

When this goes into effect please notify us (when possible) at least two weeks in advance, iOS users who update their App need at least 1 week review time. I will make the change as soon as i can but i just released an update which will be available at the end of next week.

And will the direct option now be lowered to 1? because that's not clear to me, it is mentioned but not conclusive for my idea.

Maybe change the first post to mention this.

From then it is possible for me to send out another update, so for me making this change available for the end user takes at least 14 days from today (today as in 19 april).

BTW. I will of course make this change directly so i can send out an update as soon as possible.

[1ApRiL]

I implemented now a prioritized queue in our workers.
But it is not yet in effect for the default type: cronjob
You can test it with type: low

Usually you don't see any difference.
The only effect it can have is if the queues are under load you see your queue position decreased a bit slower.

I will apply the behavior from low to cronjob next week.

[ronaldmathies]

Hi 1ApRiL,

I am looking at the API documentation but i don't seem to see any reference to this. But i assume nothing has changed in the way this should work reading through this post. So Apps that directly want to refresh the statistics should add:

type=direct

So i will add this to my App and make a new release this weekend. By next week do you mean from Monday?

[1ApRiL]

It's important that you use type=direct only if a user requests the data directly.

Now this feature is active.

no type or type=cronjob is now lower prioritized. This should not change much. Only if our queue is filles like at full hours or half an hour, other updates get preferred.

[cottton]

sup!
is this included yet?
Asking because i think its not.

[1ApRiL]

Yes it is. Why do you think it's not?

[Tex-Twil]

How many cronjob updates are allowed per hour for API keys? I still think that 2 direct per hour is very little so I'd like to use the cronjob as a fallback in my iOS app.