The purpose of this feature is to allow administrators to selectively control which API calls are throttled, without having to rebuild and deploy a new version every time a throttle is changed.

Current throttle implementation

The current frequency throttle is only able to throttle each user's total call frequency using a semaphore. The UserThrottleFilter uses an InMemoryTimeBlockCountingSemaphore that maps from a key(user id in this case) to a SimpleSemaphore, which keeps a count of calls made and the the after which the count will reset. The UserThrottleFilter then compares the user's call count to the maximum allowed.

New throttle implementation

The new throttle will employ a similar mechanism, except that the keys will be userId+normalizedAPI so that each user can be throttled on each API call. 

The throttle limits for each API call will be defined by a SQL table, allowing administrators to set the throttling of any API calls dynamically.

Table Schema

This table would contain the api calls that administrators would want to throttle, the maximum number of calls per period, and the period of time after which the max calls would reset.

(optional: expiration of throttle rule?? not sure if necessary)

CREATE TABLE `THROTTLE_RULES`(
    `THROTTLE_ID` bigint(20) NOT NULL,                                            -- id of the throttle rule
    `NORMALIZED_URI` varchar(255) NOT NULL,                                        -- normalized api URL, numbers such as {id} replaced with #
    `MAX_CALLS` int(20) unsigned NOT NULL,                                         -- maximum number of calls per user per URI per bin
    `CALL_PERIOD_IN_SECONDS` int(20) unsigned NOT NULL,                            -- bin of time in which a user is allowed to make MAX_CALLS API calls.
    `MODIFIED_ON` TIMESTAMP DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP, -- Timestamp of when the row was changed. used as an etag for migration
    PRIMARY KEY(`THROTTLE_ID`),
    UNIQUE (`NORMALIZED_URI`)
);

The NORMALIZED_URI column must be unique since having 2 different throttling rules for the same API call would not make sense(maybe consider making this primary key instead of using numeric id?).

Reducing Table Accesses

UserThrottleFilter will have a in-memory cached version of this table. It would check the THROTTLED_CALLS table and update its cached version to reflect the information in the database.

The cached in memory version(refreshed every 5 minutes) will be a Map from the normalized URI to a pair of values for maxLocks and lockTimeoutSeconds. 

Proposed ways to update the throttle limits:

• By having UserThrottleFilter periodically read the table (via a DAO) and updating the Map. 
• Wrap the Map with a synchronized singleton. Allowing a worker to update it.

Throttle Logic

When an request comes in, it must first pass the concurrent throttle, then the frequncy throttle, then the service specific throttle.

For the service specific throttle, the request URI will be normalized using normalizeMethodSignature() in AccessRecordUtils of the Synapse-Warehouse-Records project (move/refactor?).

Once normalized, the URI is compared to the cached throttle rules to see if it is being throttled. 

• If it is not throttled, proceed to next filter.
• If it is throttled and the user has exceeded the limit. Return 503 error code (will be changed to 429 when clients are able to handle this code).

Services

No services. Administrators will update the table in MySQL.

Potential problems

If there are many calls being throttled, the throttle could potentially use up a lot of memory. With N throttled calls and M users, the throttle's map for call counts could have up to M x N entires. Additionally, the map will not remove entries for users that are are no longer making calls so memory will not be freed until an administrator calls clearAllLocks().

Updates to the throttle rules will not immediately take effect because they are only written into the SQL table. The actual enforcement of the throttle will not happen until UserThrottleFilter updates its cached version of the rules.