WAF: Diagnosing and Triaging Synapse Geo-Restriction Blocks

Audience: Synapse service desk and platform engineers responding to a user who cannot reach Synapse and reports a GEO_RESTRICTION error.

Purpose: determine why a specific user was blocked, decide whether the block was correct, and record the evidence for that decision.

1. Background: why Synapse blocks by country

NIH notice NOT-OD-25-083 requires that access to NIH controlled-access data be restricted from a defined set of countries.
Synapse implements this at the network edge, before any request reaches the application: a rule in the AWS Web Application Firewall (WAF) inspects the address each request arrives from, and rejects the request outright if that address belongs to a restricted country.

Because the block happens at the edge, the user receives an HTTP 403 with this body and nothing is recorded in Synapse's own application logs:

That error message is the reason tickets arrive: it invites anyone who believes they were blocked in error to open a service desk request.

The one situation where a legitimate user gets blocked

Many large organisations — hospitals, pharmaceutical companies, universities — route all outbound staff traffic through a corporate proxy or secure web gateway.
When they do, the proxy commonly rewrites the X-Forwarded-For header, which is the header proxies use to record the original visitor's address, replacing the real address with an internal, private address from the organisation's own network.

Private addresses (such as those beginning 10.) are not assigned to any country.
Anyone can use them inside their own network, so no geographic lookup can succeed against one.

Historically the rule treated "no country could be determined" the same as "restricted country", so these users were blocked even though their actual traffic originated in a permitted country.
The rule now makes a narrow exception: if the forwarded address can be positively confirmed to be a private or loopback address, that check no longer blocks the request.
Everything else is unchanged — traffic genuinely arriving from a restricted country is still blocked, and so is traffic whose forwarded address is merely unreadable rather than confirmed-private.

That distinction matters more than it first appears, and §5 explains why.

Each stack carries its own copy of this exception, so when a ticket's evidence points at the proxy problem, confirm the exception is actually in force on the stack that served the user — §6.3 shows how.

2. The decision you need to make

Every geo-restriction ticket resolves to exactly one of three outcomes.

Outcome 3 accounts for the large majority of blocks where no country could be determined.
Distinguishing it from outcome 2 is the central skill in this runbook, and §5 is devoted to it.

Confirming the user's location is mandatory for outcome 2.
Leadership approved allowing proxied users on the explicit condition that the platform team verify each one's geographic location before restoring access.
Step 4 of the walkthrough is that verification step, and it is not optional.

3. Setting up

3.1 Access

You need credentials for the Synapse production AWS account and the AWS CLI installed.

All WAF resources are regional and live in us-east-1, attached to the load balancers in front of the Synapse repository service.

3.2 The three helper scripts

Three small scripts do the repetitive work of submitting a query, waiting for it to finish, and formatting the result.

The complete source for all three is in §10.
Before you start, save each one into a single directory, and run every command in this runbook from that directory.
They need no installation and no execute permission — each is invoked through bash or python3 directly.

How attribute-users.py identifies a user.
Firewall log records contain no Synapse user ID.
What they do contain, for a request made by a logged-in client, is the Authorization header holding the user's access token, whose payload carries a sub claim identifying the account.
Reading that one claim is what links a blocked request to a user, and it is the only way to do so — see §7.4 for why nothing else works.

The script reads only that claim and discards the rest of the token.
It never prints, stores, or transmits token material. Preserve that property if you modify it, and see §7.7 for why it matters.

4. Triage walkthrough

Work through these steps in order.
Each one narrows the question, and most tickets are resolved by step 3.

Step 1 — Identify the user

Tickets usually give a username or an email rather than a numeric ID. Convert it to the numeric principal ID that the rest of the process uses.

The response is {"principalId": <number>}. Save it:

Then read their public profile to confirm you have the right person and to see the organisation they claim:

Profile fields are self-reported, so treat company and location as corroboration only.
They never substitute for the location confirmation in step 4.

Step 2 — Find the user's blocked requests

This is the query that connects a user ID to firewall activity.
It gathers geo-restriction blocks whose forwarded address is a private internal address, then attributes each one to a Synapse account.

Restricting to private forwarded addresses keeps the result small enough that a single query covers the whole retention window.

Example output:

Read that as: the requests genuinely arrived from public addresses in the 203.0.113.x range, but the forwarded header carried private 10.x addresses from inside the user's corporate network.
That is the outcome 2 signature. Continue to step 4.

If your user does not appear, they are not affected by the private-address problem.
Remove the x-forwarded-for condition to see all attributable blocks, but narrow the time window to the dates on the ticket first:

Narrowing matters: attributable geo blocks are dominated by traffic genuinely arriving from restricted countries, so a wide window fills the result limit with unrelated records and silently omits your user.
If your user appears here with a restricted country and no forwarded header, that is outcome 1 — keep the block.

If the user still does not appear at all, their client sends no access token, which is normal for someone browsing the web portal while logged out.
No query can attribute those requests to an account. Use step 3 to obtain their addresses, then step 4.

Step 3 — Recover the user's addresses when attribution fails

Blocked requests never reach the Synapse application, so they leave no trace in application access records.
What those records do hold is the address and proxy fingerprint of every request that succeeded — which is how you learn which addresses to look for in the firewall logs.

How to read the result:

• A via value naming a proxy product confirms the user sits behind a corporate gateway.

• x_forwarded_for here is the whole chain of addresses the application saw. The load balancer appends the address the request actually arrived from, so the last entry in the chain is the real origin address and matches what the firewall recorded. The firewall's geographic check reads only the first entry, which is why a private first entry causes a block even though a perfectly good public address appears later in the same chain.

• The date where successful traffic stops is when the problem began. Compare it against the ticket date and against any proxy rollout at the user's institution.

Take any origin address from this result and confirm the blocks directly:

Step 4 — Confirm the user's location

This step is required before access is restored. Two independent checks, both quick.

4a. The firewall's own country determination.
The firewall records a country for the address the request actually arrived from — an address the user cannot influence, unlike the forwarded header. This is the authoritative signal.

A country outside the restricted list is positive confirmation that the traffic did not originate in a restricted country.

4b. Who owns the address.

Send nothing but the address to this third-party service.

You are looking for an owner consistent with the user's stated employer — a corporate network, or a cloud provider if the user works from a hosted virtual machine.
An address belonging to a consumer VPN or an anonymising service is grounds to ask further questions before restoring access.

Note that a single user may legitimately appear from several addresses, including addresses in different cities or cloud regions.
Do not treat an unexpected city as evidence that you have the wrong person; the token attribution in step 2 is far more reliable than reasoning about geography.

Step 5 — Record the determination

Add to the ticket: the principal ID, the origin addresses, the country the firewall determined for each, the address owner, the private forwarded address, and the proxy named in the Via header.

Together these show the block was a proxy side-effect rather than a restricted-country access attempt, and they leave an auditable basis for the compliance decision.

5. Separating real users from attack traffic

Do not search for blocks by "the country was not restricted".
That set is dominated by automated attacks, and filtering that way buries the handful of genuine users under thousands of irrelevant records.

Internet-wide scanners probe Synapse continuously, and a common technique is to place an injection payload in request headers — including X-Forwarded-For.
A scanner-generated header looks like this rather than like an address:

That string is an obfuscated attempt at a jndi:ldap:// lookup, a well-known remote code execution probe.
The firewall cannot read an address from it, so the "cannot determine a country" path is taken and the request is blocked — which is the correct outcome.
These requests still record a harmless-looking country, because that country comes from the scanner's real address.

The reliable filter is therefore whether the forwarded address parses as a private or loopback address. That is exactly the condition the rule's exception uses:

Example output:

Typically a handful of addresses account for nearly all of it, each belonging to one large organisation's proxy estate — the same employer will often appear several times, once per regional gateway.

A private forwarded address does not by itself mean a permitted country.
The last row above is a request that both carried a private forwarded address and genuinely arrived from a restricted country.
It remains blocked, because the check against the real origin address is independent and still applies.
Always read the country column per row.

Tracking the affected population over time

Sort client-side as shown; a sort clause has no effect after stats ... by bin(1d), which returns rows in arbitrary order.

Where the private-address exception is active, this series should sit near zero for permitted countries while the overall geo-restriction block count stays high, since scanner traffic continues to be blocked.
A sustained non-zero count for permitted countries means the exception is not in effect on at least one stack — check §6.3.

6. Reference: how the rule works

Read this section when the walkthrough gives an unexpected result, or when the rule itself changes.

6.1 The two independent checks

The rule sits at the highest priority of each regional WebACL and is defined in the Synapse-Stack-Builder repository under the web ACL template directory.
It combines two checks with OR, so either one alone will block a request.

Check 1 — the real origin address.
A geographic match against the address the request actually arrived from at the load balancer.
The client cannot forge this, so this check is what actually enforces NOT-OD-25-083.

Check 2 — the forwarded address.
A geographic match against the first entry of the X-Forwarded-For header, configured so that a header present but unreadable counts as a match, combined with an exception for addresses confirmed to be non-routable.

The restricted country codes applied by both checks are CN, HK, MO, RU, IR, KP, CU, VE.
The non-routable ranges used by the exception are the private ranges 10.0.0.0/8, 172.16.0.0/12 and 192.168.0.0/16, plus the loopback range 127.0.0.0/8.

Confirm both lists against the template in Synapse-Stack-Builder rather than trusting this document, since the country list changes whenever NIH updates its guidance.

6.2 What check 2 does in each situation

Two behaviours here are easy to get backwards:

An absent header is not the same as an unreadable one.
Per the AWS ForwardedIPConfig reference: "If the specified header isn't present in the request, AWS WAF doesn't apply the rule to the web request at all."
The "treat as restricted" fallback fires only when the header exists but holds no readable address.
This is why ordinary API and command-line traffic, which sends no forwarded header, is unaffected by check 2 entirely.

Check 2 is not a security control.
X-Forwarded-For is supplied by the client and trivially forged, so anyone determined to evade check 2 already can.
Check 1, over the unforgeable origin address, is the enforcement mechanism.
A related consequence: the geographic part of check 2 reads only the first entry in the chain, while the non-routable exception matches if any entry in the chain is non-routable.
A chain whose first entry is a restricted-country address and which also contains a private address therefore escapes check 2 — and is still caught by check 1 if it genuinely originates in a restricted country.
Given that the header is forgeable in the first place, this is an accepted trade-off.

6.3 Confirming which protections are active on a stack

Synapse production runs paired stacks and the numbering changes with every release, so never hard-code a stack name. List the live WebACLs:

Confirm the non-routable exception is deployed. Each stack has its own copy, so an incomplete result means at least one stack still blocks proxied users:

Inspect the deployed rule itself, which is the definitive answer for a stack whose behaviour is in doubt:

7. Reference: where the evidence lives, and its limits

All production stacks write to the single aws-waf-logs-prod log group, so you never need to know which stack served a request in order to find it.

Check the available history of the access record table directly rather than assuming:

7.1 A successful request leaves no firewall record

The firewall's logging configuration keeps only blocked and counted requests.
You therefore cannot prove from firewall logs that a request succeeded, and you cannot calculate a block rate from them.
Use the access record table for successes, or the load balancer logs when you need both in one place and can do without headers.

7.2 Firewall logs cover 30 days only

Tickets frequently arrive after that, and users often retried for weeks beforehand.
Beyond the window, the load balancer logs still show that an address was blocked by the firewall, though not which rule:

The stack partition must be supplied as an equality condition — the table cannot be scanned across stacks — so repeat the query once per live stack.

7.3 Attribute by account, never by geography

A single user can legitimately reach Synapse from several addresses in different cities or cloud regions.
Excluding an address because its city looks wrong will discard genuine evidence.
Token attribution (step 2) is the reliable method.

7.4 There is no user ID in the firewall logs

The firewall operates below the application and has no concept of a Synapse account, so no query can filter its logs by user.
The access token in the Authorization header is the only link, which is why attribute-users.py exists and why attribution works only for requests from a signed-in client.
Anonymous portal traffic cannot be attributed at all; fall back to step 3.

7.5 One principal ID is not a person

Synapse represents not-logged-in traffic with a single reserved anonymous principal, defined in the platform source as AuthorizationConstants.BOOTSTRAP_PRINCIPAL.ANONYMOUS_USER.
Blocks attributed to it are unauthenticated requests from many unrelated sources, not one individual's activity.
It usually has the highest block count in any result, which is expected and not a finding.

7.6 Unattributed credentials are normal

attribute-users.py reports a reason code instead of an account when it cannot attribute a request safely. None of these indicate a broken query.

A large UNDECODABLE count over a wide window is expected; it is mostly scanner traffic sending junk in the Authorization header.

7.7 Never copy credentials out of the logs

The firewall logs request headers without redaction, so raw log records contain live, working access tokens.
Do not paste raw records into tickets, chat, or email.
attribute-users.py exists so that you never need to handle the token itself — use it instead of reading raw records.
If a token is exposed, have the user revoke it in their Synapse account settings.

7.8 Query cost and duration

Insights queries are billed by volume scanned, and the firewall log group holds tens of gigabytes across its retention window.
The cost is small but queries over a wide window take minutes.
Start from the dates on the ticket and widen only if the result is empty.

7.9 A stale Athena database exists

An older Glue database of web ACL log tables remains in the account from a long-retired stack.
It is not connected to current logging and querying it returns nothing useful.
Current firewall logs are only in CloudWatch, as listed above.

8. Replying to the ticket

Outcome 1 — genuinely in a restricted country.

Our access logs show your requests reaching Synapse from an address that geolocates to {COUNTRY}. Access to Synapse from this location is prohibited under NIH notice NOT-OD-25-083 (https://grants.nih.gov/grants/guide/notice-files/NOT-OD-25-083.html). We are unable to grant an exception.

Outcome 2 — corporate proxy, location confirmed.

Thank you for the report. We confirmed that your requests reach us from an address operated by {ORGANISATION}, which our logs place in {COUNTRY} — not a restricted location. The block was caused by your organisation's network proxy replacing your address with an internal one that cannot be matched to any country, and our rule treating "cannot determine a country" as restricted. This is addressed in {RELEASE}. We have recorded the location determination on this ticket.

Outcome 3 — automated attack traffic.
No reply. Close the ticket.

9. Command summary

Run from the directory holding the three scripts.

10. Appendix: the helper scripts

Save each of these into one directory and run the runbook's commands from there.
No installation, no execute permission, and no changes to your PATH are needed.

waf-query.sh

Submits a CloudWatch Logs Insights query against the firewall logs, waits for it to finish, and prints the raw JSON result.

athena-query.sh

Submits an Athena SQL query, waits for it, and prints tab-separated results with a header row.

attribute-users.py

Reads the JSON from waf-query.sh and reports, per Synapse account, how many requests were blocked and which addresses were involved.