org.hd.d.pg2k.webSvr.util
Class ThroughputMonitorFilterPG2K

java.lang.Object
  org.hd.d.tmf.ThroughputMonitorFilterBase
      org.hd.d.pg2k.webSvr.util.ThroughputMonitorFilterPG2K

All Implemented Interfaces:

javax.servlet.Filter

public final class ThroughputMonitorFilterPG2K
extends org.hd.d.tmf.ThroughputMonitorFilterBase

Filter to monitor throughput (output rate) to help regulate flow. Can be used in association with the compression filter to enable compression when our output pipe is likely to get congested.

(This filter has had a few miscellaneous bits of functionality thrown in for now:

• This captures changes to the standard set of session attributes.
• This redirects users to a canonical hostname that have arrived at a non-standard or deprecated one.

This also makes a rough estimate of the number of users (without forcing a session) by counting the number of IP addresses used to visit the site through this servlet container instance (probably a slight overestimate). This ignores clients that are probably spiders (ie that never send a referring URL), and that appear to be one-off hotlinks from external sites.

A truely distributed front end will need a more sophisticated communication mechanism to share data efficiently between front-end instances.

TODO: This filter is not yet suitable for multi-JVM front-ends (clustering) or otherwise where the WAR container creates multiple instances of this filter for what is logically one front-end.

PNAME_SAVED_REQ_URI

public static final java.lang.String PNAME_SAVED_REQ_URI

Request attribute (property name) in which we save the incoming requestURI. This is useful when we do a redirect internally.

See Also:

Constant Field Values

DO_PREEMPTIVE_GC

private static final boolean DO_PREEMPTIVE_GC

If true, then we may try to force a preemptive GC when the system is idle.

See Also:

Constant Field Values

BPS_TO_GBP30D

private static final float BPS_TO_GBP30D

Conversion factor from byte/second to GB/30day.

See Also:

Constant Field Values

RAMP_UP_TIME_MS

private static final int RAMP_UP_TIME_MS

Ramp up time (ms) from zero bandwidth, eg at start-up or on recovery from overload; non-negative.

See Also:

Constant Field Values

maxGBytesPer30D

private final int maxGBytesPer30D

Any GB-per-30-days traffic limit; strictly positive if set.

maxBpsFor30D

private final int maxBpsFor30D

EX_PREFIX

private static final java.lang.String EX_PREFIX

Unique prefix for exhibits.

See Also:

Constant Field Values

CATPAGE_PREFIX

private static final java.lang.String CATPAGE_PREFIX

Unique prefix for catalogue pages.

See Also:

Constant Field Values

CATAREA_PREFIX

private static final java.lang.String CATAREA_PREFIX

Common (possibly ambiguous) prefix for catalogue page and search page.

See Also:

Constant Field Values

PROFILING_EVENT

private static final SimpleVariableDefinition PROFILING_EVENT

System event to which profiling/busy events are recorded.

logger

private final WebUtils.ServletLoggerWithFallback logger

Our logger which falls back to System.out if servlet log not available; never null.

statsIDTHRFGEN

private final StatsLogger.StatsConfig statsIDTHRFGEN

The stats set to which we log general throughput events. The unique codes are the constants THRFGNAME_XXX.

THRFNAMEPR_CLIENT_REGION

public static final java.lang.String THRFNAMEPR_CLIENT_REGION

General stats event name prefix: client region by IP address.

See Also:

Constant Field Values

THRFNAMEPR_STICKY_CLIENT_REGION

public static final java.lang.String THRFNAMEPR_STICKY_CLIENT_REGION

General stats event name prefix: "sticky" client region by IP address.

See Also:

Constant Field Values

THRFNAMEPR_CLIENT_PROXIMITY

public static final java.lang.String THRFNAMEPR_CLIENT_PROXIMITY

General stats event name prefix: client estimated proximity from IP address.

See Also:

Constant Field Values

THRFNAMEPR_DNSRBL_DETECTION

public static final java.lang.String THRFNAMEPR_DNSRBL_DETECTION

General stats event name prefix: client SPAMmer/compromised detected by given DNSRBL.

See Also:

Constant Field Values

THRFNAMEPR_CODATIME

public static final java.lang.String THRFNAMEPR_CODATIME

General stats event name prefix: coda processing time log(ms).

See Also:

Constant Field Values

THRFNAMEPR_VHOST

public static final java.lang.String THRFNAMEPR_VHOST

General stats event name prefix: virtual host name used to fetch page.

See Also:

Constant Field Values

THRFNAME_REDIR_UNKNOWN

public static final java.lang.String THRFNAME_REDIR_UNKNOWN

General stats event name: redirect issued to client because of unknown name.

See Also:

Constant Field Values

THRFNAME_REDIR_TO_LOCAL

public static final java.lang.String THRFNAME_REDIR_TO_LOCAL

General stats event name: redirect issued to client to get them to local mirror.

See Also:

Constant Field Values

THRFNAME_REDIR_ALIAS

public static final java.lang.String THRFNAME_REDIR_ALIAS

General stats event name: redirect issued to client because of deprecated alias.

See Also:

Constant Field Values

THRFNAME_kHIT

public static final java.lang.String THRFNAME_kHIT

General stats event name: 1000 hits (GETs, POSTs, redirects, etc).

See Also:

Constant Field Values

THRFNAME_kHIT_CAT_PAGE

public static final java.lang.String THRFNAME_kHIT_CAT_PAGE

General stats event name: 1000 catalogue page hits (GETs).

See Also:

Constant Field Values

THRFNAME_kHIT_STICKY_CLIENT_PAGE

public static final java.lang.String THRFNAME_kHIT_STICKY_CLIENT_PAGE

General stats event name: 1000 "sticky client" page hits (GETs).

See Also:

Constant Field Values

DO_QUICK_GEO_LOOKUP

private static final boolean DO_QUICK_GEO_LOOKUP

If false, be prepared to consume extra resources working out geographical location of client. Else (if true), just do quick lookup with internal/cached data.

See Also:

Constant Field Values

REPORT_UNEXPECTED_HOSTNAME

private static final boolean REPORT_UNEXPECTED_HOSTNAME

If true, report any unexpected host name used to access the site.

See Also:

Constant Field Values

REDIRECT_DEPRECATED_HOSTNAME

private static final boolean REDIRECT_DEPRECATED_HOSTNAME

If true, redirect users arriving with a deprecated host name. FIXME: possibly this functionality might be better moved elsewhere.

See Also:

Constant Field Values

nextSendAllParams

private volatile long nextSendAllParams

Time when we will next report the slow-changing parameters, initially 0 to force immediate send. Marked volatile for safe lock-free access, though in fact this is probably unnecessary.

_lastForcedZero

private volatile long _lastForcedZero

Last time reported bandwidth was (forced to) zero, eg because of overload. Set at start-up to allow gradual warm-up rather than sucking in loads of traffic and falling over.

Marked volatile for safe lock-free access, though in fact this is probably unnecessary.

KEY_debugFlag

private static final java.lang.String KEY_debugFlag

Key in generic props of slow-profile flag and/or as system property.

See Also:

Constant Field Values

permSlowProfile

private static final boolean permSlowProfile

Allow a hard-wired profiling from the get-go if required as if centrally enabled.

dontDumpThreadsBefore

private long dontDumpThreadsBefore

Don't dump busy threads again before this time; initially zero so first dump is immediately as required. Volatile for thread-safety and atomicity of access.

LOG_ONLY_TOPMOST

private static final boolean LOG_ONLY_TOPMOST

If true then only show top-most method in log rather than entire trace...

See Also:

Constant Field Values

LOG_ONLY_INJVM

private static final boolean LOG_ONLY_INJVM

If true then omit threads in native code, ie not actively burning CPU in the JVM.

See Also:

Constant Field Values

STARTUP_EXECUTION_SAMPLING_ENABLED

private static final boolean STARTUP_EXECUTION_SAMPLING_ENABLED

Allow start-up fast execution sampling for performance tuning.

See Also:

Constant Field Values

STARTUP_EXECUTION_SAMPLING_MS

private static final int STARTUP_EXECUTION_SAMPLING_MS

Start-up sampling time (ms); strictly positive.

See Also:

Constant Field Values

STARTUP_EXECUTION_SAMPLING_MAX_DUMP

private static final int STARTUP_EXECUTION_SAMPLING_MAX_DUMP

Maximum number of (top) sampled sites to dump; strictly positive.

See Also:

Constant Field Values

stickyClientHitCounter

private final java.util.concurrent.atomic.AtomicInteger stickyClientHitCounter

Counter of "sticky" hits on site pages (rather than graphics). These are batched up and submitted to stats in bundles of 1000 (kHits). This value should remain between 0 and around 1000.

We use an AtomicInteger to avoid having to take a lock.

hitCounter

private final java.util.concurrent.atomic.AtomicInteger hitCounter

Counter of all (successful) hits. These are batched up and submitted to stats in bundles of 1000 (kHits). This value should remain between 0 and around 1000.

We use an AtomicInteger to avoid having to take a lock.

catPageHitCounter

private final java.util.concurrent.atomic.AtomicInteger catPageHitCounter

Counter of all catalogue-page GET hits. These are batched up and submitted to stats in bundles of 1000 (kHits). This value should remain between 0 and around 1000.

We use an AtomicInteger to avoid having to take a lock.

CHECK_DNSRBL_EARLY

private static final boolean CHECK_DNSRBL_EARLY

If true, check clients in DNS BL before each access if status is not known. This can severely impact performance as seen by the user if used for (for example) all HTML pages so a "lazy" approach is generally preferred, but if before downloads of expensive items (eg exhibits) then an eager approach may be better.

See Also:

Constant Field Values

ANTI_DUP_ATTR_NAME

private static final java.lang.String ANTI_DUP_ATTR_NAME

Name of attribute that we use to avoid duplicate application of this filter on one filter chain.

See Also:

Constant Field Values

DNSBL_RESULT_CACHE_MS

private static final int DNSBL_RESULT_CACHE_MS

Approximate time we cache a positive or negative DNS BL lookup value for; strictly positive. A value of the order of an hour or so probably matches the policy of the underlying DNS BLs reasonably well.

inBL

private final MemoryTools.SimpleLRUMapAutoSizeForHitRate<java.net.InetAddress,Tuple.Pair<java.lang.Long,java.lang.String>> inBL

LRU set of clients currently known to be in a DNS BL. A mapping is kept from the full/exact client IP address to the String name of the RBL for a bad client (or to null for a good client not in any BL).

A thread-safe LRU map is used to limit memory use (it does not grow if it can discard an old/expired entry). The underlying DNS service should also be cacheing for us, so a cache-miss here is probably not desperately expensive. We allow a large range in capacity to help cope with DDoS-like bursts of distinct clients (ie distinct remote IP addresses), while whittling down memory consumption in quieter times.

We're prepared to discard this entirely under acute memory stress.

ThroughputMonitorFilterPG2K

public ThroughputMonitorFilterPG2K()

onIdle

protected void onIdle()

Called when the system is idle according to the throughput monitor. We can use this to request a preemptive GC.

Overrides:

onIdle in class org.hd.d.tmf.ThroughputMonitorFilterBase

destroy

public void destroy()

Unhook from servlet logger.

Specified by:

destroy in interface javax.servlet.Filter

Overrides:

destroy in class org.hd.d.tmf.ThroughputMonitorFilterBase

init

public void init(javax.servlet.FilterConfig filterConfig)

Hook into servlet logger.

Specified by:

init in interface javax.servlet.Filter

Overrides:

init in class org.hd.d.tmf.ThroughputMonitorFilterBase

expectedBpsPerStickyClient

protected int expectedBpsPerStickyClient()

Returns expected approximate bytes-per-second per sticky client; strictly positive. Picks up our hard-wired value.

Overrides:

expectedBpsPerStickyClient in class org.hd.d.tmf.ThroughputMonitorFilterBase

tick

protected void tick(int availableBpsPerClientBasic,
                    int allVisitorCount,
                    int stickyVisitorCount,
                    float ltbpss,
                    int dailyUniqueVisitors,
                    float busyFractionSmoothed,
                    boolean tooManyVisitors,
                    boolean connectionTooBusy,
                    boolean systemTooBusy)

Called on every tick, and passed the most important metrics.

Overrides:

tick in class org.hd.d.tmf.ThroughputMonitorFilterBase

Parameters:

availableBpsPerClientBasic - estimated available bytes/sec available for one new client (or an existing client); non-negative

allVisitorCount - approximate count of "live" visitors; non-negative

stickyVisitorCount - approximate count of "live" "sticky" visitors; non-negative

ltbpss - long-term bytes/sec smoothed throughput

dailyUniqueVisitors - estimated unique visitors per day; non-negative

busyFractionSmoothed - smoothed system "busy-ness" value ranging from 0.0 (idle) to 1.0 (busy)

tooManyVisitors - true if well over capacity by user count

connectionTooBusy - true if well over capacity by bandwidth

systemTooBusy - true if system busy fraction dangerously high

slowProfileEnabled

private static boolean slowProfileEnabled(GenProps gp)

If true then slow profiling is enabled.

logBusyThreads

protected void logBusyThreads(int connCount)

Make a note of the thread state in a (busy) system. Log the "busy" methods to the distributed logging store, and extra detail to the System.err.

Overrides:

logBusyThreads in class org.hd.d.tmf.ThroughputMonitorFilterBase

Parameters:

connCount - current active-connection count; non-negative

_logThreads

private void _logThreads(int connCount,
                         boolean dontPrint)
                  throws java.io.IOException

Log running threads to the event record and servlet stats, and optionally to the servlet log.

Parameters:

connCount - number of open inbound connections if non-negative

dontPrint - if false then log to the servlet context log too

Throws:

java.io.IOException

_perfSampleThreadsQuick

private static void _perfSampleThreadsQuick(java.util.concurrent.ConcurrentHashMap<java.lang.StackTraceElement,java.util.concurrent.atomic.AtomicInteger> counts)

Quickly sample running threads for performance logging, eg at start-up. This is likely to be called very frequently, so should be very fast and efficient.

This logs the top-most (current) activation record for every live thread (non-NEW, non-TERMINATED) in the system as a count in the table.

The calling thread is omitted.

It is safe for other threads to read the counts map while it is being updated in here: the map is always kept in a consistent state.

Parameters:

counts - caller-supplied map to be updated with counts of samples at each execution site; never null

isStickyClient

protected boolean isStickyClient(boolean isGET,
                                 javax.servlet.http.HttpServletRequest requestH,
                                 java.net.InetAddress clientIP)

Returns true if we deem this a hit to be from a "sticky" client. "Sticky" means:

• A GET request.
• Not a spider/bot or a hot-link or a prefetch (ie a human seems to have clicked from another of our pages).
• Not a SPAMmer or compromised machine.
• A request for a text page (not an exhibit/graphic) not top-level, eg a catalogue page (as the real meat of the site).

This means that we should capture hits by a user on their second and subsequent page views.

Overrides:

isStickyClient in class org.hd.d.tmf.ThroughputMonitorFilterBase

Parameters:

isGET - true if the method is GET

requestH - the HTTP request; never null

clientIP - the client's IP address; never null

Returns:

true iff this hit is to be treated as from a "sticky" client

stickyClientPageHit

protected void stickyClientPageHit(java.net.InetAddress clientIP)

Note a page (eg HTML) hit from a sticky client.

Parameters:

clientIP - client's IP address, never null.

doFilter

public void doFilter(javax.servlet.ServletRequest request,
                     javax.servlet.ServletResponse response,
                     javax.servlet.FilterChain chain)
              throws java.io.IOException,
                     javax.servlet.ServletException

Called to handle each (HTTP) request. Hooks into the requests to measure throughput and other stats.

We may also redirect a client if a non-preferred (or unrecognised) hostname is used to contact the server.

We call super.doFilter() to get all the basic throughput measurement tasks done.

Specified by:

doFilter in interface javax.servlet.Filter

Overrides:

doFilter in class org.hd.d.tmf.ThroughputMonitorFilterBase

Throws:

java.io.IOException

javax.servlet.ServletException

_doCoda

private void _doCoda(javax.servlet.FilterConfig config,
                     javax.servlet.http.HttpServletRequest requestH,
                     boolean isGET,
                     java.lang.String reqURI,
                     Tuple.Pair<java.lang.Long,java.lang.String> statusInRBL,
                     java.net.InetAddress clientIP,
                     boolean checkBL)
              throws java.io.IOException

Handle coda processing, ie after servicing body of request. Processing started after the bulk of the request has been dealt with may be less visible to the end user.

Throws:

java.io.IOException

getDNSBLs

private java.util.Set<java.lang.String> getDNSBLs()

Get the set of DNS RBLs to screen traffic, empty if none; never null.

dontMonitorMe

protected boolean dontMonitorMe(javax.servlet.http.HttpServletRequest httpServletRequest)

Exclude some traffic from connection-responsiveness monitoring. Exclude stuff that we expect to be unavoidably slow, or that is non-interactive and need not be fast.

Overrides:

dontMonitorMe in class org.hd.d.tmf.ThroughputMonitorFilterBase

Returns:

true to ask for this request not to be monitored