org.hd.d.pg2k.webSvr.util
Class WebUtils

java.lang.Object
  org.hd.d.pg2k.webSvr.util.WebUtils

public final class WebUtils
extends java.lang.Object

Web-server-related utility functions. This is for algorithms only of interest to Web apps, often JSPs.

One advantage of having code here rather than in-line in a JSP is that is is pre-compiled off-line for speed and robustness; code here is also easier to test.

VOTER_LOCATION_STATS_EVENT_DEF

public static final SimpleVariableDefinition VOTER_LOCATION_STATS_EVENT_DEF

Name of event/variable to which we post a voter's approximate geo location.

VOTER_LOCATION_STATS_EVENT_PREFIX

public static final java.lang.String VOTER_LOCATION_STATS_EVENT_PREFIX

Prefix of event value for voter's approximate geo location.

See Also:

Constant Field Values

_cache_cMHTBL_byClient

private static final MemoryTools.SimpleLRUMapAutoSizeForHitRate<java.net.InetAddress,Tuple.Pair<java.lang.String,java.lang.Long>> _cache_cMHTBL_byClient

Private limited-size cache for chooseMirrorHostToBalanceLoad() for best mirror for given client IP. Map from client IP address to full mirror host name and time mirror was computed.

A lock can be held on this object to make multiple operations atomic.

Entries in this map go stale very quickly (in a matter of tens of seconds at most) and are mainly here to avoid repeated recalculations while preparing one HTML page to serve to the client for example.

This has a fixed maximum size to limit memory consumption and to avoid the need for explicit cleaning of old entries (they can just hang around until displaced). We vary this maximum size a little to try to avoid deadly embraces.

We allow this to be entirely discarded when memory is very stressed.

A size limit of a few thousand to allow for a few mega-proxies, and a time limit of some tens of seconds, is probably about right.

MAX_cache_cMHTBL_byClient_AGE_MS

private static final int MAX_cache_cMHTBL_byClient_AGE_MS

Maximum time an entry in _cache_cMHTBL_byClient is valid in ms; strictly positive. Should be no more than the maximum staleness of the mirror liveness variables, and is mainly intended to avoid repeated recalculations during the preparation of a single HTML page for a user giving multiple different values on that one page...

See Also:

Constant Field Values

_lock_fallback_URL_poll

private static final java.lang.Object _lock_fallback_URL_poll

Lock to prevent multiple simultaneous polls of the main or "loopback" URLs at once. Otherwise multiple incoming requests may trigger multiple outgoing polls which actually overwhelm (or trigger throttling by) the remote machine(s).

REDIST_FRAC

private static final int REDIST_FRAC

Fraction of time we redistribute load to possibly non-optimal mirrors; strictly positive. A higher value means that we redistribute traffic to possibly-non-optimal mirrors less often.

A little bit of redistributed traffic makes the system more robust and primed for action, but may give some unlucky users poor performance.

We make this probability lower (this value higher) as the potential (maximum) cost of a wrong choice goes up. The biggest error we can make for most user is to send them to the furthest corner of the planet when there is a same-COUNTRY server available.

ALWAYS_SELECT_FASTEST_MIRROR

private static final boolean ALWAYS_SELECT_FASTEST_MIRROR

If true, always select the fastest available mirror; don't use any stochastic factor.

See Also:

Constant Field Values

IGNORE_REMOTE_MIRRORS_UNLESS_OVERLOADED

private static final boolean IGNORE_REMOTE_MIRRORS_UNLESS_OVERLOADED

If true, then ignore mirrors not close to the user unless we're desperate to shed load. This can take precedence over ALWAYS_SELECT_FASTEST_MIRROR if the fastest mirrors are remote.

See Also:

Constant Field Values

_gPE_vars

private static final SimpleVariableDefinition[] _gPE_vars

System variables tried, in order, for a "popular" exhibit; private to getPopularExhibit(). We put the download var first to get a decent rate of update since this value changes quite frequently.

We don't put the vote var first so as to reduce the temptation to "throw" a vote to get an exhibit shown on the front page.

CHECK_FOR_SPIDERS_BY_UA

private static final boolean CHECK_FOR_SPIDERS_BY_UA

If true then check for bots/spiders by UA (User-Agent); note that clients can forge their UA.

See Also:

Constant Field Values

spiderUAName1stWordsLC

private static final java.util.Set<java.lang.String> spiderUAName1stWordsLC

Immutable Set of known spider/bot UA strings; should probably be moved to a text/properties file. This is a set of lower-cased first (space/tab/bracket-delimited) words from the UA strings.

The names consist only of non-regex-metacharacters in the set [a-z'_-], so are safe to embed in a regex.

UA_TERMINATOR_CHARS

private static final java.lang.String UA_TERMINATOR_CHARS

Set of characters taken as a main-part terminator in a User-Agent header, including whitespace. This should be usable as the separator arg to StringTokenizer, and in a regex when wrapped in "[]" square brackets.

See Also:

Constant Field Values

UA_BOT_PATTERNS

private static final java.lang.String UA_BOT_PATTERNS

Extra case-insensitive patterns matched in UA names, "|"-separated, or null if none. Essentially, anything alphanumeric-ish ending in "bot", or anything alphanumeric-ish containing "spider".

See Also:

Constant Field Values

UA_REGEX

public static final java.util.regex.Pattern UA_REGEX

Case-insensitive regex match for all non-empty UA names from spiderUAName1stWordsLC; null if not checking UA names. Made public to enable some unit testing.

_rPFS_CACHE_PNAME

private static final java.lang.String _rPFS_CACHE_PNAME

Name of Boolean attribute in request we cache result of requestProbablyFromSpider() by.

See Also:

Constant Field Values

_isBot_cache

private static final MemoryTools.SimpleProbabilisticCache<java.lang.String,java.lang.Boolean> _isBot_cache

LRU cache from (common) whole UAs to "bot"ness to save some repeated/slow String parsing; never null when checking for bots by UA. We're prepared to discard all of this under memory stress as we only have to work this out at most once per request for example.

_lastNotLightlyLoaded

private static volatile long _lastNotLightlyLoaded

Private flag for isLightlyLoaded() to note time when we were last non lightly loaded. Once we discover that we are not lightly loaded we set this to the current time, and then do not re-test for a little while in case the testing itself adds to the load.

Initially zero.

Marked volatile for thread-safe lock-free access.

_lastResetLA

private static volatile long _lastResetLA

Time that we last reset the target load average to zero to restart our load ramp-up. Initially 'now' in encourage a gentle start.

Marked volatile for thread-safe lock-free access.

LOAD_AVERAGE_RAMP_UP_TIME

private static final int LOAD_AVERAGE_RAMP_UP_TIME

Time over which to ramp up load limit in isLightlyLoaded(), ms; strictly positive. Should be at least the 60s time claimed in the documentation for OperatingSystemMXBean.getSystemLoadAverage() to dampen CPU-load flapping, especially when emerging from power-conservation mode.

TN_AVAIL_CACHE

public static final boolean TN_AVAIL_CACHE

If true then exhibitHasThumbnail() will try to cache its results. This should avoid us having to load thumbnails into memory just to see if they exist or not.

See Also:

Constant Field Values

TN_AVAIL_CACHE_SIZE_MIN_TARGET

private static final int TN_AVAIL_CACHE_SIZE_MIN_TARGET

The target thumbnail availability cache minimum size, strictly positive. Should be large enough that most thumbnail availability requests will be caught by it regardless of exhibit set size, but small enough to limit memory requirements to something reasonable. Should be *much* larger than the full set of thumbnails ever seen on one page and/or that might be in our "popular" page set so as to avoid thrashing fruitlessly.

We are prepared to give up all cache entries rather than cause an OOME.

See Also:

Constant Field Values

tnHTMLCacheKey

private static final DataSourceBean.UnlinkedKey tnHTMLCacheKey

Private key used by exhibitHasThumbnail(); never null.

_gACSI_cache

private static final MemoryTools.SimpleLRUMapAutoSizeForHitRate<java.lang.String,java.lang.Object> _gACSI_cache

Private cache for getAndCacheStaticImage(); never null. Is a map from root-relative URL to a BufferedImage or SoftReference to a BufferedImage.

Thread-safe LRU-managed limited-size memory-sensitive map.

We are prepared to discard everything if very short of memory.

All getAndCacheStaticImage() activity is synchronized on this instance.

VOTE_PRO_PARAM_NAME

public static final java.lang.String VOTE_PRO_PARAM_NAME

Name of additional parameter to record if this is a vote "for" ("pro"). Value of the parameter is to be "true" or "false".

See Also:

Constant Field Values

VOTE_CON_PARAM_NAME

public static final java.lang.String VOTE_CON_PARAM_NAME

Name of additional parameter to record if this is a vote "against" ("con"). Value of the parameter is to be "true" or "false".

See Also:

Constant Field Values

metadataCacheKey

private static final DataSourceBean.AEPLinkedKey metadataCacheKey

Private key used by getCatPageExhibitMetaDataHTML(); never null.

sDictMD

public static final Compact7BitString.StaticDictionary sDictMD

Static dictionary used by getCatPageExhibitMetaDataHTML() for compression of in-memory data; never null. The static dictionary content should be reviewed after any major format changes, though this is not a correctness issue, only a matter of compression efficiency.

_inlineableInXHTML

private static final MemoryTools.SimpleProbabilisticCache<Tuple.Pair<ExhibitMIME.ExhibitTypeParameters,java.lang.String>,java.lang.Boolean> _inlineableInXHTML

LRU thread-safe private cache mapping from exhibit type and Accept header to acceptability of that MIME type for inlining in XHTML mobile text. We assume that the Accept headers will be more or less constant for a given device, and probably constant between instances of the device, so we have enough entries to cover the likely different types of device using the Gallery at any one time.

We take care to avoid using huge Accept values in keys to avoid DoS-style issues.

We're prepared to discard this entirely when short of memory.

_getCategoryTreeFilterBeanKey

private static final DataSourceBean.AEPLinkedKey _getCategoryTreeFilterBeanKey

Private key used by getCategoryTreeFilterBean(); never null.

MOBILE_REGEX_FLAGS

private static final int MOBILE_REGEX_FLAGS

Flags for User-Agent pattern matching checking for mobile phones.

See Also:

Constant Field Values

MOBILE_REGEX_DMB_1_20100630

private static final java.util.regex.Pattern MOBILE_REGEX_DMB_1_20100630

Mobile-browser detection regex 1 c/o detectmobilebrowser.com 2010/06/30.

MOBILE_REGEX_DMB_2_20100630

private static final java.util.regex.Pattern MOBILE_REGEX_DMB_2_20100630

Mobile-browser detection regex 2 c/o detectmobilebrowser.com 2010/06/30.

ALLOW_SNEAKY_HTTP_CONCURRENCY

public static final boolean ALLOW_SNEAKY_HTTP_CONCURRENCY

If true then allow some "sneaky" browser concurrency. For browsers that do not always pipeline by default but do allow fetches from different-named hosts in parallel (eg the biggies such as IE6/IE7, FF1/1.5/2 as of 2006Q4), then we can possibly help throughput by fetching some images (etc) from a different name for this same host.

The trick is that where the user has arrived at the site with a name other than the local mirror name, and the URI that we were going to use was a rrURL (root-relative URL), then we can prepend the local mirror name to make a new absolute URL that the browser may be prepared to fetch from concurrently.

This has to be done completely consistently for any given item, eg a static "page-furniture" image, to avoid defeating cacheing.

See Also:

Constant Field Values

GEN_NEWSFLASH_HTML

private static final java.lang.String GEN_NEWSFLASH_HTML

Name of generic GenProps property for news-flash HTML segment.

See Also:

Constant Field Values

TEST_PNAME_PREFIX

private static final java.lang.String TEST_PNAME_PREFIX

Prefix of all user testimonials in the global properties.

See Also:

Constant Field Values

_gUT_cache

private static final java.util.Map<java.lang.String,java.util.List<java.lang.String>> _gUT_cache

Logically immutable cached lists of testimonal Strings by language (not locale); never null. Private to getUserTestimonial().

No null keys, not null/empty values.

Small fixed size.

Thread-safe (and supporting reasonable concurrency if possible).

trailDataCacheKey

private static final DataSourceBean.AEPLinkedKey trailDataCacheKey

Private key used by getTrialData(); never null.

_awc_CacheKey

private static final DataSourceBean.AEPLinkedKey _awc_CacheKey

Private key used by approxWordCount(); never null.

_iPCE_vars

private static final java.util.List<SimpleVariableDefinition> _iPCE_vars

Events to be examined by isPopularCatalogueEntry(); never null. These must all have VLONG data stored.

findLatestCodeBundleKey

private static final DataSourceBean.AEPLinkedKey findLatestCodeBundleKey

Private key used by findLatestCodeBundle(); never null.

CODE_SECTION_DIR

private static final java.lang.String CODE_SECTION_DIR

Name of the section/dir in which code/doc bundles are filed.

See Also:

Constant Field Values

WebUtils

private WebUtils()

Prevent construction of an instance.

ignoreRemoteMirrorsNow

private static final boolean ignoreRemoteMirrorsNow(javax.servlet.ServletContext ctxt)

Returns true if we should ignore remote (proximity 'NONE') mirrors as candidates right now. We will generally ignore 'NONE' proximity mirrors unless we're busy or trying to conserve power or otherwise needing to reduce workload.

chooseMirrorHostToBalanceLoad

public static java.lang.String chooseMirrorHostToBalanceLoad(javax.servlet.http.HttpServletRequest request,
                                                             DataSourceBean vars)

Get name (and optional port) of server for best user experience; never null/empty. This is a hostname[:port] suitable to insert into the URL for retrieving an exhibit or another page, etc.

This will attempt to pick one based on a number of factors such as current load, client proximity, which mirrors are up, which mirrors are one the same AEP version as us, etc.

If this cannot chose a suitable server/mirror name it returns the generic "main" name, CoreConsts.MAIN_DATA_HOST.

Though either or both arguments can be null, this is most likely to give useful non-default answers when both values are non-null.

A couple of items of hardwired policy need to be pointed out:

• If the client for this request looks as if it is from a spider (and therefore may cache the results for a long time) then we always return the generic main hostname for speed and for long-term robustness.
• If there is no request/IP address for the client (so we could do no more than statistical load balancing at best) then we always return the generic main hostname for speed and in the hope that we will be doing DNS-based load/proximity stuff sometime.

Though we may sometimes randomise the selected mirror to spread load around, we try to avoid egregious selections, eg on the other side of the planet.

Parameters:

request - the incoming request or null if none available

vars - access to the system variables or null if not available

Returns:

a "mirror" subdomain of CoreConsts.MAIN_DATA_HOST, or by default CoreConsts.MAIN_DATA_HOST thus relying on DNS-based balancing

getPopularExhibit

public static Name.ExhibitFull getPopularExhibit(DataSourceBean dsb,
                                                 ExhibitMIME.ExhibitTypeParameters type,
                                                 java.util.Collection<java.lang.String> excludeFullNames,
                                                 boolean beQuick)

Gets name of "popular" exhibit, possibly filtered by type; null if none available. Tries to pick a "popular" exhibit by looking at one recently voted for, or downloaded, etc, in the system variables, and that has both thumbnails available where they are possible.

This rejects exhibits with a below-par (negative) rating.

Using the system variables should mean that this can pick up values set from any mirror, etc, fairly quickly.

This cannot guarantee to return a non-null value, but any value that it does return is a current, valid exhibit.

Parameters:

dsb - handle on the system variables and data; never null

type - if not null only exhibits of this type are candidates

excludeFullNames - if non-null, any exhibits included by full name are not candidates to be returned

beQuick - if true then don't spend too long trying to calculate this but instead give up quickly if need be (so as not to block page generation for example)

requestProbablyFromSpider

public static boolean requestProbablyFromSpider(javax.servlet.http.HttpServletRequest request)

Attempt to determine quickly if the current request is probably from a spider/bot (ie not a human). NB: This does not attempt to distinguish between good spiders (ie bona fide SEs) and bad/broken/rude bots/scrapers/spiders.

This tries to distinguish human from non-human, at least in part to conserve (prime interactive) resources for humans, and does not claim to be perfect.

This returns true if there is no (valid) referring page (and this visitor is not new to the site, ie has recently pulled another page), though some browsers/firewalls may routinely block this info, and some referrals, eg to target="_blank", may show no Referer value.

This should be quick as most or all requests may be tested with this.

TODO: Should expand this to test against well-known IP addresses.

TODO: Should include a behavioural element, eg mindless, rapid, pauseless browsing.

Parameters:

request - the incoming request; never null

Returns:

true if the requesting client is probably a bot, false if probably a human

requestProbablyReferredFromExternalSite

public static java.lang.String requestProbablyReferredFromExternalSite(javax.servlet.http.HttpServletRequest request)

Attempt to detect if the current request has been referred from an external Web site. This simply checks if the "Referer" is apparently from any of our sites or their aliases.

Note that since this info can be forged, or disabled/knobbled for security reasons, this is not completely reliable.

A missing "Referer" will be taken as indicating an "external" reference, and will catch most well-behaved spiders as a result.

Since we may have to do some string parsing this may not be very fast, but it should not be very slow either.

Even if there is more than one "Referer" header we will only look at one.

Parameters:

request - the incoming request; never null

Returns:

null if referred from one of our sites/hosts, "" if unparsable or no referring URL, else normalised (lowercase, stripped of common prefixes, etc) referring host's name

makeExhibitURL

public static java.net.URL makeExhibitURL(java.lang.CharSequence exhibitName,
                                          javax.servlet.http.HttpServletRequest request,
                                          DataSourceBean vars)
                                   throws java.net.MalformedURLException

Generate full URL for exhibit given exhibit name; never null. This may take account of such factors as which servers are up, how heavily loaded, etc, in order to perform automatic load balancing and give the best possible user experience.

In order to do this well we may need the request details, so they should be supplied if possible. These should be the client's request to a Gallery site, else null.

Throws:

java.net.MalformedURLException

makeExhibitRRURL

public static java.lang.String makeExhibitRRURL(java.lang.CharSequence exhibitName)

Generate root-relative URL for exhibit given exhibit name; never null.

makeThumbnailURL

public static java.net.URL makeThumbnailURL(java.lang.CharSequence exhibitName,
                                            boolean std)
                                     throws java.net.MalformedURLException

Generate full URL for thumbnail/sample given exhibit name and standard/small selector; never null.

Throws:

java.net.MalformedURLException

makeThumbnailRRURL

public static java.lang.String makeThumbnailRRURL(java.lang.CharSequence exhibitName,
                                                  boolean std)

Generate root-relative URL for thumbnail/sample given exhibit name and standard/small selector; never null.

makeCatPageURL

public static java.net.URL makeCatPageURL(java.lang.CharSequence exhibitName,
                                          java.lang.String mediaTypeSuffix)
                                   throws java.net.MalformedURLException

Generate full URL for catalogue page given exhibit name; never null. This should always generate the canonical/"official" form of the URL, eg not including mirrors or aliases.

The media-type suffix (eg ".html" or ".wml") must be supplied.

Throws:

java.net.MalformedURLException

makeCatPageRRURL

public static java.lang.String makeCatPageRRURL(java.lang.CharSequence exhibitName,
                                                java.lang.String mediaTypeSuffix)

Generate root-relative URL for catalogue page given exhibit name; never null. This should take account of such factors as which servers are up, how heavily loaded, etc.

The media-type suffix (eg ".html" or ".wml") must be supplied.

isOverloaded

public static boolean isOverloaded(javax.servlet.ServletContext ctxt)

Returns true if this Web server is overloaded (eg for bandwidth or CPU). If this server is positively overloaded then this routine returns true.

If not overloaded or not possible to tell, this returns false.

(If the context is null then this routine may have to return false.)

This may gather information from a number of sources, but in the main the ThroughputFilter's data will be used. We may explicitly set the overload flag here if we detect the system to be overloaded to make it quicker for us and others on a subsequent call; this will get overwritten by TMF when it next gets to assess load.

If true then the UI and application should attempt to use less bandwidth and CPU time than normal, perhaps by using cheaper algorithms than usual (eg less comprehensive searches) or a less-graphics-intensive UI.

This may default to true while the server is starting up to ensure that as little extra CPU (for example) as possible is consumed while the server is compiling JSPs, loading classes, etc.

This is intended to be relatively cheap to call, since it may get called frequently, for example especially when the system is busy, and/or at start-up before JIT compilation (eg HotSpot) has kicked in.

isLightlyLoaded

public static boolean isLightlyLoaded(javax.servlet.ServletContext ctxt)

Returns true if the site seems to be lightly loaded (CPU, bandwidth, etc). In case of difficulties/confusion this reports false.

This routine is quite careful and conservative, and will only return true if this server and the host system both seem to be lightly loaded and stable by all the appropriate metrics.

This never reports the system to be lightly loaded if it is low on power (eg on a nearly-expired battery) so as to avoid unnecessary background work.

Where this can check 'uptime' then it tries to enforce a modest CPU ramp-up over approximately the 60s-or-so of OperatingSystemMXBean.getSystemLoadAverage() to limit rapid fluctuations in CPU load (and, for example, power consumption).

This routine is designed to be called frequently, ie is reasonably fast and efficient.

exhibitsHaveThumbnail

public static java.util.BitSet exhibitsHaveThumbnail(DataSourceBean dataSource,
                                                     java.util.List<Name.ExhibitFull> exhibitNames,
                                                     boolean standard,
                                                     boolean forceCreation)

Test if the given exhibits have thumbnails (locally) available; never null. This is suitable for a bulk check, eg before rendering a table, and may be internally parallelised to overcome I/O latency, etc.

Parameters:

exhibitNames - non-null list of full exhibit names; not altered by this routine and must not be changed by the caller while this routine is running

Returns:

a bit in the result set for each thumbnail that definitely exists in the requested size; an unset bit may represent "not known"

exhibitHasThumbnail

public static boolean exhibitHasThumbnail(DataSourceBean dataSource,
                                          Name.ExhibitFull exhibitName,
                                          boolean standard,
                                          boolean forceCreation)

Test if the given exhibit has a thumbnail (locally) available. Does not try to force one to be made if one is not extant (or in cache) unless the forceCreation argument is true.

Since this is likely to be testing what is in (local) cache, its results can be considered at best a (good) hint.

We do not cache entirely negative answers (ie that an exhibit has no thumbnails) as this may change if we overcome (eg) a temporary resource issue. But we assume the converse, ie that once we have a thumbnail then it will always be available.

Always returns false for exhibit types that do not support thumbnails.

Parameters:

dataSource - the data source; never null

exhibitName - the full exhibit name; must be valid (eg non-null)

standard - if true, tests for the presence of a standard-size

forceCreation - if true, we can try to force creation/fetch of the underlying thumbnail if not already available locally

Returns:

true if exhibit definitely has/had one/both thumbnails, false if unknown or thumbnails are not currently available

makeHTMLInlineImageThumbnailURL

public static java.lang.String makeHTMLInlineImageThumbnailURL(DataSourceBean dataSource,
                                                               Name.ExhibitFull exhibitName,
                                                               boolean std,
                                                               boolean relativeURL,
                                                               java.awt.Dimension tnDim,
                                                               boolean dontCreateTn)
                                                        throws java.io.IOException

Get thumbnail image URL to embed in HTML page (usually JPEG/GIF/PNG); null if none available. By preference uses purpose-built thumbnail, else tries to use image itself if small enough.

Has to be passed a dataSource and a full exhibit name.

Returns null if no suitable thumbnail URL can be generated.

This may cache its results, in particular assuming that once a particular thumbnail becomes available that it does not go away again.

Parameters:

tnDim - (output argument) if not null and result is not null, is filled in with the thumbnail dimensions

dontCreateTn - if true, don't force creation of a thumbnail if one is not already readily available

Throws:

java.io.IOException

getExhibitVariantRange

public static java.util.List<Name.ExhibitFull> getExhibitVariantRange(java.util.List<Name.ExhibitFull> allExhibitNames,
                                                                      java.util.Comparator<Name.ExhibitFull> comparator,
                                                                      int thisExhibitIndex)

Find the set of exhibits with the same subject as the indicated one. Given a List of String exhibit names sorted by ExhibitPropsGlobalImmutable.SMART_ORDER (or possibly just by ExhibitPropsGlobalImmutable.SUBJECT_ORDER), the SUBJECT_ORDER comparator, and the index to a given item in that List, finds all the items surrounding that have the same SUBJECT_ORDER, ie that differ only in attribute words and are variants on the same exhibit. (The result will always contain the input item.)

The List must not contain nulls, all entries must be valid exhibit names, and the List must be sorted implicitly with the comparator passed in. The input index must be within bounds.

This does not alter the List passed in. The return value is an unmodifiable sublist of the input.

This returns the sublist of variants on the indicated exhibit; this will degenerate to just containing the input exhibit name if there are no other exhibit names for the same exhibit.

This assumes that the number of variants of any given exhibit is small, and so a linear search is used.

If the List does not support efficient bidirectional movement and seeks then this routine will be very inefficient.

minimalUniqueENTitlePrefix

public static java.lang.CharSequence minimalUniqueENTitlePrefix(java.util.List<Name.ExhibitFull> exhibits,
                                                                int ourIndex)

Computes a minimal human-readable immutable unique prefix of an exhibit short name to distinguish given exhibit from most others. Uniqueness is not guaranteed, but this is meant to give a reasonably short result that a human can read in the title of a page, for example.

If inputs are bad then this will try to fail gently with an empty-string result rather than an exception to make it robust if called directly from JSP code, for example.

This is passed a List of exhibits sorted in a "smart" order that sorts mainly on the file component of the name, probably in a case-insensitive way.

This routine finds a short word prefix that (case-insensitively) is different from neighbouring exhibits and is thus (depending on the overall sort order) probably the shortest unique prefix amongst the entire collection.

(If no unique value is possible, the entire prefix is returned.)

If the List of size 0 we return the empty string and do not use the index parameter at all. This simplifies use where the list may transiently be empty during start-up.

This routine may run very slowly if the List argument does not support efficient random seeking.

The result is designed to be used in headings and other display text such as image alt/title attributes.

TODO: fix inefficient double-parsing of main words...

Parameters:

exhibits - sorted (smart-ish sorted) list of exhibit names (String value); must not change while routine is running

ourIndex - index (strictly positive) of the exhibit whose abbreviated name we wish to produce; must be within the List

Returns:

"" in case of invalid arguments, else some whole-word prefix of the name

getAndCacheStaticImage

public static java.awt.image.BufferedImage getAndCacheStaticImage(boolean copyResult,
                                                                  java.lang.String resourceRRURL,
                                                                  boolean forceToARGBTrueColour,
                                                                  javax.servlet.ServletContext context,
                                                                  boolean cacheViaSoftReference)
                                                           throws java.lang.IllegalStateException

Get BufferedImage containing expanded image loaded as static resource from WAR; never null. Loaded on first use and cached statically, optionally via a SoftReference to allow the system to reclaim memory.

There may be an adverse effect on system behaviour if many large images are cached in memory; this may be mitigated by storing them via a SoftReference so that the memory can be recycled automatically.

This method is internally synchronized to allow only one image load/decode to happen at once to conserve CPU and memory (and other) resources.

If the image is indexed and forceToARGBTrueColour is true then we expand it to true-colour to make it possible to add markings.

This uses our built-in mediahandler classes to decode the image, so the image type must be one that we have a decoder for.

This may not handle alpha correctly in all cases.

Under memory pressure this may release cached resources whether hard or soft cached.

Beware: since a reference is returned rather than a copy, be careful not to adjust the returned image unintentionally.

Parameters:

context - gives context for WAR from which to load the raw base clickable-map image

resourceRRURL - root-relative URL (eg "/my/image.gif") of source image in WAR; must not be null and must be a type (and with a file extension) that we have a MIME mediahandler for

forceToARGBTrueColour - if true, and the source image is not ARGB true-colour, then we force conversion to ARGB true-colour before cacheing to make it easier to draw on the image

cacheViaSoftReference - if true then we attempt to cache the image via a SoftReference; if at any point this is true when the image needs to be (re)cached, the image remains permanently cached via a strong reference thereafter

copyResult - if true, we force the result to be a copy of the cached value to avoid accidentally changing the cached copy; this may force a change in colour model and/or discarding properties

Returns:

image, possibly a copy, possibly with a converted colour model

Throws:

java.lang.IllegalStateException - if the image is not loadable

generateUserDataPointID

public static java.lang.String generateUserDataPointID(java.lang.String prefix,
                                                       javax.servlet.http.HttpServletRequest request)

Generate a unique key for the given HTTP request; returns null if not possible to generate. For example, we use this to help limit the number of times that a given user is asked to vote.

Generates a string starting with the given prefix and some leading portion of the client's IP address...

It is not fatal if this conflates users, but it is more useful that this never thinks one user is more than one to avoid pestering them too much (or letting a spider inject too much noise for example).

Parameters:

prefix - unique prefix to the generated key; non-empty, non-null

request - the user's request

createAndFileVoteListener

public static java.lang.String createAndFileVoteListener(Name.ExhibitFull exhibitFullName,
                                                         javax.servlet.http.HttpServletRequest request,
                                                         SimpleVariablePipelineIF vars)

Create and post the handler for a vote if possible and returns the unique listenerID. This replaces any existing listener for this voter.

This returns null if we could not create a listener.

If the handler is invoked, then this registers a dummy (inactive) handler to delay the next request to the user to vote.

Parameters:

exhibitFullName - valid exhibit name; never null

request - client's HTTP request; never null

vars - where stats updates are posted; never null

Returns:

listenerID, or null if one could not be generated

getCategoryListSortedAsHTML

public static final java.lang.String getCategoryListSortedAsHTML(DataSourceBean dsb,
                                                                 LocaleBeanBase localeBean,
                                                                 boolean asList)
                                                          throws java.io.IOException

Get sorted, hyperlinked HTML i18n text list of exhibit categories; never null. The list is sorted:

1. By dictionary-order i18n localised title.

We may flag categories as good or bad if significantly so.

Parameters:

asList - if true, entries are preceded by <li> else they are followed by <br />\r\n;

Throws:

java.io.IOException

getCategoryListSortedAsHTML

public static final java.lang.String getCategoryListSortedAsHTML(DataSourceBean dsb,
                                                                 LocaleBeanBase localeBean,
                                                                 java.lang.String entrySeparator)
                                                          throws java.io.IOException

Get sorted, hyperlinked HTML i18n text list of exhibit categories; never null. The list is sorted:

1. By dictionary-order i18n localised title.

We may flag categories as good or bad if significantly so.

Parameters:

entrySeparator - text (followed by CRLF) to terminate entries; null if entries are to be wrapped with li tags

Throws:

java.io.IOException

isDisconnectedSlave

public static final boolean isDisconnectedSlave(DataSourceBean dsb)

Returns true if this seems to be a slave disconnected from the master. This instance may, for example, not wish to collect votes from users if the votes may get discarded without getting to the master.

isPrecacheRequest

public static boolean isPrecacheRequest(javax.servlet.http.HttpServletRequest request)

Returns true if this request is apparently a precacheing request, eg from a "Web accelerator". This is true if a client (such as FireFox) is "reading ahead" but it may be the case that no real human gets to see the content.

See https://developer.mozilla.org/en/Link_prefetching_FAQ

getCatPageExhibitMetaDataHTML

public static java.lang.Object getCatPageExhibitMetaDataHTML(DataSourceBean dsb,
                                                             Name.ExhibitFull exhibitName)

Generates HTML form of exhibit metadata, "" if no such metadata for the specified exhibit; never null. The result is keyed to both the DataSourceBean and the exhibitName.

Cached values are discarded when the AEP changes.

The computed HTML is locale-invariant, which is why cacheing is viable.

The toString() method should be called on the result to get the String HTML text, which may be a String or some other internal representation.

Parameters:

dsb - valid non-null DataSourceBean

exhibitName - valid non-null exhibit full name

getCatPageExhibitMetaDataHTMLRaw

public static java.lang.String getCatPageExhibitMetaDataHTMLRaw(Name.ExhibitFull exhibitName,
                                                                AllExhibitProperties aep)

Computes the raw data for getCatPageExhibitMetaDataHTML() without cacheing; never null.

Returns:

"" if there is no metadata for this exhibit

getCatPageExhibitMetaDataHTMLFuture

public static java.util.concurrent.Future<?> getCatPageExhibitMetaDataHTMLFuture(DataSourceBean dsb,
                                                                                 Name.ExhibitFull exhibitName)

As for getCatPageExhibitMetaDataHTML(), but will compute a missing value asynchronously; never null. If the value is already computed then it is available immediately, else this attempts to spin off task compute the value, and get() will block until the value is ready/computed.

If the target thread pool is full the computation will be done synchronously, ie in this thread blocking this call until complete.

The toString() method should be called on the result to get the String text.

hyphenateHTMLText

public static final java.lang.String hyphenateHTMLText(java.lang.String s)

Hyphenate long HTML text (that contains zero or more `-' characters and little or no whitespace). Replaces hyhens ('-') with spaces to allow a browser to wrap the text.

Usually used with exhibit names or fragments of such names.

canInlineInXHTMLPage

public static boolean canInlineInXHTMLPage(ExhibitMIME.ExhibitTypeParameters exhibitType,
                                           javax.servlet.http.HttpServletRequest request)

Returns true if the given MIME-type can always be inlined in an XHTML (mobile) page. If the type argument is null, this returns false.

This always allows JPEG and GIF, but may also allow other (image) types listed in an incoming "Accept" header.

canInlineInHTMLPage

public static boolean canInlineInHTMLPage(ExhibitMIME.ExhibitTypeParameters exhibitType)

Returns true if the given MIME-type can always be inlined in an HTML page. If the argument is null, this returns false.

getCategoryTreeFilterBean

public static TreeFilterBean getCategoryTreeFilterBean(DataSourceBean dsb,
                                                       java.lang.CharSequence category)

Get selected by-category TreeFilterBean from entire exhibit set; never null. Used for the "by category" exhibit tree view and elsewhere.

This data is cached linked to the DSB (which in passing ensures that it can be dropped automatically under extreme memory stress).

The category name is primarily checked for syntactic validity, not for actual presence in the AEP.

isBrowserOnMobileDevice

public static boolean isBrowserOnMobileDevice(javax.servlet.http.HttpServletRequest request)

Return true if client appears to be a mobile device (browser sniffing). This attempts to detect a small-display, CPU- and bandwidth- constrained device, that might benefit from being sent small XHTML pages rather than standard HTML.

This examines the User-Agent and is unlikely to be completely reliable.

Thanks for the regex to http://detectmobilebrowser.com/

getOptionalSneakyConcurrencyRRURLPrefix

public static java.lang.String getOptionalSneakyConcurrencyRRURLPrefix(javax.servlet.http.HttpServletRequest request)

Get optional prefix for rrURL for extra "sneaky" browser concurrency. This can only apply if:

• ALLOW_SNEAKY_HTTP_CONCURRENCY is true.
• This host has a mirror prefix (ie so that there is an "alternate" name for this host.
• The incoming request hostname is not this local mirror hostname.
• We do recognise the form (a vhost for) the URL in the request.

else this routine always returns "".

This basically only works if the user is using a "generic" URL, but that is more likely to be a machine far away (ie with large RTT), so extra concurrency to try to overcome latency is especially helpful.

This slightly inflates the HTML that the client will see but only if using a generic URL.

May inflate the number of concurrent connections back to this host, but usually only by 1 or 2 at most.

Note that this scheme does not rely on any other host being up, nor having exactly the same content as us.

This is not a technique for distributing load.

Returns:

"" or the http://mirror-... name (with no trailing slash) for this host

getNewsflashHTML

public static java.lang.String getNewsflashHTML(GenProps gp)

Get "newsflash" HTML for the main site front page, or "" if none; never null. This is retrieved from the GenProps.

getUserTestimonial

public static java.lang.String getUserTestimonial(LocaleBeanBase l)

Get short user quote/testimonial at random from those available for the given locale; never null but may be "". This finds a quote, if any, suitable for the current locale (infact, currently just the language is matched), picked randomly from those available, or "" if none is available.

The text is HTML/XML safe, and is pure 7-bit printable ASCII, with any non-ASCII characters encoded as HTML/XML entities.

Parameters:

l - the required locale; never null

getTrailData

public static TrailData getTrailData(DataSourceBean dsb,
                                     Name.ExhibitFull trailExhibitFullName)

Get the TrailData for a given trail exhibit (by full name); null if none. This caches the result in the DSB, linked to the AEP, so never retaining data for expired trails.

(This may negatively cache failure to load TrailData (for a while) for efficiency.)

Safe to apply to an arbitrary/unvetted exhibit name, even an invalid/null value.

approxWordCount

public static final int approxWordCount(DataSourceBean dsb,
                                        Name.ExhibitFull fullExhibitName)

Compute (crude) estimate of words in catalogue page for given exhibit; non-negative. This is designed to be reasonably fast, though not necessarily amazingly accurate, and is intended to help decide how many ad blocks a page may reasonably support.

This may cache its results against the AEP instance.

This counts 'non-furniture' words, ie those originating from the data itself, including the exhibit name, exhibit description, tree AKA/description, etc, with different constituents possibly weighted differently.

For simplicity, this does its computations based on the default site language, even if there may be significant variation in apparent word count for other localisations.

Returns:

zero in case of difficulty (eg exhibit does not exist), else approximate (positive) word count

isPopularCatalogueEntry

public static boolean isPopularCatalogueEntry(SimpleVariablePipelineIF vars,
                                              java.lang.CharSequence exhibitFullName)

Returns true iff the named exhibit and/or catalogue page is popular (well visited/downloaded). Uses the history to decide if a catalogue page and its exhibit are frequently visited/downloaded (wrt other catalogue pages globally and locally).

Parameters:

vars - source of event history; never null

exhibitFullName - full exhibit name; never null

Returns:

true if popular, false otherwise

findLatestCodeBundle

public static Name.ExhibitFull findLatestCodeBundle(DataSourceBean dsb,
                                                    java.lang.String prefix)

Returns full exhibit name for latest version of a code bundle, or null if none. This locates the latest (with a major-minor-micro versioning) bundle in the 'code' section, for the given prefix, or null of none.

The author and extension are ignored for selection purposes.

For example, for the prefix/argument 'javadoc', if the code section includes the files 'javadoc-1-2-3-DHD.zip' and javadoc-1-10-1-ANON.zip' this will return 'code/javadoc-1-10-1-ANON.zip'.

To be found a bundled archive name must be exactly of the form:
prefix-major-minor-micro-AUTH.XTN
where the major, minor and micro components are (small, non-negative) integers. There must be no attributes present.

(Note that a '-' is appended to the supplied prefix.)

This may cache the results against the AEP instance, since the lookup may happen may times and we may have to search through a fair amount of data for each lookup. Note: this does not cache negative results in part to bound the amount of space that can be consumed.

Parameters:

dsb - current data source; never null

prefix - legitimate short-name as bundle name; never null nor empty