org.hd.d.pg2k.svrCore.props
Class LocalProps

java.lang.Object
  org.hd.d.pg2k.svrCore.props.LocalProps

public final class LocalProps
extends java.lang.Object

This is a set of properties local to the JVM. This is the NEAR-minimum set of properties to do with security or local system configuration (such as the local layout of the file-system) that must not or cannot be sent over the network and shared between master and slaves. Much of this is bootstrapping data, such as the location of various configuration and persistent-state directories. Our aim is to minimise the need for other parts of the system to get at random JVM system properties and files, etc, where possible.

Some of the more sensitive data are made available via package-visible members so that the other props classes can use them but nothing else can see the values directly.

These properties are not Serializable and are never sent across the network (unlike GenProps), and have to be made available in a local file and/or system properties for example. (Do not include local properties file in the WAR or EAR file.)

These properties can be checked for being up-to-date from time to time, and the in-memory values will be refreshed if necessary.

Values set purely from system properties may be collected once at class creation and not again.

All access to values is via static methods appropriately synchronized to ensure thread-safety.

This is necessarily a rag-tag of values used all over the system, and will be loaded very early and should not depend on much of the rest of the system if at all.

PNAME_MICRO_CLOUD_TAG

public static final java.lang.String PNAME_MICRO_CLOUD_TAG

Name of cloud-mirror tag system property; non-null and non-empty. If the named system property is set to a valid mirror tag then this instance is a light-weight 'micro' cloud instance.

This is unqualified to help keep command-line parameters short.

See Also:

Constant Field Values

rawMicroTag

private static final java.lang.String rawMicroTag

Raw value of micro tag, or null if not set.

microTag

private static final java.lang.String microTag

Validated value of micro tag, or null if not set or not valid.

noGeneralFileAccess

private static final boolean noGeneralFileAccess

True if this instance should make no filesystem access outside the WAR cache.

MIN_RECHECK_MS

private static final int MIN_RECHECK_MS

Minimum interval between rechecks of local properties file content (ms). Since much of our data comes from the filesystem and many filesystems' timestamp granularity is about one second, polling much faster than this is pointless and may simply end up blocking the caller and wasting CPU pointlessly.

(Arguably the only way that system properties can be changed is by programmatic intervention want so we don't want to recheck such values at all for fear of setting up a covert channel as well as just wasting CPU time.)

We pick a prime-ish retry interval to try to avoid collisions with other regular events.

A value of a few seconds to a few tens of seconds is probably appropriate.

See Also:

Constant Field Values

lastRecheck

private static long lastRecheck

The last time that we checked/refreshed the data that we hold. We update this whenever we do a check whether we changed any values; this avoids possibly expensive and unnecessary rechecking. This value is private to us.

Initially zero to force immediate read on first use of these properties.

reportedSecurityException

private static volatile boolean reportedSecurityException

Set true if we've noted a SecurityException trying to read LocalProperties. Marked volatile for lock-free thread-safe access.

timestamp

private static long timestamp

The last time our loaded values changed. This can be used to cascade changes to parts of the system that depend on these LocalProps.

Will never be newer than lastRecheck.

Initially zero, to indicate no (file-based) values loaded.

The timestamp is the latest timestamp of any of the files read to make up the dataset.

DEFAULT_PROPS_SUFFIX

public static final java.lang.String DEFAULT_PROPS_SUFFIX

Default suffix of local properties files, including the dot.

See Also:

Constant Field Values

DEFAULT_LOCAL_PROPS_FILENAME

public static final java.lang.String DEFAULT_LOCAL_PROPS_FILENAME

Default filename of local properties file.

See Also:

Constant Field Values

PNAME_LOCAL_PROPS_FILENAME

public static final java.lang.String PNAME_LOCAL_PROPS_FILENAME

Name of props file for local config information property.

See Also:

Constant Field Values

localPropsFilenameFromSystemProperties

private static final java.lang.String localPropsFilenameFromSystemProperties

Cache of name of props file for local config information; null if no explicit system property value set.

PNAME_DATA_DIR

public static final java.lang.String PNAME_DATA_DIR

Name of local properties parameter for name of subdirectory for (mainly read-only) data files.

See Also:

Constant Field Values

dataDir

private static java.lang.String dataDir

Local cache of name of subdirectory for data files.

PNAME_CONF_DIR

public static final java.lang.String PNAME_CONF_DIR

Name of local properties parameter for name of subdirectory for read-only conf files.

See Also:

Constant Field Values

confDir

private static java.lang.String confDir

Local cache of name of subdirectory for read-only conf files.

PNAME_SIMPLEPASS_FILE

public static final java.lang.String PNAME_SIMPLEPASS_FILE

Name of local properties parameter for name of simple password file.

See Also:

Constant Field Values

simplePassfile

private static java.lang.String simplePassfile

Local cache of name of simple password file; null if none.

PNAME_UPLOAD_DIR

public static final java.lang.String PNAME_UPLOAD_DIR

Name of local properties parameter for name of upload dir.

See Also:

Constant Field Values

uploadDir

private static java.lang.String uploadDir

Local cache of name of upload dir; null if none specified.

PNAME_MAX_UPLOAD_PER_USER_BYTES

public static final java.lang.String PNAME_MAX_UPLOAD_PER_USER_BYTES

Name of Web server maximum-upload-space-per-user (bytes) property.

See Also:

Constant Field Values

uploadMaxBytesPerUser

private static int uploadMaxBytesPerUser

Web server maximum-upload-space-per-user (bytes).

PNAME_MAX_UPLOAD_TOTAL_BYTES

public static final java.lang.String PNAME_MAX_UPLOAD_TOTAL_BYTES

Name of Web server maximum-upload-space-total (bytes) property.

See Also:

Constant Field Values

uploadMaxBytesTotal

private static int uploadMaxBytesTotal

Web server maximum-upload-space-total (bytes).

PNAME_PERSISTENT_STATE_DIR

public static final java.lang.String PNAME_PERSISTENT_STATE_DIR

Name of local directory for store of persistent state property.

See Also:

Constant Field Values

persistentStateDirFromSystemProperties

private static java.lang.String persistentStateDirFromSystemProperties

Local cache of name of local persistent-state directory; null if none.

DEFAULT_STATE_DIR

public static final java.lang.String DEFAULT_STATE_DIR

Relative path from local properties directory to persistent state dir.

See Also:

Constant Field Values

PNAME_MIRROR_TAG

public static final java.lang.String PNAME_MIRROR_TAG

Name of mirror-tag property.

See Also:

Constant Field Values

mirrorTag

private static java.lang.String mirrorTag

Local cache of mirror tag (micro tag if set); null if none.

PNAME_PERSISTENT_THUMBNAIL_DIR

public static final java.lang.String PNAME_PERSISTENT_THUMBNAIL_DIR

Name of local directory for store of thumbnail data property.

See Also:

Constant Field Values

thumbnailDirFromSystemProperties

private static final java.lang.String thumbnailDirFromSystemProperties

Local cache of name of local thumbnail directory; null if none.

DEFAULT_THUMBNAIL_DIR

public static final java.lang.String DEFAULT_THUMBNAIL_DIR

Relative path from exhibit directory to thumbnails dir.

See Also:

Constant Field Values

PNAME_WEBSVR_MAX_EX_CACHE_BYTES

public static final java.lang.String PNAME_WEBSVR_MAX_EX_CACHE_BYTES

Name of Web server maximum generic exhibit cache size (64-bit value, bytes).

See Also:

Constant Field Values

SYS_PROP_WEBSVR_MAX_EX_CACHE_BYTES_VALUE

private static final long SYS_PROP_WEBSVR_MAX_EX_CACHE_BYTES_VALUE

System-property value for maximum generic exhibit cache size; 0 if unset.

WEBSVR_MAX_EX_CACHE_BYTES

private static long WEBSVR_MAX_EX_CACHE_BYTES

Web server maximum generic exhibit cache size (64-bit value, bytes).

PNAME_SERVER_SLOWDOWN_FACTOR

public static final java.lang.String PNAME_SERVER_SLOWDOWN_FACTOR

Name of server slowdown factor; strictly positive.

See Also:

Constant Field Values

SERVER_SLOWDOWN_FACTOR

private static int SERVER_SLOWDOWN_FACTOR

Server slowdown factor; defaults to 1 except for cloud (CPU-sensitive) environment where it is higher.

PNAME_XFER_HMAC_KEY

public static final java.lang.String PNAME_XFER_HMAC_KEY

Name of local properties parameter for HMAC key used to protect data in transit.

See Also:

Constant Field Values

xferHMACKey

private static javax.crypto.SecretKey xferHMACKey

Local cache of immutable HMAC (secret) key used to protect data in transit; null if none.

PNAME_LOW_POWER_FILE_FLAGS

public static final java.lang.String PNAME_LOW_POWER_FILE_FLAGS

Name of local properties parameter for Set of low-power file-system flags.

See Also:

Constant Field Values

lowerPowerFileFlags

private static java.util.Set<java.io.File> lowerPowerFileFlags

Local cache of immutable Set of low-power file-system flags; empty if none.

PNAME_LOW_POWER_FILE_URIS

public static final java.lang.String PNAME_LOW_POWER_FILE_URIS

Name of local properties parameter for Set of low-power remote URI http: or https: flags. Should only be regarded as absent, ie not low-power mode, if a 404 is obtained.

See Also:

Constant Field Values

lowerPowerFileURIs

private static java.util.Set<java.net.URI> lowerPowerFileURIs

Local cache of immutable Set of low-power file-system flags; empty if none.

DEFAULT_LIGHT_LOAD_FRACTION_MAX

public static final float DEFAULT_LIGHT_LOAD_FRACTION_MAX

Default max fraction of available CPUs in use for system to be considered lightly loaded; positive range ]0.0,1.0[ exclusive. Set threshold to allow for functional units possibly being shared (eg HyperThreading), and thus different systems have different notions of 'processor' for the purposes of our particular workload.

Also avoid significant extra heating or power draw.

A value of between 0.1 and (say) 0.75 is probably reasonable.

PNAME_LIGHT_LOAD_FRACTION_MAX

public static final java.lang.String PNAME_LIGHT_LOAD_FRACTION_MAX

Name of CPU light-load threshold.

See Also:

Constant Field Values

lightLoadMax

private static java.lang.Float lightLoadMax

CPU light-load threshold; can be null if none set.

DEFAULT_HEAVY_LOAD_FRACTION_MIN

public static final float DEFAULT_HEAVY_LOAD_FRACTION_MIN

Min fraction of available CPUs in use for system to be considered very heavily loaded; strictly positive. Can be more than 1.0 indicating queued work for each available CPU, and maybe much higher for a usually/acceptably disc-I/O-heavy environment.

Note that there may be immediate visual effects on the Web site of exceeding this, such as an obviously-'lite' UI to reduce load, and thus doing so prematurely may be annoying to users.

Must be higher than DEFAULT_LIGHT_LOAD_FRACTION_MAX to avoid confusion, and this can be easily achieved by having a value greater than 1.0.

Keeping this significantly below 1 will not soak up all available CPU cycles but may result in better interactivity/responsiveness.

Note also that on environment such as AWS as at 2011/05 the value returned by this does not accurately indicate spare CPU available to *this* virtual instance (eg may be underlying host's uptime) so this should be ultra-conservative to help compensate in such environments.

A value of between 0.25 and (say) 5.0 is probably reasonable.

PNAME_HEAVY_LOAD_FRACTION_MIN

public static final java.lang.String PNAME_HEAVY_LOAD_FRACTION_MIN

Name of CPU heavy-load threshold.

See Also:

Constant Field Values

heavyLoadMin

private static java.lang.Float heavyLoadMin

CPU heavy-load threshold; can be null if none set.

LocalProps

public LocalProps()

isCloudMirrorInstance

public static boolean isCloudMirrorInstance()

If true then this (WAR) instance is (permanently) configured as a light-weight cloud-hosted mirror. This may imply multiple behaviour and configuration changes throughout the system.

Implications of this being true are that CPU, bandwidth and file space may all be metered, and there may be no file access outside the WAR working cache.

Fast and non-blocking.

This state does not change for a running instance.

getNoGeneralFileAccess

public static boolean getNoGeneralFileAccess()

True if this instance should make no filesystem access outside the WAR cache.

vetoFileAccessIfNecessary

private static void vetoFileAccessIfNecessary()

Veto attempted actual or implied file access with an exception if not permitted.

Throws:

java.lang.UnsupportedOperationException - to veto such attempted access

_recheck

private static void _recheck()

This routine rechecks and refreshes the data items if necessary. This is called internally by all getter methods to lazily ensure that data items are up to date.

When this has finished it sets lastRecheck and possibly timestamp; if this fails to complete the update neither (but especially lastRecheck) may be set.

When running as a light-weight mirror without filesystem access, any configuration information is limited to that statically available, mainly from system properties, and so this returns immediately.

processXferKey

private static javax.crypto.SecretKey processXferKey(java.lang.String rawKeyText)

Return computed SecretKey from raw text, or null if none/invalid.

validateMirrorTag

private static java.lang.String validateMirrorTag(java.lang.String putativeMirrorTag)

Validate putative mirror tag; returns non-null only if input is valid and non-null.

getTimestamp

public static long getTimestamp()

Returns the timestamp of the current set of data values. Forces a (re)load/refresh of data if necessary. Zero indicates that no (file-based) values have (yet) been loaded.

The timestamp is the latest timestamp of any of the files read to make up the dataset.

getLocalPropsFilename

private static java.lang.String getLocalPropsFilename()

Name of props file for local config information; never null. It may be difficult for the system to run without this parameter set correctly or if the corresponding file is not correctly set up.

This will specify a file in the current working directory if none explicitly specified, and will deliver a warning if it cannot be found.

This might be bad news if the current working directory changes or is inappropriate or unhelpful, so set the system property explicitly if this is the case.

This does not check that the specified file actually exists.

We only use this internally; for example this may be inefficient to continually recheck and we don't necessarily want to publicise the actual location more than we must!

Throws:

java.lang.UnsupportedOperationException - if this is a light-weight mirror and thus without general filesystem access

getDataDir

public static java.lang.String getDataDir()

Get name of subdirectory for data files; never null. If the PNAME_DATA_DIR local property is set then it is used, else the CoreConsts.FS_DATA_ROOT value is used.

The returned value is not necessarily canonical to avoid prodding/waking any filesystems unnecessarily.

Throws:

java.lang.UnsupportedOperationException - if this is a light-weight mirror and thus without general filesystem access

getConfDir

public static java.lang.String getConfDir()

Get name of subdirectory for read-only conf files; never null. If the PNAME_CONF_DIR local property is set then it is used, else the CoreConsts.FS_CONF_ROOT value is used, canonicalised if possible to give more meaningful error message at least.

Throws:

java.lang.UnsupportedOperationException - if this is a light-weight mirror and thus without general filesystem access

getSimplePassfile

public static java.lang.String getSimplePassfile()

Get name of simple password file; null if none. There is no default for this; if it is not set then there will be no simple passwords.

If the path supplied is not absolute, it is made relative to the result of getConfDir().

getUploadDir

public static java.lang.String getUploadDir()

Get name of upload dir or null if none was specified. If the PNAME_UPLOAD_DIR local property is set then it is used.

If a directory is specified but is not absolute, it is taken to be relative to the directory the local properties are in.

Warns if the specified directory is not present, but returns the path anyway if a path was specified; the caller should check if it is usable.

Throws:

java.lang.UnsupportedOperationException - if this is a light-weight mirror and thus without general filesystem access

getUploadMaxBytesPerUser

public static int getUploadMaxBytesPerUser()

Get the Web server maximum-upload-space-per-user (bytes). Constrained to the range 0 to 1GB, default approx 10MB.

getUploadMaxBytesTotal

public static int getUploadMaxBytesTotal()

Get the Web server maximum-upload-space-total (bytes). Constrained to the range 0 to 1GB, default approx 10MB. Also constrained to be no smaller than uploadMaxBytesPerUser.

getPersistentStateDir

public static final java.lang.String getPersistentStateDir()

Get name of directory for storing local persistent state; never null. This is used for storing (especially valuable) data that should survive over many runs of the WAR/J2EE, but which is local to each WAR instance, such as usage stats, though may also be used by the master instance to persist user login data for example.

If the system property is not set, then we take this relative to the directory in which we look for the local properties.

Warns if the specified directory is not present.

Throws:

java.lang.UnsupportedOperationException - if this is a light-weight mirror and thus without general filesystem access

getMirrorTag

public static final java.lang.String getMirrorTag()

Get valid "cc-xxxx" mirror-tag name for this host; null if none. If this host is a mirror, it has a tag of the form cc-xxxxx where "cc" is a two-letter country code (as in a top-level ccTLD domain name, eg "uk" rather than "gb") where the host resides for Internet connectivity purposes, and "xxxx" is an alphanumeric tag indicating a particular host, and the whole string is less than 256 characters.

The entire tag is a lower-case printable ASCII (no whitespace) alphanumeric value.

This host should be reachable by the full name of "mirror-cc-xxxx.gallery.hd.org" for a tag of "cc-xxxx" and a main domain name of "gallery.hd.org".

This routine will not return an invalid/unsafe tag.

This routine does NOT attempt any DNS lookups or in other ways attempt to verify that the full mirror host name exists or is valid.

getThumbnailRelDir

public static final java.lang.String getThumbnailRelDir()

Get name of directory for storing off-line master thumbnails; never null. This is used for storing off-line computed thumbnails that should survive over many runs of the WAR/J2EE, mainly intended for read-only access by the master server, and write access by an off-line tool to create thumbnails in a less restrictive environment than the WAS.

If the system property is not set, then we take this relative to the directory in which we look for the exhibits.

If the property is set to "." then the exhibit directory is used.

getWEBSVR_MAX_EX_CACHE_BYTES

public static long getWEBSVR_MAX_EX_CACHE_BYTES()

Get the Web server maximum generic exhibit cache size (64-bit value, bytes); non-negative. Constrained to the range approx 0 to 1TB, default approx 2GB.

This value can be specified with a system property of the same name if no value is supplied in the properties file.

getServerSlowdownFactor

public static int getServerSlowdownFactor()

Get server-slowdown-factor; strictly positive. Defaults to 1, and no more than about, say, 100.

Is a factor by which the server under-reports its available capacity, etc.

When > 1 then this has the effect of:

• reducing aggressiveness of precaching,
• reducing reported available bandwidth, (and thus the chance of being picked as a user's "preferred" mirror)

etc, for sites with constrained or expensive or non-dedicated bandwidth.

getXferHMACKey

public static javax.crypto.SecretKey getXferHMACKey()

Get HMAC (secret) key used to protect data in transit; null if none. There is no default for this; if it is not set then there will be no data protection key.

This first attempts to initialised from the system property then any value in the local properties file overrides this.

The key returned is immutable.

The key should be handled with care, since it is sensitive data.

hasXferKey

public static boolean hasXferKey()

Returns true if there is an xfer key available, false otherwise.

getLowerPowerFileFlags

public static java.util.Set<java.io.File> getLowerPowerFileFlags()

Get immutable Set of low-power file-system flags; empty if none. The value returned is immutable.

getLowerPowerFileURIs

public static java.util.Set<java.net.URI> getLowerPowerFileURIs()

Get immutable Set of low-power file-system URIs; empty if none. The value returned is immutable.

getLightLoadMax

public static final float getLightLoadMax()

Get CPU light-load threshold; range ]0.0,1.0[ exclusive.

getHeavyLoadMin

public static float getHeavyLoadMin()

Get heavy-load threshold; strictly positive and always higher than the light-load-threshold.

fastStartMode

public static boolean fastStartMode()

If true then this instance should run in 'fast start' mode. Mainly used to influence whether such functions as mustConservePower() assume low power mode before testing external flags.

Useful for critical customer-facing machines where start-up latency should be minimised.

Enabled by setting the system property "org.hd.d.pg2k.faststart" to true or by being a light-weight cloud instance.

Defaults to false.

Is a method rather than a field to reduce class-initialisation circularity issues, so is possibly a little expensive to test continuously.