org.hd.d.pg2k.svrCore
Class AllExhibitProperties

java.lang.Object
  org.hd.d.pg2k.svrCore.AllExhibitProperties

All Implemented Interfaces:

java.io.ObjectInputValidation, java.io.Serializable, MemoryTools.Compactable

public final class AllExhibitProperties
extends java.lang.Object
implements java.io.Serializable, java.io.ObjectInputValidation, MemoryTools.Compactable

Set of all exhibit names and properties. This includes AllExhibitImmutableData and all other loadable and calculatable exhibit properties, including emergent statistical properties computed across the entire exhibit set.

This maintains a hash over all the loaded/static properties and the exhibit set so that changes should be easily caught. Computed state is excluded since it can be computed on demand, adds no extra information and/or includes contingent information and noise such as hints from system settings.

Designed to be efficient `on-the-wire' and in memory.

This object is logically immutable except that:

• Some information is computed on demand and may be discarded if out-of-date or in low-memory conditions or for serialisation.
• The not-yet-computed state of some expensive-to-compute items is made visible for more efficient use elsewhere in the system and this state may change according to the previous point.
• Some expensive computable information may be retained across serialisation having been computed in a different environment to the current one.

In particular, EPCM values will be (re)computed dynamically as needed.

This holds per-exhibit information such as timestamp, x/y dimensions, etc, and whole-collection data such as hot/cold lists, location maps, etc.

The hashCode() and equals() methods are implemented, based on the underlying data.

See Also:

Serialized Form

epgi

public final ExhibitPropsGlobalImmutable epgi

The immutable global exhibit properties; never null. This is made available as a field for maximum performance.

aeid

public final AllExhibitImmutableData aeid

The immutable exhibit data; never null. This is made available as a field for maximum performance.

loadedProps

private final java.util.Map<Name.ExhibitFull,ExhibitPropsLoadable> loadedProps

Logically immutable map from exhibit name to loadable props (ExhibitPropsLoadable); never null. If an entry for an exhibit is null then we return ExhibitPropsLoadable.EMPTY.

The keys here are canonicalised against aeid so that we don't have duplicate String values knocking about.

Note that keys may actually be String during deserialisation of older formats.

computedProps

private final java.util.Map<Name.ExhibitFull,ExhibitPropsComputable> computedProps

Map from full exhibit name to ExhibitPropsComputable properties; never null. This is a Hashtable to be completely thread-safe, even during serialisation where Hashtable.writeObject() is synchronized.

An instance of ExhibitPropsComputable is created on first demand and posted to the map. No lock is held while the computation is done, so redundant computations are possible, but a high degree of concurrency is available in return.

Entries are never removed from this map, and null is never posted as a value.

Defensively copied during deserialisation.

Note that keys may actually be String during deserialisation of older formats.

longHash

public final long longHash

The hash of all the data held; guaranteed non-negative. Depends on the timestamp of all the exhibit (aeid) and other loadable information held (eg map data).

Is guaranteed to be zero if the exhibit set is empty, and strictly positive otherwise.

Computable data (and hashNotChangedSince) is excluded from the hash since it is derived from other data that is included in the hash.

constructedAt

private final long constructedAt

The time that this object was constructed. Strictly positive unless there are no exhibits, in which case this is zero.

We keep this private.

hashNotChangedSince

public final long hashNotChangedSince

Time since which we claim that the exhibit set/data has not changed; guaranteed non-negative. If a new AllExhibitProperties object is constructed with the aid of an old one, and the longHash has not changed between the two, then this is copied from the old object (except that as a belt and braces measure it is ensured to be no older than the aeid.timestamp), else it is the same as the constructedAt time.

If a new AllExhibitProperties object is constructed from scratch each time then this will be the time that the object was constructed, and so we can regard this as a an `exhibits have not changed since' time.

Strictly positive unless there are no exhibits, in which case this is zero (to prompt early recomputation!).

ONE

private static final java.lang.Integer ONE

Frequently-used value instance Integer(1).

authorExhibitCounts

private transient volatile java.util.Map<java.lang.String,java.lang.Integer> authorExhibitCounts

Map from author initials (String) to exhibit count for that author (Integer). Only authors that have have one or more exhibits are stored in the map.

This map is immutable, and is computed on first use.

This map is transient (ie not stored in the persistent state), and volatile for thread-safe lockless access.

categoryExhibitCounts

private transient volatile java.util.Map<java.lang.String,java.lang.Integer> categoryExhibitCounts

Map from category [top-level directory] (String) to exhibit count for that category (Integer). Only categories that have have one or more exhibits are stored in the map.

This map is immutable, and is computed on first use.

This map is transient (ie not stored in the persistent state), and volatile for thread-safe lockless access.

GOODBAD_LIMIT

public static final float GOODBAD_LIMIT

Limit of goodness*correlation to consider something (eg category) significantly good or bad.

See Also:

Constant Field Values

dottedExtensionExhibitCounts

private transient volatile java.util.Map<java.lang.String,java.lang.Integer> dottedExtensionExhibitCounts

Map from dotted-file-extension (String) to exhibit count for that extension (Integer). Only extensions that have have one or more exhibits are stored in the map.

This map is immutable, and is computed on first use.

This map is transient (ie not stored in the persistent state), and volatile for thread-safe lockless access.

exhibitBytes

private transient volatile long exhibitBytes

Total size in bytes of all exhibits. A zero value is taken to mean "not computed", though in the case where there are no exhibits the resulting redundant recomputations will be cheap.

This is transient since it can be recalculated on demand.

This is volatile to allow safe lock-free access.

exhibitsByAttribute

private transient java.lang.ref.SoftReference<java.util.Map<java.lang.String,java.util.Set<Name.ExhibitFull>>> exhibitsByAttribute

A Map from attribute (String word) to a Set of full names of all exhibits with each attribute; never null though the referent may be. An exhibit can have a particular attribute either by having the attribute in the name (in the usual place at the end), or in some circumstances by the exhibit type or content being examined.

This is immutable, and is transient and recomputed on first use, and/or after being flushed due to memory shortage, being entirely derived from other information in this AEP.

We hold a private lock while recomputing this information to allow other processing to continue in parallel with this possibly slow task.

exhibitCountsByAttribute

private transient volatile java.util.Map<java.lang.String,java.lang.Integer> exhibitCountsByAttribute

A Map from attribute (String word) to a count of full names of all exhibits with each attribute; null until first use. Set/computed when exhibitsByAttribute is but not discarded as should require little memory.

_exhibitsByAttribute_lock

private final transient java.lang.Object _exhibitsByAttribute_lock

Private lock for manipulating exhibitsByAttribute; must use construction to create correctly.

mainWords

private transient java.lang.ref.SoftReference<java.util.Set<java.lang.String>> mainWords

Case-sensitive immutable cache (transient, possibly memory-sensitive) of Set of all exhibit main words; never null but referent may be. Computed on first use, and possibly recomputed after a memory shortage.

Accessed under the _mainWords_lock.

Used, for example, by the upload routines to warn of possibly-misspelt words.

_mainWords_lock

private final transient java.lang.Object _mainWords_lock

Private lock for manipulating mainWords; must use constructor to get set up correctly.

epcmMap

private final transient java.util.concurrent.ConcurrentMap<Name.ExhibitFull,ExhibitPropsComputableMutable> epcmMap

Private cache; Map from full exhibit name to ExhibitPropsComputableMutable. May or may not be serialised (may be transient to economise on space); never null if a constructor has been called.

A thread-safe Map implementation is used; one that allows decent concurrency, especially without locking.

COMPUTE_FULL_EPCM_RESULT_SPECULATIVELY_IN_BG

private static final boolean COMPUTE_FULL_EPCM_RESULT_SPECULATIVELY_IN_BG

If true, then we may speculatively spin off threads to compute full EPCM values even when stale ones OK.

See Also:

Constant Field Values

_vcCache

private final transient ExhibitPropsComputableMutableVoteCacheIF _vcCache

Private vote/correlation cache; never null after construction/deserialisation. This state could be serialised to save some recomputation on re-use.

This is thread-safe.

hashMD5ToNameSR

private transient volatile java.lang.ref.SoftReference<java.util.Map<ROByteArray,Name.ExhibitFull>> hashMD5ToNameSR

Immutable map from MD5 hash to exhibit full name; never null but referent my be null. Volatile for lock-free thread-safe access.

Transient; (re)computed on demand after deserialisation and/or after memory shortage.

hashMD5Err

private transient volatile java.util.Set<Name.ExhibitFull> hashMD5Err

Immutable Set of all exhibits with duplicate or missing MD5 hashes; null until computed on first use. Ideally this is empty; assumed to be small.

Volatile for lock-free thread-safe access.

Transient; (re)computed on demand after deserialisation.

_compacted

private transient volatile boolean _compacted

Set true once compact() has run the first time. Marked volatile for thread-safe lock-free access.

Marked transient since deserialisation may require some compaction to be re-done.

serialVersionUID

private static final long serialVersionUID

Our serial version...

See Also:

Constant Field Values

AllExhibitProperties

public AllExhibitProperties()

Construct a default (empty) AllExhibitProperties object with zero timestamp. This is operation is fast, and produces a small object, with no data but entirely consistent.

The hash of this object is 0, as is the timestamp.

AllExhibitProperties

public AllExhibitProperties(ExhibitPropsGlobalImmutable _epgi,
                            AllExhibitImmutableData _aeid,
                            java.util.Map<? extends java.lang.CharSequence,ExhibitPropsLoadable> _loadedProps,
                            java.util.Map<? extends java.lang.CharSequence,ExhibitPropsComputable> _computedProps)

Construct a new AllExhibitProperties object from scratch, checking for consistency and validity.

AllExhibitProperties

public AllExhibitProperties(AllExhibitProperties old,
                            ExhibitPropsGlobalImmutable _epgi,
                            AllExhibitImmutableData _aeid,
                            java.util.Map<? extends java.lang.CharSequence,ExhibitPropsLoadable> _loadedProps,
                            java.util.Map<? extends java.lang.CharSequence,ExhibitPropsComputable> _computedProps,
                            long unchangedSince)

Build a new AllExhibitProperties object, preserving any computed state possible from the old one. Because computing some information is very expensive, this can save significant time.

If old is null, the object is built from scratch.

Parameters:

_aeid - immutable core exhibit data

_loadedProps - map from exhibit name (as a CharSequence, eg a String, but preferably a Name.ExhibitFull) to loaded properties (ExhibitPropsLoadable); not all exhibits need have a mapping (in which case they are assumed no have no properties)

unchangedSince - if non-zero, the hashNotChangedSince value is set from this

canonicalise

public static AllExhibitProperties canonicalise(AllExhibitProperties in)

Canonicalise AEP, ie reconstruct new instance from old data; null only if input is. Can be useful for recomputing hashes, etc, from AEP instances created and serialised before an minor implementation change.

getExhibitPropsLoadable

public ExhibitPropsLoadable getExhibitPropsLoadable(Name.ExhibitFull exhibitName)

Get loadable props for exhibit; never null. This is completely thread-safe as loadedProps is read-only/immutable once construction/deserialisation is complete.

Returns:

non-null instance; ExhibitPropsLoadable.EMPTY if none

getExhibitPropsLoadableMap

public java.util.Map<Name.ExhibitFull,ExhibitPropsLoadable> getExhibitPropsLoadableMap()

Get immutable map from full exhibit name to ExhibitPropsLoadable; never null. May omit entries that map to ExhibitPropsLoadable.EMPTY.

getExhibitPropsComputable

public ExhibitPropsComputable getExhibitPropsComputable(Name.ExhibitFull exhibitName)

Get computable (immutable) props for exhibit; if none available then returns null.

getExhibitPropsComputableMap

public java.util.Map<Name.ExhibitFull,ExhibitPropsComputable> getExhibitPropsComputableMap()

Get immutable map from full exhibit name to ExhibitPropsComputable; never null.

getAuthorExhibitCounts

public final java.util.Map<java.lang.String,java.lang.Integer> getAuthorExhibitCounts()

Get a map from author initials (String) to exhibit count for that author (Integer); never null. Only authors that have have one or more exhibits are stored in the map.

getCategoryExhibitCounts

public final java.util.Map<java.lang.String,java.lang.Integer> getCategoryExhibitCounts()

Get a map from category [top-level directory] (String) to exhibit count for that category (Integer); never null. Only categories that have have one or more exhibits are stored in the map.

This map is immutable, and is computed on first use.

isCategoryGood

public java.lang.Boolean isCategoryGood(java.lang.String categoryDir,
                                        BasicVarMgrInterface vars,
                                        boolean force)
                                 throws java.io.IOException

Find out if a category is rated "good"/popular or not. Returns TRUE if rated good, FALSE if bad, null if not significantly either or if not known.

Parameters:

categoryDir - the initial directory component of an extant exhibit

force - if true may force (expensive) computation to give a more accurate answer, else may return a more approximate or stale answer, or none at all (null)

Throws:

java.io.IOException

getDottedExtensionExhibitCounts

public final java.util.Map<java.lang.String,java.lang.Integer> getDottedExtensionExhibitCounts()

Get a map from dotted-file-extension (String) to exhibit count for that author (Integer); never null. Only extensions that have have one or more exhibits are stored in the map.

This map is immutable, and is computed on first use.

getLegalSuffixes

public static java.lang.String[] getLegalSuffixes()

Returns set of available extensions, sorted case insensitively. This may be more than actually appear in the data set (and this result does not depend on the data set, in fact). This result contains no duplicates.

These are suffixes starting with `.' eg ``.jpg''.

Never returns null.

getTotalExhibitBytes

public long getTotalExhibitBytes()

Get total size in bytes of all exhibits; non-negative. Computed on first use.

getExhibitsByAttribute

public java.util.Map<java.lang.String,java.util.Set<Name.ExhibitFull>> getExhibitsByAttribute()

Returns a Map from attribute (String word) to a Set of full names of all exhibits with each attribute; never null. An exhibit can have a particular attribute either by having the attribute in the name (in the usual place at the end), or in some circumstances by the exhibit type or content being examined.

The key set is from ExhibitAttrUtils.getAttrWords().getAttrWordsSortedSet().

The returned value is immutable (and the keys and values are immutable) and is never null.

Attributes which no exhibits have are present in the Map with an empty Set as the mapped-to value, not null.

getExhibitCountsByAttribute

public java.util.Map<java.lang.String,java.lang.Integer> getExhibitCountsByAttribute()

Returns an immutable Map from attribute (String word) to a Set of full names of all exhibits with each attribute; never null. An exhibit can have a particular attribute either by having the attribute in the name (in the usual place at the end), or in some circumstances by the exhibit type or content being examined.

The key set is from ExhibitAttrUtils.getAttrWords().getAttrWordsSortedSet().

The returned value is immutable (and the keys and values are immutable) and is never null.

Attributes which no exhibits have are present in the Map with zero as the mapped-to value, not null.

getMainWords

public java.util.Set<java.lang.String> getMainWords()

Get case-sensitive Set of all main words (String values) in extant exhibits; never null. Result will be empty if there are no exhibits.

Computed on demand (and cached) under a private lock as this may be slow and we do not want to waste CPU time on redundant computations.

recoverOldExhibitPropsComputableMutableData

public void recoverOldExhibitPropsComputableMutableData(AllExhibitProperties oldAEP,
                                                        SimpleLoggerIF logger)

Recover EPCM (and possibly other details) from old AEP. This can save a great deal of recomputation after an AEP update.

Will only use details from old AEP:

• That match current exhibits.
• That are not trivially stale.
• If we have no such data at all (to prevent later overwriting of newly-computed data).

This should not be called once threads other than the caller have access to this AEP since after performing our initial checks there is a danger of races that might leave old data in place.

This does not compute any new entries.

getExhibitPropsComputableMutable

public ExhibitPropsComputableMutable getExhibitPropsComputableMutable(Name.ExhibitFull exhibitName)

Get ExhibitPropsComputableMutable for specified exhibit; null if no such exhibit or no result yet computed. Returns whatever is currently available as-is and never forces any (re)computation of values nor blocks (on any externally-visible lock).

Parameters:

exhibitName - full name of exhibit for which data is desired; never null

getExhibitPropsComputableMutable

public ExhibitPropsComputableMutable getExhibitPropsComputableMutable(Name.ExhibitFull exhibitName,
                                                                      boolean allowStale,
                                                                      GenProps gp,
                                                                      SimpleExhibitPipelineIF dataSource,
                                                                      ScorerCacheIF scorers,
                                                                      long maxTimeMs)

Get ExhibitPropsComputableMutable for specified exhibit within given time; null if no such exhibit or out of time. If the "allowStale" parameter is false, the system will make best efforts to compute an accurate, non-stale value. However, this may not be possible for a number of contingent reasons, such as transient I/O errors.

Equally, if the "allowStale" flag is true, there may be a non-stale value cached that can be returned.

So the "allowStale" parameter should be regarded as a strong hint at most.

This will spend up to the specified amount of time attempting to (re)compute the value if needed, else will return what (if anything) is already available. If there is a shortage of resources (eg threads) then this may not succeed in (re)calculation.

Parameters:

exhibitName - full name of exhibit for which data is desired; never null

allowStale - allows "approximate" or "stale" version to be returned if need be, ie does not force re-computation of new value, useful if site is busy or time is short

gp - system properties; never null and not default/empty for non-stale result

dataSource - source of event and exhibit (meta) data; must not be null if a non-stale result is desired

scorers - source of exhibit-content based scoring; must not be null if a non-stale result is desired

maxTimeMs - maximum time (ms) to spend computing result; strictly positive

Returns:

non-null result if exhibit exists and we don't run out of time or other resources; possibly stale/approximate if allowStale==true or gp/dsb are empty/null

getExhibitPropsComputableMutable

public ExhibitPropsComputableMutable getExhibitPropsComputableMutable(Name.ExhibitFull exhibitName,
                                                                      boolean allowStale,
                                                                      GenProps gp,
                                                                      SimpleExhibitPipelineIF dataSource,
                                                                      ScorerCacheIF scorers)

Get ExhibitPropsComputableMutable for specified exhibit; null if no such exhibit else never null. If the "allowStale" parameter is false, the system will make best efforts to compute an accurate, non-stale value. However, this may not be possible for a number of contingent reasons, such as transient I/O errors.

Equally, if the "allowStale" flag is true, there may be a non-stale value cached that can be returned.

So the "allowStale" parameter should be regarded as a strong hint at most.

Parameters:

exhibitName - full name of exhibit for which data is desired; never null

allowStale - allows "approximate" or "stale" version to be returned if need be, ie does not force re-computation of new value, useful if site is busy or time is short

gp - system properties; never null and not default/empty for non-stale result

dataSource - source of event and exhibit (meta) data; must not be null if a non-stale result is desired

scorers - source of exhibit-content based scoring; must not be null if a non-stale result is desired

Returns:

non-null result if exhibit exists; possibly stale/approximate if allowStale==true or gp/dsb are empty/null

updateVoteCache

public void updateVoteCache(SimpleVariablePipelineIF vars,
                            boolean noTimeLimit)
                     throws java.io.IOException

Bring up-to-date any vote/correlation data. Useful to call from a background thread from time to time in order that all correlations will be precomputed when required.

Parameters:

vars - non-null handle on system variables

noTimeLimit - if true, this will try to complete the work in one go

Throws:

java.io.IOException

_createNewEpcmMap

private static java.util.concurrent.ConcurrentHashMap<Name.ExhibitFull,ExhibitPropsComputableMutable> _createNewEpcmMap(int capacity)

Create new epcmMap; never null. This is where we set policy on map segments, load factor, etc.

We use a concurrency-friendly, lock-free Map for lookup and update speed. We don't expect lots of concurrent writes, so a few segments (fewer than say 10) should be OK.

getLocation

public Location.Base getLocation(Name.ExhibitFull exhibitName)

Get exhibit Location information, specific or generic, if any; Location.NONE if none, never null. This tries first for a "specific" location for this exhibit, then tries for a generic location from the pattern-matching mechanism.

Parameters:

exhibitName - the full name of the exhibit; must not be null

Returns:

the location information for this exhibit; NONE if none

getHashMD5ToName

public java.util.Map<ROByteArray,Name.ExhibitFull> getHashMD5ToName()

Get immutable map from MD5 hash to valid exhibit full name; never null. Computed on first use.

Sets internal hashMD5ToNameSR and hashMD5Err on first use as a side-effect. Note that hashMD5Err is never cleared again to avoid races in getHashMD5Err().

getHashMD5Err

public java.util.Set<Name.ExhibitFull> getHashMD5Err()

Get immutable Set of all exhibits with duplicate or missing MD5 hashes; never null. Ideally this is empty.

select

public Name.ExhibitFull[] select(AllExhibitProperties.AEPFilter f,
                                 java.util.List<Name.ExhibitFull> initialNames,
                                 int stopAfter)

Filter results optionally starting with an input set of exhibit names; never null. This returns a List of full exhibit name results, starting with the given input set of names, or all exhibits (set up in smart-sorted order) if null is passed in.

This search may be parallelised.

Note that is its up to the filters to decide whether to call getExhibitPropsComputable() to guarantee correct answers or getExhibitPropsComputableIfPresent() to guarantee fast answers.

If the stopAfter parameter is greater than zero, then this routine will stop as soon as possible after stopAfter results, and will then truncate to exactly stopAfter results.

The results array is sorted in a `smart'ish way.

The filter argument must not be null.

Parameters:

f - filter to apply; never null

initialNames - initial set of names in non-null, else all names

stopAfter - +ve number of results to stop after, or zero for no limit

Returns:

non-null, possibly empty, smart-sorted full exhibit names

toString

public java.lang.String toString()

Generate human-readable summary of this instance's internal state.

Overrides:

toString in class java.lang.Object

hashCode

public int hashCode()

Returns a hash code value for the object; derived from the longHash. Guaranteed zero if there are no exhibits, non-zero otherwise.

Overrides:

hashCode in class java.lang.Object

Returns:

a hash code value for this object.

See Also:

Object.equals(Object), Hashtable

equals

public boolean equals(java.lang.Object obj)

Indicates whether some other object is "equal to" this one; the underlying data is the same if true. The hashNotChangedSince value is excluded from the comparison.

Loaded data is included in this notion of equality; computed/derived data is excluded. The construction time and "unchanged since" values are also excluded.

This routine is likely to be very slow, especially to return a "true" result, ie fully verify that two instances are identical in all significant aspects.

Overrides:

equals in class java.lang.Object

Parameters:

obj - the reference object with which to compare.

Returns:

true if this object is the same as the obj argument; false otherwise.

See Also:

Boolean.hashCode(), Hashtable

compact

public void compact()

Compact the internal representation of this instance (and its sub-objects) if possible. This has no effect on the logical content of this instance in-memory or serialised, is guaranteed to be safe to run concurrently with other uses of this instance (and will take any locks as needed to work incrementally), and may do nothing but consume some CPU cycles.

This may be able to convert some state to a more memory-efficient representation after construction or deserialisation, and is suitable to call in a background thread.

We don't prevent multiple concurrent calls to this routine, since they are at worst wasteful of CPU but not unsafe.

(Should not be called on a partially-constructed object.)

Specified by:

compact in interface MemoryTools.Compactable

getCompactableInstanceName

public java.lang.String getCompactableInstanceName()

For Compactable interface, or null if none.

Specified by:

getCompactableInstanceName in interface MemoryTools.Compactable

estimateSerialBytes

public int estimateSerialBytes()

Estimate of bytes taken up by uncompressed serialised form; strictly positive. This is only a hand-waving empirical estimate, and may not actually inspect the internal state much.

The result is capped at Integer.MAX_VALUE.

readResolve

protected java.lang.Object readResolve()

Deserialise: use constructor for validation, defensive copying, renormalisation, etc.

validateObject

public void validateObject()
                    throws java.io.InvalidObjectException

Validate fields/state. Called in the constructor and possibly after de-serialising.

Barf if something bad is found. (Maybe allow some extra info in debug version.)

Specified by:

validateObject in interface java.io.ObjectInputValidation

Throws:

java.io.InvalidObjectException

_updatePeerStats

public void _updatePeerStats(ExhibitDataSimpleCache exhibitDataSimpleCache,
                             java.lang.String peer,
                             boolean fetchSuccessful,
                             long timeTaken)

Update data-transfer stats for the given peer. A failed fetch is treated as a very slow access, so that a failing peer compares numerically unfavourably with reliable peers.

Parameters:

exhibitDataSimpleCache - TODO

peer - TODO

fetchSuccessful - TODO

timeTaken - TODO