ExhibitDataSimpleCache (DHD Multimedia Gallery V1.50.55 Javadoc)

Overview

Package

Class

Tree

Deprecated

Index

Help

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

org.hd.d.pg2k.svrCore.datasource
Class ExhibitDataSimpleCache

java.lang.Object
  org.hd.d.pg2k.svrCore.datasource.ExhibitDataSimpleCache

All Implemented Interfaces:: SimpleExhibitPipelineIF, BasicVarMgrInterface, SimpleVariablePipelineIF

public final class ExhibitDataSimpleCache
extends java.lang.Object
implements SimpleExhibitPipelineIF
extends java.lang.Object
implements SimpleExhibitPipelineIF

Exhibit pipeline cache stage. This performs transparent persistent cacheing of exhibit data and variables.

The presence of an instance of this stage upstream of a tunnel or other potentially slow/expensive/unreliable data source should, for normal data access (eg sequential download of exhibits) significantly reduce upstream bandwidth requirements and reduce downstream latency by answering requests from local cache.

This also attempts to shelter its downstream callers/users from I/O errors upstream, by fulfilling requests from cache, or, when synchronous calls upstream have to be made, transforming some requests and replies into async forms where possible.

When caching exhibit data this class only does so as a continuous prefix from offset zero; other (random) accesses may have to read-through the cache.

This cache is also able to pre-cache data likely to be valuable, such as thumbnails and the initial portions of exhibits, though this will only be attempted if the cache appears to be in use. Bandwidth/resource consumption used by pre-cacheing are regulated. This cache regards thumbnails and meta-data as precious, and tries not to let them go once collected and cached because reasonable application performance will often depend on fast access to these data.

This attempts to cache data well enough that, especially if aggressive (pre)cacheing is available and the cache area is large enough, almost no reference should be needed to the backend server in response to a data request on the pipeline except in response to the tail end of very long exhibits; all requests are answered from the local cache where possible. Ideally, it should be possible for the backend server to go down altogether and have the front end still provide a high degree of functionality. The front-end and back-end are almost completely decoupled in this cache design.

This includes limited in-memory cacheing, in many cases using soft references to allow peaceful coexistence with other (potentially heavy) users of memory.

This class relies mainly on the asynchronous calls to poll() to fetch meta-data updates such as GenProps and AllExhibitImmutableData. These happen under a private lock and do not block cache access much or at all.

The full lock ordering (where multiple locks need to be taken at once) is:

ExhibitDataSimpleCache.rwl
ExhibitDataSimpleCache.metaData

This cache may serialise access to raw exhibit data (and may serialise accesses to back-end resources too). No two live instances of this class should refer to the same cache directory at once else madness and corruption will almost certainly break out.

On disc, the files are some prefix of the full exhibit, retrieved if possible in MAX_TRANSFER_CHUNK_SIZE chunks. They are touched every time accessed or updated and the timestamps can therefore be used as the basis of an LRU cache. We expect almost all access to be sequential, starting at the beginning.

We statisfy requests wholy within already-cached data immediately, and will extend (up to the limit) by up to one chunk each time by downloading from the server to statisfy requests just beyond the current end. Requests starting well beyond the current end of cache are punted directly to the server, which is rather ugly and slow, but there we go.

This cache only considers general precaching until the low-water mark is reached, and by default only deletes existing entries if it has to in order to statisfy an incoming cacheable request. This means that stale entries for deleted/renamed exhibits may persist for a while, but this is mainly harmless.

This maintains a bi-directional cache of variable values and updates, and also some running parameters of the cache may be read as variables.

This cache supports a limited amount of peer-to-peer (P2P) data transfers to reduce load on the master. The general policy is that any synchronous (and thus presumably time-sensitive) data request from an end user that cannot be satisfied from local cache is satisfied upstream from the master. Asynchronous data fetches, such as read-ahead and pre-cache activity, can be fetched P2P. Also if the master fails or is unavailable then it may be acceptable to use P2P if loops/cycles can be avoided.

TODO: re-analyse/reduce locking and possibly avoid locks on the metaData object.

Nested Class Summary
`private static class`	`ExhibitDataSimpleCache.CachedFile` Object representing one (partially) cached file on disc.
`private static class`	`ExhibitDataSimpleCache.MetaData` Cache meta-data class.
`private static class`	`ExhibitDataSimpleCache.TIMSoftReference` Enqueueable SoftReference for cached-in-memory thumbnails.

Nested classes/interfaces inherited from interface org.hd.d.pg2k.svrCore.datasource.SimpleExhibitPipelineIF
`SimpleExhibitPipelineIF.PropsKey`

Field Summary
`private AllExhibitProperties`	`_AEP` Cached AllExhibitProperties; never null.
`private java.util.concurrent.Future<?>`	`_AEPPoller` Future for task fetching _AEP in the background; never null after construction.
`private boolean`	`_aggressive` Flag to adjust the aggressiveness of the cache; by default not aggressive.
`private java.util.Set<java.lang.String>`	`_bestExhibits` Contains the full exhibit names of the "best" few exhibits for enhanced precacheing; never null.
`private long`	`_checkMetaData_notBefore` Time before which next _checkMetaData() call should not be initiated.
`private java.util.concurrent.locks.ReentrantLock`	`_cMD_lock_` Private lock for _checkMetaData() to avoid starting more than one thread.
`private java.util.concurrent.locks.ReentrantLock`	`_gAEP_lock` Private lock for _getAllExhibitProperties()/constructor to prevent re-entry and multiple concurrent AEP fetches.
`private static int`	`_gXL_offset` Stack offset for _getXXXXLock() to find caller's stack frame.
`private long`	`_handleSysVars_evSave` Last time we saved (any) event histories, private to _handleSysVars(), initially zero.
`private java.lang.Long`	`_handleSysVars_lastFetch` Last time we flushed/fetched variables, private to _handleSysVars().
`private java.util.Iterator<java.lang.String>`	`_iCMEE_iterator` Private iterator over all cached full exhibit names for _incrCheckMRUExhibitEntries().
`private long`	`_lastPollAEP` Last time we polled for AllExhibitImmutableData; initially zero.
`private long`	`_lastPollGp` Last time we polled for genProps.
`private long`	`_lastPollGSp` Last time we polled for genSecProps.
`private long`	`_noMorePrecacheUntil` Time before which we will not do more precacheing.
`private java.lang.Long`	`_precacheExhibitHash` Indicator for which image set we are working on.
`private java.util.Iterator<java.lang.String>`	`_precacheIterator` An iterator over a snapshot of all exhibit names.
`private java.util.concurrent.locks.ReentrantLock`	`_preCacheLock` Precache lock to prevent multi-threaded precache runs.
`private long`	`_saveMetaData_notBefore` Time before which not to to save metaData again; private to _cleanAndSaveMetaData().
`private java.util.concurrent.ConcurrentMap<java.lang.String,java.lang.Object>`	`_thumbnailsInMemory` Private in-memory cache of deserialised thumbnails; never null.
`private java.lang.ref.ReferenceQueue<ExhibitThumbnails>`	`_tIM_deadRefQueue` Queue of zapped thumbnail SoftReference entries.
`private java.util.concurrent.locks.ReentrantLock`	`_tIMT_lock` Lock to avoid more than one active thread in _thumbnailsInMemoryTidy().
`private boolean`	`_userRequestedDataFromCache` Set true when a user requests data from the cache.
`private static boolean`	`ALLOW_DATA_FETCH_FROM_PEERS` If true, allow us to try fetching some exhibit data from peers rather than master.
`private static boolean`	`ALLOW_SYNC_TN_FETCH` If true then allow missing thumbnails to be fetched synchronously at the risk of blocking for extened periods.
`private java.util.Map<java.lang.String,java.lang.Long>`	`altDataSourceRating` Thread-safe Map from mirror ID to strictly-positive rating with "" for master; never null.
`private static boolean`	`ASSUME_LOADED_METADATA_OK` If true, assume that newly-loaded meta-data at is OK at start-up until proven otherwise.
`private static java.lang.String`	`CACHE_BASE_DIR` Base dir within cache dir for all our exhibit data.
`private static java.lang.String`	`CACHE_EXAUX_PREFIX` The prefix for all aux files associated with an exhibit file.
`private static java.lang.String`	`CACHE_EXAUX_TIMESTAMP_KW` The keyword for the file containing the (decimal) exhibit timestamp.
`private static java.lang.String`	`CACHE_EXAUX_TN_KW` The keyword for the file containing the serialised thumbnails object.
`private static java.lang.String`	`CACHE_EXDATA_DIR` Base dir within cache dir for all our raw exhibit content data.
`private static java.lang.String`	`CACHE_EXPROPS_FILENAME` Name of file in which to persist immutable exhibit data.
`private static java.lang.String`	`CACHE_METADATA_FILENAME` Name of file in which to persist cache meta data.
`private java.io.File`	`cacheDir` The cache dir, else null.
`private static boolean`	`CALC_MISSING_EPCM_DURING_PRECACHE` If true then try to at least partially compute EPCM while precacheing.
`private boolean`	`destroyed` Set true once destroy() is called; never set false again.
`private static int`	`DISC_RECHECK_INTERVAL_MS` Approximate minimum interval between rechecks of on-disc cache.
`private java.util.concurrent.ExecutorService`	`discardableReadAheadTaskThreadPool` Shared thread pool for I/O-bound activities for Web-driven requests.
`private static java.lang.String`	`EVENT_HISTORY_DIR` Base dir within cache dir for all our event history data.
`private AllExhibitProperties.ExhibitDataSource`	`exhibitDataSource` An AllExhibitProperties.ExhibitDataSource wrapping ourselves; never null.
`private static int`	`FALLBACK_MIN_CACHE_SIZE` Minimum cache size to allow if GenProps is not set (bytes).
`private static boolean`	`FORCE_IMMEDIATE_SAVE_ON_EXPANDED_METADATA` If true, synchronously force a save of meta-data each time we add a new entry at least.
`private GenProps`	`genProps` Our record of the current GenProps; never null.
`private java.util.Properties`	`genSecProps` Our record of the current GenProps; never null.
`private static java.lang.String`	`KEY_debugFlag_P2P_BLOCKXFER` Key in generic props of P2P-profiling flag.
`private SimpleLoggerIF`	`logger` Our local logger; never null.
`private static float`	`LOW_WATER_FRACTION` Fraction of max cache size that is the low-water mark.
`private static java.lang.String`	`MASTER_FAKE_TAG` Fake tag we use to indicate a fetch from the master via the pipeline.
`private static int`	`MAX_BEST_EX_PRECACHED` Maximum number of "best" exhibits to get enhanced precaching; non-negative.
`private int`	`MAX_dPC_BACKOFF_TIME` Max time _doPreCache() has to sleep for (ms).
`private int`	`MAX_dPC_SPIN_TIME_MS` Maximum time that _doPreCache() can spend in one go (ms).
`private static int`	`MAX_EXTD_TRANSFER_CHUNK_SIZE` Maximum extended exhibit data transfer chunk size (bytes); strictly positive.
`private static int`	`MAX_LOCK_ATTEMPTS` Maximum number of consecutive attempts to obtain lock for read or write; strictly positive.
`private static int`	`MAX_QUEUED_TN_FETCHES` Maximum number of async thumbnail fetches to queue; strictly positive.
`private static int`	`MAX_REMOTE_FETCH_TO_MAKE_THUMBNAIL` The maximum number of bytes we will force a transfer of to force an immediate thumbnail generation.
`private static int`	`MAX_THREADS_aTWQ` Maximum number of threads that may run in _asyncTNFetch() and other local discardable data read-ahead tasks; strictly positive.
`private static int`	`MAX_TRANSFER_CHUNK_SIZE` Maximum (normal) exhibit data transfer chunk size (bytes); strictly positive.
`private ExhibitDataSimpleCache.MetaData`	`metaData` In-memory copy of whole-cache meta-data; never null.
`private static int`	`METADATA_MIN_SAVE_INTERVAL_MS` Approximate minimum interval between saves of the metadata; strictly positive.
`private static int`	`MIN_AEP_POLL_TIME_UNTIL_LOADED_MS` Minimum time before attempting to poll again for AEP while we don't have a real one loaded (ms).
`private static int`	`NEARLY_TOP_FACTOR` Factor/multiplier of peers worse than top that will be considered for fetches routinely; strictly positive.
`private static boolean`	`ORPHANED_EXHIBIT_EXPIRY_ALLOWED` If true then we may purge cached data for exhibits that appear to have been deleted (or renamed).
`private static long`	`ORPHANED_EXHIBIT_MIN_UNUSED_TIME_MS` The minimim time before we will pre-emptively purge orphaned cache entries (ms); strictly positive.
`private int`	`P2P_NEXT_BEST_FRAC` Fraction of the time to choose a 2nd-tier peer rather than the best peer; strictly positive.
`private int`	`P2P_RND_FRAC` Fraction of the time to pick a peer completely at random; strictly positive.
`private static boolean`	`PEER_SELECTION_CAUTIOUS` If true then use a cautious strategy to select a peer to talk to.
`private static int`	`PEER_STATS_TC` Time-constant for updating peer fetch time value; strictly positive.
`private static int`	`PEER_STATS_UNKNOWN_MS` Default rating/time (ms) for "unknown" data source/mirror/peer; strictly positive.
`private static boolean`	`PREFER_PEERS_TO_MASTER_FOR_DATA_PREFETCH` If true, deter use of master where peers are available.
`private java.util.concurrent.locks.ReentrantReadWriteLock`	`rwl` The read/write lock for the whole cache except system variables; never null.
`static java.lang.String`	`SCGNAME_CACHE_VALIDATION` General stats event name: an exhibit in the cache was fully validated against checksums, etc.
`static java.lang.String`	`SCGNAME_CACHE_VALIDATION_PART` General stats event name: an exhibit in the cache was partially validated against checksums, etc.
`static java.lang.String`	`SCGNAME_CACHEADD` General stats event name: an exhibit was added to the cache.
`static java.lang.String`	`SCGNAME_CACHEEVICTLRU` General stats event name: an exhibit was evicted from the cache in LRU order.
`static java.lang.String`	`SCGNAME_CACHERAWDATAHIT` General stats event name: cache raw data read hit.
`static java.lang.String`	`SCGNAME_CACHERAWDATAMISS` General stats event name: cache raw data read miss.
`static java.lang.String`	`SCGNAME_CACHEREM` General stats event name: an exhibit was removed from the cache.
`static java.lang.String`	`SCGNAME_CACHEREM_CORRUPT` General stats event name: a corrupt exhibit was removed from the cache.
`static java.lang.String`	`SCGNAME_CACHETNHIT` General stats event name: on-disc cache hit for thumbnail.
`static java.lang.String`	`SCGNAME_CACHETNMEMHIT` General stats event name: in-memory cache hit for thumbnail.
`static java.lang.String`	`SCGNAME_CACHETNMISS` General stats event name: in-memory cache hit for thumbnail.
`static java.lang.String`	`SCGNAME_DATAFETCHFROMPEER_PREFIX` General stats event name: fetched a data block from a peer.
`static java.lang.String`	`SCGNAME_EXDATAREQIN` General stats event name: incoming request for exhibit data.
`static java.lang.String`	`SCGNAME_EXDATAREQINDC` General stats event name: incoming request for exhibit data with "dontCache" flag set.
`static java.lang.String`	`SCGNAME_EXTHUCREATED` General stats event name: created thumbnails locally from cached data.
`static java.lang.String`	`SCGNAME_EXTHUREQIN` General stats event name: incoming request for exhibit thumbnails.
`static java.lang.String`	`SCGNAME_EXTHUREQINDC` General stats event name: incoming request for exhibit thumbnails with "dontCreate" flag set.
`static java.lang.String`	`SCGNAME_MDSAVE` General stats event name: an exhibit was evicted from the cache in LRU order.
`static java.lang.String`	`SCGNAME_PRECACHEERROR` General stats event name: errors encountered during precaching.
`static java.lang.String`	`SCGNAME_PRECACHEEXAMINED` General stats event name: exhibits examined for precaching.
`static java.lang.String`	`SCGNAME_PRECACHEEXDATABLOCK` General stats event name: exhibit data block precached.
`static java.lang.String`	`SCGNAME_PRECACHERESTART` General stats event name: restarted scanning all exhibits for precaching.
`static java.lang.String`	`SCGPREF_PRECACHEEXDATABLOCKFETCHTIME` General stats event name prefix: exhibit data block precache (succesful) fetch time (log2 ms).
`static java.lang.String`	`SCGPREF_PRECACHEEXDATABLOCKSRC` General stats event name prefix: exhibit data block precache source (if not from master/upstream).
`static java.lang.String`	`SCGPREF_PRECACHEEXDATABLOCKSRCERR` General stats event name prefix: exhibit data block precache source for error (if not from master/upstream).
`private SimpleExhibitPipelineIF`	`source` The upstream data source; never null.
`private StatsLogger.StatsConfig`	`statsIDSCGEN` The stats set to which we log general cache behaviour.
`private static boolean`	`STORE_EXPROPS_GZIPED` If true, store exprops data (and cache metadata) GZIPed to possibly save space and I/O time.
`private static boolean`	`THUMBNAIL_ACCESS_UPDATES_ACCESS_TIMESTAMP` If true then accessing a thumbnail marks its exhibit as accessed.
`private static boolean`	`UPDATE_EPCM_DURING_PRECACHE` Deprecated. Use `CALC_MISSING_EPCM_DURING_PRECACHE` instead
`private static int`	`VAR_CACHE_HOLD_TIME_MS` Variable flush/retrieval interval (ms); strictly positive.
`private PipelineVarMgr`	`varMgr` Manages our local cache of variables, etc; never null.
`private static long`	`WAIT_BETWEEN_THUMBNAIL_REPEAT_FETCHES_MS` Wait time between attempts to fetch or generate thumbnails (ms).

Fields inherited from interface org.hd.d.pg2k.svrCore.datasource.SimpleExhibitPipelineIF
`MAX_USER_READ_SIZE`

Constructor Summary
`private`	`ExhibitDataSimpleCache(SimpleExhibitPipelineIF dataSource, java.io.File cacheDir, SimpleLoggerIF logger)` Wrap a new cache instance around a data source.

Method Summary
`private void`	`_asyncTNFetch(java.lang.String exhibitName)` Attempt to asynchronously fetch/create thumbnail that we have failed to return to the user.
`private void`	`_checkMetaData()` Initiates, in the background, a check of the in-memory cache meta data against disc.
`private void`	`_cleanAndSaveMetaData(boolean force)` Saves the cache metadata if needed.
`private void`	`_doCacheDataValidityTest(AllExhibitProperties aep, ExhibitStaticAttr esa)` Partly check the cache data (including metadata, tns, etc) for validity.
`private void`	`_doPreCache(GenProps gp)` Routine to do incremental pre-cacheing.
`private void`	`_getAllExhibitProperties_postUpdate(AllExhibitProperties new_AEP, boolean inCons)` Accepts the value posted from the background thread.
`private void`	`_getAllExhibitProperties()` Attempts to get all exhibit properties if our cached copy may be stale.
`private void`	`_getExhibitDataFromUpstreamToPrecache(ExhibitStaticAttr esa, AllExhibitImmutableData aeid, GenProps gp, long start, int len, boolean forceIt)` Try to extend cached data for given exhibit.
`private void`	`_getGenProps()` Attempts to get sysprops if our cached copy may be stale.
`private void`	`_getGenSecProps()` Attempts to get gensecprops if our cached copy may be stale.
`private static void`	`_getReadLock(java.util.concurrent.locks.ReentrantReadWriteLock rwl, java.lang.String detail, SimpleLoggerIF logger)` Get a cache read lock, complaining/aborting if we have to wait for a long time.
`private ExhibitThumbnails`	`_getThumbnails(java.lang.String name, boolean create, boolean allowSyncFetch)` Get the thumbnails for an exhibit.
`private static void`	`_getWriteLock(java.util.concurrent.locks.ReentrantReadWriteLock rwl, java.lang.String detail, SimpleLoggerIF logger)` Get the cache write lock, complaining/aborting if we have to wait for a long time.
`private void`	`_handleSysVars(boolean force)` Handle (update, sync, persist) system variables as required.
`private void`	`_incrCheckMRUExhibitEntries(java.util.Set<java.lang.String> done)` Incrementally check cached exhibits for integrity.
`private void`	`_incrPurgeOrphanedExhibits()` Do incremental purge of orphaned cache entries conditions are right.
`private java.lang.String`	`_pickPeer(java.util.Set<java.lang.String> activeMirrors)` Pick one of the supplied peers to attempt to fetch the exhibit data from; never null.
`private void`	`_removeCorruptData(ExhibitDataSimpleCache.CachedFile cf)` Remove cached exhibit data identified as corrupt.
`private void`	`_save_AEP()` Save AllExhibitProperties to disc.
`private void`	`_thumbnailsInMemoryTidy()` Incrementally purge stale entries from the in-memory thumbnail cache.
`private boolean`	`_updateOneExhibit(ExhibitStaticAttr esa, GenProps gp, AllExhibitProperties aep, boolean forceDataFetch)` Update one exhibit incrementally during precaching.
`private void`	`_updatePeerStats(java.lang.String peer, boolean fetchSuccessful, long timeTaken)` Update data-transfer stats for the given peer.
`static ExhibitDataSimpleCache`	`cacheFactory(SimpleExhibitPipelineIF dataSource, java.io.File cacheDir, SimpleLoggerIF logger)` Get the an instance copy of this class; may be a singleton.
`void`	`destroy()` Shut down the data pipeline.
`AllExhibitImmutableData`	`getAllExhibitImmutableData(long oldStamp)` Gets all static exhibit data if its timestamp is not that specified.
`AllExhibitProperties`	`getAllExhibitProperties(long oldHash)` Gets set of all exhibit properties if its hash is not that specified.
`EventVariableValue`	`getEventValue(SimpleVariableDefinition def, EventPeriod intervalSelector, boolean current)` Get the current partial, or previous full, event set at the specified interval; never returns null.
`EventVariableValue[]`	`getEventValues(SimpleVariableDefinition def, EventPeriod intervalSelector, long intervalNumber, java.util.BitSet whichValues)` Get the specified event sets for the specified intervals; never null.
`GenProps`	`getGenProps(long oldStamp)` Gets the general properties as a GenProps object if its timestamp is not that specified.
`java.util.Properties`	`getGenSecProps(long oldStamp)` Gets the security properties as a Properties object if its timestamp is not that specified.
`int`	`getLiveCachedExhibitCount()` Return directly the number of partly- or fully- cached exhibits; never negative.
`java.util.Properties`	`getProperties(SimpleExhibitPipelineIF.PropsKey key, long versionID)` Get requested Properties selected by key and versionID.
`void`	`getRawFile(java.nio.ByteBuffer buf, java.lang.String exhibitName, int position, boolean dontCache)` Get a chunk of the raw exhibit binary.
`ExhibitStaticAttr`	`getStaticAttr(java.lang.String name)` Get the static attributes for a given exhibit.
`ExhibitThumbnails`	`getThumbnails(java.lang.String name, boolean create)` Gets the thumbnails for an exhibit.
`SimpleVariableValue`	`getVariable(SimpleVariableDefinition var)` Get a single variable value; returns null if no such value or wrong type.
`SimpleVariableValue[]`	`getVariables(long changedSince)` Get immutable Set of variable values altered on or after a given time, or all for -1.
`private boolean`	`isPopularDownload(ExhibitStaticAttr esa)` In the top-N (global) downloads recently, with at least 2 downloads.
`void`	`poll(GenProps _gp)` Poll periodically (of the order of a second) to do cache maintenance.
`static void`	`rmCache(java.io.File cacheDir)` Remove a persistent cache.
`void`	`setAggressive(boolean isAggressive)` Set the aggressiveness of the cache; by default not aggressive.
`void`	`setVariable(SimpleVariableValue newValue)` Set variable.
`int`	`setVariables(SimpleVariableValue[] newValues)` Update number of variables at once for efficiency.
`void`	`syncVariables(boolean force)` Synchronise with upstream values.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

ORPHANED_EXHIBIT_EXPIRY_ALLOWED

private static final boolean ORPHANED_EXHIBIT_EXPIRY_ALLOWED

If true then we may purge cached data for exhibits that appear to have been deleted (or renamed). These are exhibits that have been "orphaned" in the cache, ie they are not accessible because they do not logically exist (though they do potentially serve as a backup in case of disaster).

We will generally only do this if:

the exhibit data has not been used for a long time, and
if cache size is above the low-water mark that would prevent us from doing useful precacheing.

We do not rush to delete exhibits' data in case a transient problem has made an exhibit disappear temporarily. (Data for deleted exhibits will in any case eventually be deleted LRU (Least-Recently-Used) if the cache becomes full.)

See Also:: Constant Field Values

ORPHANED_EXHIBIT_MIN_UNUSED_TIME_MS

private static final long ORPHANED_EXHIBIT_MIN_UNUSED_TIME_MS

The minimim time before we will pre-emptively purge orphaned cache entries (ms); strictly positive. We don't expect to delete or rename exhibits very often, and the only harm in NOT purging them may be to prevent precaching of new exhibits, ie a minor performance issue rather than a correctness issue.

Add a random component so that all clients do not purge orphans at once!

A value of the order of a few days to a few months is probably reasonable.

CACHE_BASE_DIR

private static final java.lang.String CACHE_BASE_DIR

Base dir within cache dir for all our exhibit data.

See Also:: Constant Field Values

CACHE_EXPROPS_FILENAME

private static final java.lang.String CACHE_EXPROPS_FILENAME

Name of file in which to persist immutable exhibit data.

See Also:: Constant Field Values

CACHE_METADATA_FILENAME

private static final java.lang.String CACHE_METADATA_FILENAME

Name of file in which to persist cache meta data.

See Also:: Constant Field Values

STORE_EXPROPS_GZIPED

private static final boolean STORE_EXPROPS_GZIPED

If true, store exprops data (and cache metadata) GZIPed to possibly save space and I/O time.

See Also:: Constant Field Values

CACHE_EXDATA_DIR

private static final java.lang.String CACHE_EXDATA_DIR

Base dir within cache dir for all our raw exhibit content data.

See Also:: Constant Field Values

EVENT_HISTORY_DIR

private static final java.lang.String EVENT_HISTORY_DIR

Base dir within cache dir for all our event history data.

See Also:: Constant Field Values

CACHE_EXAUX_PREFIX

private static final java.lang.String CACHE_EXAUX_PREFIX

The prefix for all aux files associated with an exhibit file.

See Also:: Constant Field Values

CACHE_EXAUX_TIMESTAMP_KW

private static final java.lang.String CACHE_EXAUX_TIMESTAMP_KW

The keyword for the file containing the (decimal) exhibit timestamp.

See Also:: Constant Field Values

CACHE_EXAUX_TN_KW

private static final java.lang.String CACHE_EXAUX_TN_KW

The keyword for the file containing the serialised thumbnails object.

See Also:: Constant Field Values

MAX_TRANSFER_CHUNK_SIZE

private static final int MAX_TRANSFER_CHUNK_SIZE

Maximum (normal) exhibit data transfer chunk size (bytes); strictly positive. Maximum chunk transferred in one call (bytes), to avoid creating huge gaps in other activity by jamming up transactions and/or hogging all I/O bandwidth.

Should probably be of the order of a few tens of kBytes to allow efficient transfers on the wire, and a power of two to be more likely to interact efficiently with other caches.

If we use (a small multiple of) the bulk data transfer chunk size this will be reasonably efficient in terms of disc/network traffic, and if we can also keep it aligned to whole chunk boundaries then we may get maximally efficient accesses into disc (etc) data.

MAX_EXTD_TRANSFER_CHUNK_SIZE

private static final int MAX_EXTD_TRANSFER_CHUNK_SIZE

Maximum extended exhibit data transfer chunk size (bytes); strictly positive. When asked for data but we have a small gap in our cache before the request start normally we would have to pass the result upstream and not cache any. This is potentially very wasteful in P2P sharing of new exhibits.

So in this case we may allow the upstream request window to be moved back to patch up the hole from the end of our cached data to the start of the request, thus allowing us to capture the result.

This value should be a small multiple of the MAX_TRANSFER_CHUNK_SIZE.

MAX_REMOTE_FETCH_TO_MAKE_THUMBNAIL

private static final int MAX_REMOTE_FETCH_TO_MAKE_THUMBNAIL

The maximum number of bytes we will force a transfer of to force an immediate thumbnail generation. This might as well be at least one block, and might as well be a little bigger than we anticipate the resulting thumbnail size to be (limited to), but no larger than the maximum single transfer that we can make.

FALLBACK_MIN_CACHE_SIZE

private static final int FALLBACK_MIN_CACHE_SIZE

Minimum cache size to allow if GenProps is not set (bytes). This prevents thrashing on an empty cache and should be enough to store a few thumbnails and a few reasonable image chunks.

THUMBNAIL_ACCESS_UPDATES_ACCESS_TIMESTAMP

private static final boolean THUMBNAIL_ACCESS_UPDATES_ACCESS_TIMESTAMP

If true then accessing a thumbnail marks its exhibit as accessed. This means that for the purposes of avoiding eviction from cache, accessing a thumbnail is taken as being as significant as downloading (part of) the exhibit itself, if true.

If false then only actually reading part of the exhibit itself helps keep the exhibit ``fresh'' in the cache.

See Also:: Constant Field Values

DISC_RECHECK_INTERVAL_MS

private static final int DISC_RECHECK_INTERVAL_MS

Approximate minimum interval between rechecks of on-disc cache. When a check is done, the in-memory record of disc cache status is reloaded from disc, any debris is removed, etc.

A period of the order of at least a day is probably about right; not being exactly a multiple helps to ensure that we do not hit the same time every day, which might otherwise collide with other regular activity. Note that this recheck may take at least several minutes, so we don't want to do it too often!

The chosen interval is less than 2D (including a random component). We use less than a day-multiple to tend to be waiting for energy to become available at a similar time each day, eg from solar PV, when the system is in energy-conserving mode. This seems frequent enough in practice.

METADATA_MIN_SAVE_INTERVAL_MS

private static final int METADATA_MIN_SAVE_INTERVAL_MS

Approximate minimum interval between saves of the metadata; strictly positive. Since access to the exhibit/thumbnail data causes this to be updated (along with more significant changes to the cache), and saving can take significant time, we do not want to save this immediately we enocunter a change.

We can postpone a save for a while at the risk that if the system crashes or shuts down during that time and there was a structural change to the cache, we might have to abandon the old metadata and start again, which could be slow and a bit messy (losing some useful though not vital information).

Taking our cue from the old UNIX sync interval of 30s, a value in the range 30s to a few minutes is probably reasonable. Larger values of several minutes help reduce disc (write) activity which may be important for (say) solid-state storage such as Flash.

logger

private final SimpleLoggerIF logger

Our local logger; never null.

statsIDSCGEN

private final StatsLogger.StatsConfig statsIDSCGEN

The stats set to which we log general cache behaviour. The unique codes are the constants SCGNAME_XXX.

SCGNAME_MDSAVE

public static final java.lang.String SCGNAME_MDSAVE

General stats event name: an exhibit was evicted from the cache in LRU order.

See Also:: Constant Field Values

SCGNAME_CACHEEVICTLRU

public static final java.lang.String SCGNAME_CACHEEVICTLRU

General stats event name: an exhibit was evicted from the cache in LRU order.

See Also:: Constant Field Values

SCGNAME_CACHEREM

public static final java.lang.String SCGNAME_CACHEREM

General stats event name: an exhibit was removed from the cache.

See Also:: Constant Field Values

SCGNAME_CACHEADD

public static final java.lang.String SCGNAME_CACHEADD

General stats event name: an exhibit was added to the cache.

See Also:: Constant Field Values

SCGNAME_CACHEREM_CORRUPT

public static final java.lang.String SCGNAME_CACHEREM_CORRUPT

General stats event name: a corrupt exhibit was removed from the cache.

See Also:: Constant Field Values

SCGNAME_CACHE_VALIDATION

public static final java.lang.String SCGNAME_CACHE_VALIDATION

General stats event name: an exhibit in the cache was fully validated against checksums, etc.

See Also:: Constant Field Values

SCGNAME_CACHE_VALIDATION_PART

public static final java.lang.String SCGNAME_CACHE_VALIDATION_PART

General stats event name: an exhibit in the cache was partially validated against checksums, etc.

See Also:: Constant Field Values

SCGNAME_CACHERAWDATAMISS

public static final java.lang.String SCGNAME_CACHERAWDATAMISS

General stats event name: cache raw data read miss. We had to go upstream for at least part of the data.

(It is possible to have a hit and a miss on the same read if part is satisfied from cache and part not.)

See Also:: Constant Field Values

SCGNAME_CACHERAWDATAHIT

public static final java.lang.String SCGNAME_CACHERAWDATAHIT

General stats event name: cache raw data read hit. We satisfied at least part of the read from cache.

(It is possible to have a hit and a miss on the same read if part is satisfied from cache and part not.)

See Also:: Constant Field Values

SCGNAME_CACHETNHIT

public static final java.lang.String SCGNAME_CACHETNHIT

General stats event name: on-disc cache hit for thumbnail.

See Also:: Constant Field Values

SCGNAME_CACHETNMEMHIT

public static final java.lang.String SCGNAME_CACHETNMEMHIT

General stats event name: in-memory cache hit for thumbnail.

See Also:: Constant Field Values

SCGNAME_CACHETNMISS

public static final java.lang.String SCGNAME_CACHETNMISS

General stats event name: in-memory cache hit for thumbnail.

See Also:: Constant Field Values

SCGNAME_DATAFETCHFROMPEER_PREFIX

public static final java.lang.String SCGNAME_DATAFETCHFROMPEER_PREFIX

General stats event name: fetched a data block from a peer.

See Also:: Constant Field Values

SCGNAME_PRECACHERESTART

public static final java.lang.String SCGNAME_PRECACHERESTART

General stats event name: restarted scanning all exhibits for precaching.

See Also:: Constant Field Values

SCGNAME_PRECACHEEXAMINED

public static final java.lang.String SCGNAME_PRECACHEEXAMINED

General stats event name: exhibits examined for precaching.

See Also:: Constant Field Values

SCGNAME_PRECACHEEXDATABLOCK

public static final java.lang.String SCGNAME_PRECACHEEXDATABLOCK

General stats event name: exhibit data block precached.

See Also:: Constant Field Values

SCGPREF_PRECACHEEXDATABLOCKSRC

public static final java.lang.String SCGPREF_PRECACHEEXDATABLOCKSRC

General stats event name prefix: exhibit data block precache source (if not from master/upstream).

See Also:: Constant Field Values

SCGPREF_PRECACHEEXDATABLOCKSRCERR

public static final java.lang.String SCGPREF_PRECACHEEXDATABLOCKSRCERR

General stats event name prefix: exhibit data block precache source for error (if not from master/upstream).

See Also:: Constant Field Values

SCGPREF_PRECACHEEXDATABLOCKFETCHTIME

public static final java.lang.String SCGPREF_PRECACHEEXDATABLOCKFETCHTIME

General stats event name prefix: exhibit data block precache (succesful) fetch time (log2 ms).

See Also:: Constant Field Values

SCGNAME_PRECACHEERROR

public static final java.lang.String SCGNAME_PRECACHEERROR

General stats event name: errors encountered during precaching.

See Also:: Constant Field Values

SCGNAME_EXDATAREQIN

public static final java.lang.String SCGNAME_EXDATAREQIN

General stats event name: incoming request for exhibit data.

See Also:: Constant Field Values

SCGNAME_EXDATAREQINDC

public static final java.lang.String SCGNAME_EXDATAREQINDC

General stats event name: incoming request for exhibit data with "dontCache" flag set.

See Also:: Constant Field Values

SCGNAME_EXTHUREQIN

public static final java.lang.String SCGNAME_EXTHUREQIN

General stats event name: incoming request for exhibit thumbnails.

See Also:: Constant Field Values

SCGNAME_EXTHUREQINDC

public static final java.lang.String SCGNAME_EXTHUREQINDC

General stats event name: incoming request for exhibit thumbnails with "dontCreate" flag set.

See Also:: Constant Field Values

SCGNAME_EXTHUCREATED

public static final java.lang.String SCGNAME_EXTHUCREATED

General stats event name: created thumbnails locally from cached data.

See Also:: Constant Field Values

ASSUME_LOADED_METADATA_OK

private static final boolean ASSUME_LOADED_METADATA_OK

If true, assume that newly-loaded meta-data at is OK at start-up until proven otherwise.

See Also:: Constant Field Values

FORCE_IMMEDIATE_SAVE_ON_EXPANDED_METADATA

private static final boolean FORCE_IMMEDIATE_SAVE_ON_EXPANDED_METADATA

If true, synchronously force a save of meta-data each time we add a new entry at least. May be slow (O(n^2) for n exhibits), especially where the cache is not large enough to hold all exhibits so items are continually coming and going, but potentially makes the system more robust against loss of data.

See Also:: Constant Field Values

KEY_debugFlag_P2P_BLOCKXFER

private static final java.lang.String KEY_debugFlag_P2P_BLOCKXFER

Key in generic props of P2P-profiling flag.

See Also:: Constant Field Values

MAX_BEST_EX_PRECACHED

private static final int MAX_BEST_EX_PRECACHED

Maximum number of "best" exhibits to get enhanced precaching; non-negative.

See Also:: Constant Field Values

_bestExhibits

private final java.util.Set<java.lang.String> _bestExhibits

Contains the full exhibit names of the "best" few exhibits for enhanced precacheing; never null. Maintained/updated by _doPrecache().

This is a snapshot of what the "best" exhibits are estimated to be as each precacheing round starts, often based on "quick approximation" data.

The implementation is optimised for fast lookup with "contains()".

Thread-safe.

metaData

private final ExhibitDataSimpleCache.MetaData metaData

In-memory copy of whole-cache meta-data; never null. Note that the read/write status may change at any time.

The instance is never replaced; the state is replaced in-situ if need be to ensure that we never have two instances of this that believe they control the disc cache.

source

private final SimpleExhibitPipelineIF source

The upstream data source; never null.

cacheDir

private final java.io.File cacheDir

The cache dir, else null. If this is not a valid dir at class creation time we ensure that we save a null here.

rwl

private final java.util.concurrent.locks.ReentrantReadWriteLock rwl

The read/write lock for the whole cache except system variables; never null. Any access that may update the cache state in memory or on disc must be protected by a write lock.

Any access that may just read the cache need only have a read lock.

Most accesses may have to start by taking a write lock (for example because they may have to fetch data from upstream and insert it into the cache) but can downgrade it to a read lock as soon as they know that they will not be altering the cache at all or any further.

The main exception is any state component held in volatile fields.

Note that the variable store is internally thread-safe and does not require protection by this lock.

We want performance (ie best throughput) rather than fairness.

We generally do not want lock attempts to block forever, which means that we give up attempting to obtain a lock after a given number of attempts (with a maximum time per attempt). While this is intended to prevent to limit delays in the face of I/O problems, this may rescue us from logic errors in extremis.

MAX_LOCK_ATTEMPTS

private static final int MAX_LOCK_ATTEMPTS

Maximum number of consecutive attempts to obtain lock for read or write; strictly positive.

See Also:: Constant Field Values

_gXL_offset

private static final int _gXL_offset

Stack offset for _getXXXXLock() to find caller's stack frame.

See Also:: Constant Field Values

exhibitDataSource

private final AllExhibitProperties.ExhibitDataSource exhibitDataSource

An AllExhibitProperties.ExhibitDataSource wrapping ourselves; never null.

MAX_THREADS_aTWQ

private static final int MAX_THREADS_aTWQ

Maximum number of threads that may run in _asyncTNFetch() and other local discardable data read-ahead tasks; strictly positive. We limit the amount of threading by:

The number of available CPUs (as sometimes this work may be CPU intensive).
The likely limit on back-end connectivity (eg HTTP-connection-count limited) (given that this cache will often by upstream of an HTTP tunnel).
Likely limits on local and upstream disc throughput.
The likely strain on other resources, such as memory.

This limit/count/cap should generally be >1 since the work is mainly I/O bound, but should generally be not much more than (say) half the maximum simultaneous outbound tunnel HTTP connection count.

MAX_QUEUED_TN_FETCHES

private static final int MAX_QUEUED_TN_FETCHES

Maximum number of async thumbnail fetches to queue; strictly positive.

See Also:: Constant Field Values

discardableReadAheadTaskThreadPool

private final java.util.concurrent.ExecutorService discardableReadAheadTaskThreadPool

Shared thread pool for I/O-bound activities for Web-driven requests. Suitable for mainly-I/O-bound threads, thus we have a fixed thread limit. This limit also protects upstream servers from excess load.

A limited amount of work can be queued, but excess is handled by discarding the oldest queued it

org.hd.d.pg2k.svrCore.datasource Class ExhibitDataSimpleCache