org.hd.d.pg2k.svrCore.datasource.simpleCache
Class MetaData

java.lang.Object
  org.hd.d.pg2k.svrCore.datasource.simpleCache.MetaData

All Implemented Interfaces:

java.io.ObjectInputValidation, java.io.Serializable

public final class MetaData
extends java.lang.Object
implements java.io.Serializable, java.io.ObjectInputValidation

Cache in-memory meta-data representation. Holds information about what exhibit data is cached on disc.

Is serialisable to be able to persist to disc and allow for a fast start-up.

Can be marked read-only to hold the cache in its current state.

This object is completely thread-safe and holds its instance lock to synchronise activity.

The names of all members affecting the state of cached exhibit data and meta-data start with `exhibit'.

Note that holding the instance lock does not shut out all changes, a little like attempting to lock a WeakHashMap or ConcurrentHashMap.

Marked public only for testing, else could be package-visible only.

See Also:

Serialized Form

cachedExhibits

private transient java.util.Hashtable<Name.ExhibitFull,CachedFile> cachedExhibits

Set of cached exhibits, from full exhibit name to CachedFile entry; never null after construction/deserialisation. We may optimise memory use by sharing the names (etc) with those from AllExhibitImmutableData held elsewhere.

This is the primary store of data; other (transient) values are derived from it; this is saved to and restored from the serialised form in an efficient and defensive way.

This is a Hashtable for thread-safety.

Compound operations may be made atomic by a lock on this object, but no other lock may be taken while this happens.

Any access that updates this table should be under the instance lock.

exhibitsLRU

private transient java.util.SortedSet<CachedFile> exhibitsLRU

LRU ordered set of the CachedFile values in cachedExhibits; never null after construction/deserialisation. This should be exactly the same set as cachedExhibits.valueSet().

The least-recently-accessed items are first in this list, and are the first to be discarded if the cache becomes over-full.

This is not of itself thread-safe and all access must be under the instance lock.

totalBytes

private transient volatile long totalBytes

Estimated total bytes of disc used by cached entries. Computed as sum of estimates from CachedFile values.

Write access restricted to _setTotalBytes(); marked volatile to allow unlocked read access.

totalBytesHighWaterMark

private transient volatile long totalBytesHighWaterMark

The highest value of totalBytes, ie the cache actual high-water mark; never negative. Computed as sum of estimates from CachedFile values.

Write access restricted to _setTotalBytes() and by _setTotalBytesHighWaterMark() and must be under the instance lock.

Marked volatile to allow lock-free read access.

needsSave

private transient volatile boolean needsSave

If true, this needs a save to disc (ie this instance's state has changed). Is transient so always false when recovered from disc.

Is volatile (so accessed with no lock) only by setNeedsSave() (to set it true) and saveToDisc() (to clear it after a successful save to disc).

USE_EXTANT_DISC_ENTRY_ON_CREATE

private static final boolean USE_EXTANT_DISC_ENTRY_ON_CREATE

If true, when creating cache entry if data found on disc use it rather than refuse to build meta-data entry. This is useful for co-existing with old cache mechanism.

See Also:

Constant Field Values

MIN_FS_PERCENT_FREE

static final int MIN_FS_PERCENT_FREE

Minimum percentage free in cache filesystem for us to expand the cache; non-negative in range ]100,0]. Set to prevent accidental exhaustion of space in a shared filesystem (eg inducing transient write errors for us or other storage users, etc) due to incorrect usage estimates by us or unexpected use by others, and to avoid very poor performance from a nearly-full filesystem.

A value in the range 1-10 is probably good.

See Also:

Constant Field Values

_cFSBHWM_cache

private transient volatile Tuple.Triple<java.io.File,java.lang.Long,java.lang.Long> _cFSBHWM_cache

Cache of estimated usable free space in the underlying cache filesystem; initially null. Notes the cacheDir and time and free space of the last request.

It is in fact assumed that in general:

• The presented cacheDir will be the same each time.
• There will be relatively little thread-racing in any case.
• That filesystem usable status can reasonably be cached for a reasonable time.

Marked volatile for thread-safe unlocked access.

MAX_FS_USABLE_SPACE_CACHE_MS

private static final int MAX_FS_USABLE_SPACE_CACHE_MS

Maximum time in ms that we may retain a cached 'usable filespace' estimate; strictly positive. If the file system in question doesn't contain much activity/data other than our cache, and we are checking mainly to deal gracefully with internal usage-estimation errors and from slowly-accumulated crud from our (and third-party) logs etc, then we could probably cache the value for hours at a time without great loss, but probably cacheing for tens of seconds would in practice eliminate much of the cost.

Pick a prime-ish value to minimise clashes with other activity...

See Also:

Constant Field Values

TRY_TO_RETAIN_TN_IN_PURGE

private static final boolean TRY_TO_RETAIN_TN_IN_PURGE

If true then try to retain metadata and thumbnails of exhibits we purge. Thumbnails in particular may be expensive to regenerate/fetch, and do not take much space.

See Also:

Constant Field Values

serialVersionUID

private static final long serialVersionUID

My serial ID.

See Also:

Constant Field Values

_fullyCachedCount

private transient volatile int _fullyCachedCount

Cache of estimate of full-cached exhibits; non-negative. Private to getFullyCachedCount().

Marked volatile to allow thread-safe lock-free access.

Initially zero to force recomputation.

Not serialised.

MetaData

private MetaData()

Build an empty, read-only set of meta-data.

MetaData

private MetaData(MetaData other)

Build a lightweight copy. Takes lock on the core data while copying it for atomicity/consistency; no other lock should be required.

Parameters:

other - instance to copy from; never null

MetaData

private MetaData(AllExhibitProperties aepCurrent,
                 java.io.File cacheDir,
                 SimpleLoggerIF logger)
          throws java.io.IOException

Build a new meta-data set by examining the given cache directory. This is understood not to be perfect, because, for example, the last-accessed time may not be available on disc and may have to be approximated with an available last-modified-time; generally this is used when:

1. no persisted MetaData value is available, and we have to do the best we can, or,
2. we want to load a copy of what is on disc to check that what we have in memory is not badly wrong.

No locks (ie especially the rwl or cache's instance lock) should be held while this is running.

This does not modify data on disc.

The constructed object is marked as needing to be saved.

This relies in part on being able to find the data files for the cached (prefixes of) exhibits by their valid full exhibit names.

Explicitly attempts to recover any plausible entries corresponding to current AEP entries.

Thus this may miss entries which have no current data file, eg because there were zero bytes cached for that file, and may have to recover such data later as encountered.

Explicitly attempts to recover any plausible entries corresponding to current AEP entries.

Parameters:

aepCurrent - reasonably current AEP as a hint for entries to look for; never null

cacheDir - the cache top-level directory; never null

logger - to log to; never null

Throws:

java.io.IOException - in case of major problems in reconstructing the cache state (minor problems will be ignored)

createMetaData

public static MetaData createMetaData()

Creates empty read-only instance; never null. Marked public to allow testing.

createLightweightCopy

public MetaData createLightweightCopy()

Returns fast lightweight read-only copy ready to serialise/save or check; never null. Does not copy some of contingent state on running copy, in particular transient or derived values.

Can be used to free up main lock on meta-data more quickly than might otherwise be the case.

The instance returned may not be suitable for performing any operations on other than serialising/saving or verifying against another copy.

Takes lock on the core data while copying it for consistency; not other lock should be required.

Marked public to allow testing.

createMetaData

public static MetaData createMetaData(AllExhibitProperties aepCurrent,
                                      java.io.File cacheDir,
                                      SimpleLoggerIF logger)
                               throws java.io.IOException

Creates new instance and populates from the filesystem; never null. Marked public to allow testing.

Parameters:

aepCurrent -

cacheDir -

logger -

Returns:

Throws:

java.io.IOException

getKnownExhibits

public java.util.Set<Name.ExhibitFull> getKnownExhibits()

Get Set of exhibits on which metadata is currently held in this cache. This takes a private atomic copy of the Set of exhibits for which we have some data, thumbnails, etc, in the cache.

getExhibitsLRUList

java.util.List<CachedFile> getExhibitsLRUList()

Get exhibit metadata in LRU order; never null but may be empty. Cost goes up with number of exhibits whose metadata is held.

getLeastRecentlyUsed

CachedFile getLeastRecentlyUsed()
                          throws java.util.NoSuchElementException

Get most-recently-used exhibit.

Throws:

java.util.NoSuchElementException

getMostRecentlyUsed

CachedFile getMostRecentlyUsed()
                         throws java.util.NoSuchElementException

Get least-recently-used exhibit.

Throws:

java.util.NoSuchElementException

getTotalBytesCurrentlyUsedByCache

long getTotalBytesCurrentlyUsedByCache()

Get estimated total bytes of disc used by cached entries; never negative.

_setTotalBytes

private void _setTotalBytes(long newTB)

Set estimated total bytes of disc used by cached entries; never negative.

getTotalBytesHighWaterMark

long getTotalBytesHighWaterMark()

Get the highest value of totalBytes, ie the cache actual high-water mark; never negative.

_recomputeDerivedValues

private void _recomputeDerivedValues()

This private routine updates all (transient) data derived from the primary cachedExhibits map. This is used after deserialisation, construction or insertion where it is easier to recompute such derived data from scratch than change it incrementally as usual.

getNeedsSave

boolean getNeedsSave()

Get the 'needs to be saved to disc' value. Always false after deserialisation and construction of a new instance.

setNeedsSave

void setNeedsSave()

Set the 'needs to be saved to disc' value to be true.

saveToDisc

void saveToDisc(java.io.File cacheDir,
                SimpleLoggerIF logger,
                StatsLogger.StatsConfig statsIDSCGEN)
          throws java.io.IOException

Save (serialise) to disc; throws IOException in case of difficulty. Takes a lightweight copy of the data and saves that, avoiding holding any lock (and shutting out other activity) for significant time.

Clears needsSave (to false) just before the save, but sets it true again if the save appeared to fail.

Parameters:

cacheDir - specified the top-level cache directory; never null

Throws:

java.io.IOException

size

int size()

Get count of cached exhibits, including partially-cached ones; never negative.

isEmpty

boolean isEmpty()

Is our meta-data store empty, ie no exhibits cached at all?

exhibitGetInfo

CachedFile exhibitGetInfo(Name.ExhibitFull name)

Get cached-file details for a given exhibit; null if exhibit not cached. This may incrementally verify cache data against the disc copy, though for performance reasons this may not happen very often.

This is thread-safe and does not need nor grab any cache locks, but to ensure this data remains valid during any compound operation, at least a read lock should usually be obtained.

Parameters:

name - the exhibit whose meta-data is required; never null

exhibitIsFullyLoaded

boolean exhibitIsFullyLoaded(ExhibitStaticAttr esa)

Check if the exhibit is fully loaded.

This does not grab any cache locks, but to ensure this data remains valid during any compound operation, at least a read lock should generally be obtained.

Parameters:

esa - the exhibit whose meta-data is required; never null

Returns:

false if no such exhibit or it is not fully loaded

_removeCacheRecordFromMemoryCopy

private void _removeCacheRecordFromMemoryCopy(Name.ExhibitFull name)

Remove cached-file details from memory record; does nothing if not present. The name argument must not be null. Private to this instance.

_updateCacheRecordInMemoryCopy

void _updateCacheRecordInMemoryCopy(CachedFile newCf,
                                    SimpleLoggerIF logger)

Update or add new cached-file details, adjusting cache data in situ. The argument must not be null. Private to this instance.

Marks the meta-data as needing to be saved.

isEquivalent

boolean isEquivalent(MetaData other)

Returns true if this metadata is essentially equivalent to another one. This has a sufficiently loose notion to allow for discrepancies in (for example) last-accessed timestamps which cannot be completely reconstructed from the disc copy.

The result is false if the argument is null.

pruneOrAdjustBadInMemoryCacheRecords

void pruneOrAdjustBadInMemoryCacheRecords(long reloadStartTime,
                                          MetaData newData,
                                          SimpleLoggerIF logger)
                                    throws java.lang.IllegalStateException

Install info from new cache object in this one. This would allows incorporation of the results of a disc scan to see the actual state of the disc. (Usually the argument is the best-efforts reconstruction from the disc of the cache state, with the timestamps arbitrarily too old.)

This is conservative and removes state from memory that appears absent from disc (eg because stuff is being deleted out from under the process), and updates any record to have the least cached-length seen or current.

The newData instance is considered essentially private to this call and should be marked read-only.

The readWrite status is not altered nor updated by this operation (but will usually be read/write to avoid blocking cache activity)

The needsSave flag may be set by this call.

Only the cache-checking routines should call this at all.

TODO: Allow entries relevant to the current AEP to be added from the disc-based data

Parameters:

reloadStartTime - a time just before the newData was loaded from disc

Throws:

java.lang.IllegalStateException

exhibitMarkAsAccessed

CachedFile exhibitMarkAsAccessed(Name.ExhibitFull name,
                                 SimpleLoggerIF logger,
                                 java.io.File cacheDir)
                           throws java.io.IOException

Mark exhibit as accessed (`touch' it) and returns the possibly-modified CachedFile record. Returns null if the named exhibit does not exist in the cache at all, else never returns null.

This call does not access disc (ie affects in-memory status only), and silently does nothing if there is no entry for the named exhibit.

This marks the last-accessed time in the MetaData as now, to be preserved on disc at some point in the future.

This MetaData object is marked as needing to be saved if an exhibit was `touched' successfully.

This is preferably used to:

• Mark end-user access to an exhibit's data.
• Mark some expensive operation (such as computing thumbnails) which we would like to avoid throwing away too soon.

Parameters:

cacheDir - if not null then we force an update of a timestamp on disc as a backup in case the serialised metadata is lost; this causes extra disc traffic though not necessarily synchronous

Throws:

java.io.IOException

_exhibitRemoveLRUCacheEntry

private boolean _exhibitRemoveLRUCacheEntry(java.io.File cacheDir,
                                            Name.ExhibitFull dontRemoveName,
                                            boolean justData,
                                            SimpleLoggerIF logger,
                                            StatsLogger.StatsConfig statsIDSCGEN)
                                     throws java.io.IOException

Remove the least-recently-used exhibit cache item, if any. We won't remove the given named item if non-null; this may prevent us removing anything if it is the only cached item.

If the named item is the oldest then we'll remove the next oldest, if there is one.

Parameters:

justData - if true then we try to retain metadata/thumbnails

Returns:

true if we removed something

Throws:

java.io.IOException

exhibitRemoveCacheEntry

boolean exhibitRemoveCacheEntry(java.io.File cacheDir,
                                Name.ExhibitFull name,
                                boolean justData,
                                SimpleLoggerIF logger,
                                StatsLogger.StatsConfig statsIDSCGEN)
                          throws java.io.IOException

Remove an exhibit entry from cache, possibly including all its data and metadata. Returns true if successful, false if not.

This will fail if the metaData is read only, or if an entry for the exhibit does not exist in the metadata and on disc.

Since these failures will be by returning false rather than by causing an exception then this can be used as a gentle unconditional way to make sure that an entry does not exist whether or not one did before.

This needs to access disc and so will need to hold the main write lock while it does so, grabbing the metaData instance lock inside the main lock where both need to be held simultaneously.

Can optionally attempt to just remove the exhibit data, leaving metadata and any thumbnails intact.

Parameters:

justData - if true then we try to retain metadata/thumbnails

Throws:

java.io.IOException

exhibitCreateNewCacheEntry

boolean exhibitCreateNewCacheEntry(java.io.File cacheDir,
                                   ExhibitStaticAttr esa,
                                   SimpleLoggerIF logger,
                                   StatsLogger.StatsConfig statsIDSCGEN)
                             throws java.io.IOException

Create a new exhibit entry. Returns true if successful, false if not.

This will fail if the metaData is read only, or if an entry for the exhibit already exists in the metaData or on disc.

Since these failures will be by returning false rather than by causing an exception then this can be used as a gentle unconditional way to make sure that an entry does exist whether or not one did before.

This needs to access disc and so will need to grab a write lock on the main cache lock while it does so, grabbing the metaData instance lock inside the main lock where both need to be held simultaneously.

Marks the meta-data as needing to be saved if a new meta-data entry had to be created...

A write lock should be held on the cache while this is called.

Parameters:

esa - the basic info on the exhibit to have an entry created

Throws:

java.io.IOException - in case of serious problems accessing the on-disc cache state

computeTargetHighWaterMark

long computeTargetHighWaterMark()

Computes the target high-water mark (target maximum cache size); strictly positive. If we currently only have a default (zero-timestamp) genProps then we use the initial size of the cache as our high-water mark, ie do not allow the cache to expand.

We may further restict cache size to leave a given percentage free in the underlying filesystem.

computeFreeSpaceBelowLowWaterMark

long computeFreeSpaceBelowLowWaterMark(java.io.File cacheDir)

Compute current definite free-space in cache. This returns the amount of space that we have below the low-water mark; if this returns a positive number then the cache is within that low water mark and can be considered nowhere near full.

If we don't trust the GenProps value then the low-water mark is the current actual high-water mark for the cache, ie we try to maintain its size approximately.

Is synchronized to maintain consistency while working.

lotsFree

boolean lotsFree(java.io.File cacheDir)

Returns true if we have lots of disc space free. This means that usage is well below the low-water mark.

If we don't trust GenProps this always returns false.

someFree

boolean someFree(java.io.File cacheDir)

Returns true if we have some (not lots of) disc space free. This means that usage is below the low-water mark.

If we don't trust GenProps this always returns false.

computeFreeSpaceBelowHighWaterMark

long computeFreeSpaceBelowHighWaterMark(java.io.File cacheDir)

Compute available free space in cache (bytes); zero or negative if none. This ignores the purge state of the system and simply reports what space is available before the cache overflows.

This is thread-safe though may be expensive.

canPrecacheExhibitData

boolean canPrecacheExhibitData(java.io.File cacheDir)

If true, we can pre-cache data blocks for (new or existing) exhibits. POLICY: only true if we have (lots of) free space and the server-slowdown factor is no greater than 2 (ie moderate or absent).

canPrecacheThumbnails

boolean canPrecacheThumbnails(java.io.File cacheDir)

If true, we can pre-cache thumbnails for (existing) exhibits. POLICY: only true if the free space is enough to store another thumbnail and fetch the data to do so first if necessary

Basically we're prepared to go right up to the wire to generate thumbnails because they are so useful to the user.

We leave a little bit of wiggle room on space.

canPrecache

boolean canPrecache(java.io.File cacheDir)

If true, some precaching may be possible, going by our generally-least-strict limit. We will only allow precaching if we have a valid LocalProps value.

purge

void purge(GenProps gp,
           java.io.File cacheDir,
           long howMuch,
           Name.ExhibitFull dontPurgeName,
           SimpleLoggerIF logger,
           StatsLogger.StatsConfig statsIDSCGEN)
     throws java.io.IOException

Purges the cache of old entries to make space for new, if needed. This will do nothing unless the space used by the cache is above the low-water mark. We compute the low-water mark from GenProps, or from the actual high-water mark in the current use of the cache if the passed GenProps is the default (has a zero timestamp). (We attempt to roughly maintain the cache or possibly run it down slightly while waiting for GenProps to arrive.)

This will avoid purging the cache of the item passed to it by name in case it was a candidate for removal.

If the argument is zero this tries to ensure that space for at least one new (empty) exhibit cache entry to be created.

If greater than zero this tries to ensure that an existing cache entry can be extended by at least the given number of bytes (either for the exhibit data or for thumbnail data).

This grabs the instance lock for most of its work.

This will fail if the cache is marked read-only.

Parameters:

howMuch - howMuch space we need to ensure is free; positive

Throws:

java.io.IOException

exhibitGetThumbnails

ExhibitThumbnails exhibitGetThumbnails(java.io.File cacheDir,
                                       Name.ExhibitFull name,
                                       SimpleLoggerIF logger)
                                 throws java.io.IOException

Get the cached thumbnails for an exhibit, or null if none cached. This needs to access disc and so will need to hold the main cache while it does so, grabbing the metaData instance lock inside the main lock where both need to be held simultaneously.

If a problem is encountered deserialising thumbnails, the cache is not-read-only, and AUTO_REPAIR_DURING_IO is true, we may zap the offending serialised file to try to recreate it later.

Should nominally be in the scope of an appropriate main read lock, but as no metadata change is made then the only risk is a race from the disc cache entry being removed during this call, resulting in an IOException.

Throws:

java.io.IOException - in case of serious problems accessing the on-disc cache state, eg the entry was missing or removed during this call

exhibitSaveThumbnails

boolean exhibitSaveThumbnails(GenProps gp,
                              java.io.File cacheDir,
                              ExhibitStaticAttr esa,
                              ExhibitThumbnails tns,
                              SimpleLoggerIF logger,
                              StatsLogger.StatsConfig statsIDSCGEN)
                        throws java.io.IOException

Saves the thumbnails for an exhibit. Returns true if successful, false if not.

This will fail if the metaData is read only.

If the thumbnails argument is null, this will purge any extant thumbnails for this exhibit.

This will create a new zero-length exhibit entry if necessary.

Since these failures will be by returning false rather than by causing an exception then this can be used as a gentle unconditional way to make sure that an entry does exist whether or not one did before.

A cache write lock will need to be held while this is running.

Parameters:

esa - the basic info on the exhibit to have an entry created

Throws:

java.io.IOException - in case of serious problems accessing the on-disc cache state

writeObject

private void writeObject(java.io.ObjectOutputStream oos)
                  throws java.io.IOException

Write out a less-redundant form of our internal information. Prevent (lazy-evaluation-caused) state changes while serialising by being synchronized.

The more-efficient on-the-wire format also makes defensive reading easier.

Throws:

java.io.IOException

readObject

private void readObject(java.io.ObjectInputStream in)
                 throws java.io.IOException,
                        java.lang.ClassNotFoundException

Deserialise.

Throws:

java.io.IOException

java.lang.ClassNotFoundException

validateObject

public void validateObject()
                    throws java.io.InvalidObjectException

Validate fields/state. Called in the constructor and possibly after de-serialising.

Barf if something bad is found. (Maybe allow some extra info in debug version.)

Assumes _recomputeDerivedValues() has been called before this is called since deserialisation or construction.

Specified by:

validateObject in interface java.io.ObjectInputValidation

Throws:

java.io.InvalidObjectException

getFullyCachedCount

int getFullyCachedCount(AllExhibitProperties aep,
                        boolean force)

Return estimated count of known-fully-cached exhibits; non-negative. It would be wise to force recomputation when a new AEP is loaded, or after a precacheing round is complete, or after any other major change in cache status.

Note that since the cached count is not serialised with the metadata then it will need to be recomputed when metadata is loaded/deserialised.

Even without a recomputation being forced, we may decide to do one if it seems that the value might be stale.

This will take time O(n) for an n-exhibit AEP to (re)compute its result, but does minimal or no locking, and none for the duration of the method, so is safe to launch in a background thread for example.

May be expensive (continually recomputing) if/while no exhibit is fully locally cached, but this is assumed to be unlikely in practice.

Parameters:

aep - if non-null, allow recomputation against this AEP

force - if true, force immediate recomputation against this AEP if non-null

Returns:

the estimate of exhibits whose data is entirely cached locally; the count does not take account of locally-recreatable data such as thumbnails, and may be too low or too high