org.hd.d.pg2k.svrCore
Class FileTools

java.lang.Object
  org.hd.d.pg2k.svrCore.FileTools

public final class FileTools
extends java.lang.Object

This class has tools for common file operations. (Derived from old FileTools 1.100 01/02/03.)

F_tmpPrefix

public static final java.lang.String F_tmpPrefix

Prefix used on temporary files, eg while doing atomic replacements. This used to be in GlobalParams but we may even need it while loading GlobalParams.

See Also:

Constant Field Values

FS_EST_BLOCK_SIZE_BYTES

public static final int FS_EST_BLOCK_SIZE_BYTES

Rough estimate of file-system allocation size for usage estimate (bytes); positive power of 2. We should separately round up both the file size and the filename size to a multiple of this in order to make a conservative estimate of filesystem usage when we are creating files, and especially if we cannot measure our actual usage.

Magnetic disc is often 512-byte blocks and/or 1kB fragments. Optical disc is often 1kB blocks. Flash memory SSD is often 2kB blocks.

Something around 512--8192 bytes is reasonable for many filesystems and media.

See Also:

Constant Field Values

_runCmdLock

private static final java.lang.Object _runCmdLock

Private lock for runCmd.

_runCmdFix

private static final boolean _runCmdFix

If _runCmdFix is true, we are working round a JDK 1.2 bugs in Process. In the JDK 1.2 bug, if some more than one thread runs a Process, even if not simultaneously, there is a small change of one spawned by a thread that is not the main thread hanging.

See Also:

Constant Field Values

_runCmdSignal

private static final java.lang.Object _runCmdSignal

Private object used for signalling within the _runCmdLock. Used as a signal between the runner thread and the runCmd() code that a new command is being posted or the result is ready.

_runCmdArgs

private static java.lang.String[] _runCmdArgs

Private location for runCmd() where a new command's args are put. Null when no command currently running.

When this is set non-null by runCmd() it does a notifyAll() on _runCmdSignal, then waits on the same signal until _runCmdResult becomes non-null. This is cleared by the runner thread when it accepts and starts to run the new command.

Change under the _runCmdSignal lock.

_runCmdResult

private static java.lang.Object _runCmdResult

The result of running a command. Thgis can be Integer for an exit value, or IOException or a RuntimeException.

Set non-null by the runner thread (and a signal sent on _runCmdSignal) when the command terminates; cleared by runCmd() on accepting the result.

Change under the _runCmdSignal lock.

_cmdRunner

private static final java.lang.Thread _cmdRunner

The runner thread for commands. Assumed immortal.

rPF_rwlock

private static final java.util.concurrent.locks.ReentrantReadWriteLock rPF_rwlock

Private lock for replacePublishedFile(). We use a read/write lock to improve available concurrency.

TODO: We could extend this to a lock per distinct directory or filesystem.

STF_ALWAYS_STREAMS_TO_FILESYSTEM

public static final boolean STF_ALWAYS_STREAMS_TO_FILESYSTEM

If true then try to stream (often large) serialised objects directly to the filesystem. If true the serialiseToFile() will always write something to the filesystem (and update the timestamp at least) even if nothing has changed, so the caller should probably try to avoid redundant/frequent calls.

See Also:

Constant Field Values

TYPICAL_DEFLATE_MIN_TEXT_SIZE_COMPRESSABLE

public static final int TYPICAL_DEFLATE_MIN_TEXT_SIZE_COMPRESSABLE

Typical size in bytes/characters of (English, ASCII) text below which deflate/zlib compression unlikely to be very effective. This is for typical (English, ASCII) text given the overheads of the compression methods etc.

Avoiding trying to compress things smaller than this may save lots of CPU time without too much memory/space cost, and may allow better compression of aggregates.

See Also:

Constant Field Values

FileTools

private FileTools()

Prevent creation of instances of this class.

getExtension

public static final java.lang.String getExtension(java.lang.CharSequence name)

Get the extension of a file name, not including the leading dot. Returns null if no extension present.

runCmd

public static int runCmd(java.lang.String[] cmd)
                  throws java.io.IOException

Runs a command, and returns the exit status. Does not expect the process to use stdin/stdout/stderr; the process may hang if it tries to use them.

Because of possible deadlock/hang problems in JDK1.2, I am serialising all sub-processing runs at the cost of some parallelism.

Throws:

java.io.IOException

rmRecursively

public static void rmRecursively(java.io.File fileOrDir)
                          throws java.io.IOException

Recursively removes specified file/directory. Equivalent of UNIX "rm -rf file".

Fails if it cannot remove the target, but does not complain if the target does not exist.

Throws:

java.io.IOException

rmRecursively

public static void rmRecursively(java.io.File fileOrDir,
                                 java.io.FileFilter filter)
                          throws java.io.IOException

Recursively removes specified file/directory. Equivalent of UNIX "rm -rf file".

Fails if it cannot remove the target, but does not complain if the target does not exist.

Parameters:

filter - matches files/dirs to remove; null to remove all files/dirs from that specified

Throws:

java.io.IOException

replacePublishedFile

public static boolean replacePublishedFile(java.lang.String name,
                                           byte[] data)
                                    throws java.io.IOException

Replaces an existing published file with a new one (see 3-arg version). Is verbose when it replaces the file.

Throws:

java.io.IOException

replacePublishedFile

public static boolean replacePublishedFile(java.lang.String name,
                                           byte[] data,
                                           boolean quiet)
                                    throws java.io.IOException

Replaces an existing published file with a new one. This replaces (atomically if possible) the existing file (if any) of the given name, ensuring the correct permissions for a file to be published with a Web server (ie basically global read permissions), provided the following conditions are met:

• The filename extension is acceptable (not checked yet).
• The data array is non-null and not zero-length.
• The content of the data array is different to the file.
• All the required permissions are available.

If the file is successfully replaced, true is returned.

If the file does not need replacing, false is returned (and the file is not replaced or touched).

If an error occurs, eg in the input data or during file operations, an IOException is thrown.

This routine enforces locking so that only one such operation may be performed at any one time. This does not avoid the possibility of externally-generated races.

The final file, once replaced, will be globally readable, and writable by us.

(If the final component of the file starts with ".", then the file will be accessible only by us.)

Parameters:

quiet - if true then only error messages will be output

Throws:

java.io.IOException

_atomicishFileReplace

private static boolean _atomicishFileReplace(java.io.File extant,
                                             java.io.File tempFile,
                                             int length,
                                             boolean quiet)
                                      throws java.io.IOException

Attempt to atomically (or nearly so) replace extant file with tempfile. FIXME: Where renameTo() not atomic, rename old file to .bak version to preserve it in case of crash

Throws:

java.io.IOException

makeTempFileNameInSameDirAsTarget

private static java.io.File makeTempFileNameInSameDirAsTarget(java.io.File extant)

Create a new temporary filename for the same directory as the extant file; never null. A file in the same directory is usually guaranteed to be in the same filesystem, and thus may often allow an atomic update/replace and in any case ensures that we cannot run out of space (barring other concurrent unrelated activity in the filesystem) when we try to replace the extant file with the temporary one.

To avoid internal races this should be generated and used within the scope of our internal 'filesystem update' lock.

The generated name starts with the 'temporary' prefix.

makePublishingDir

public static boolean makePublishingDir(java.io.File path)
                                 throws java.io.IOException

Makes a publicly-readable directory path if not already present. Like File.mkdirs(), but attempts to ensure that any directory component created by this routine is publicly readable and searchable, ie at least permissions read and execute for all. Final permissions will usually be 0755,.

Optionally, a new empty index.html file can be created inside any directory that is created as a simple Web security precaution, at least until something is put in its place.

If this routine fails it may have succeeded in creating some of the necessary parent directories.

This shares a lock with replacePublishedFile().

(This routine should maybe be merged with makeHTMLSubDirs(), though the relationship is not trivial.)

Returns:

true if and only if the directory was created, along with all necessary parent directories; false otherwise

Throws:

java.io.IOException

readTextFile

public static java.lang.String readTextFile(java.io.File f)
                                     throws java.io.IOException

Read text file into a String. Reads a line at a time, trimming whitespace off either end and putting in a single new line at the end instead.

This treats the file as ISO-8859-1 8-bit data.

Throws:

java.io.IOException - in case of trouble

serialiseToFile

public static boolean serialiseToFile(java.lang.Object o,
                                      java.io.File filename,
                                      boolean gzipped,
                                      boolean quiet)
                               throws java.io.IOException

Given a file, serialises an object to it. This atomically replaces the target file if possible.

This may be streamed directly to the filesystem (though replacing and extant file atomically once fully written) to avoid running out of memory with very large objects.

The usual internal 'filesystem update' lock is help while this works.

Parameters:

gzipped - if true, the file is written GZIP-compressed to (usually) save significant space

quiet - if true, only outputs errors

Returns:

true if a file was replaced, false it was already present with the correct content

Throws:

java.io.IOException - if something bad happens

deserialiseFromFile

public static java.lang.Object deserialiseFromFile(java.io.File filename,
                                                   boolean gzipped)
                                            throws java.io.IOException

Given a file, deserialises an object from it. This buffers the input for efficiency.

Parameters:

gzipped - if true, the file is assumed to be in GZIP format and the stream is decompressed on the fly

Throws:

java.io.IOException - if something bad happens

loadProperties

public static java.util.Properties loadProperties(java.io.InputStream is)
                                           throws java.io.IOException

Load properties for the given InputStream. In case of any problem reading the stream for properties an IOException is thrown.

Also, if the property ``end'' is not defined with the value OK, an EOFException is thrown to indicate that the file was not read completely (eg if we catch it in the middle of a save). So make sure that files are defined with this property at the end.

Throws:

java.io.IOException - in case of general I/O problems

java.io.EOFException - in case of missing end=OK value

compressDeflatableData

public static byte[] compressDeflatableData(byte[] in)

Compress data (maximally deflate without zlib/gzip headers and footers) from byte[] to byte[]; never null. Equivalent to ompressDeflatableData(in, 0, in.length).

This routine will not alter the content of the input array.

compressDeflatableData

public static byte[] compressDeflatableData(byte[] in,
                                            int off,
                                            int len)

Compress data (maximally deflate without zlib/gzip headers and footers) from byte[] to byte[]; never null. This means that there is no checksum, so any integrity checking required had better be done elsewhere.

This may increase the size of the data in some cases, especially where the input array is short.

This routine will not alter the content of the input array.

decompressDeflatedData

public static byte[] decompressDeflatedData(byte[] in)
                                     throws java.io.IOException

Decompress deflated data (without zlib/gzip headers and footers) from byte[] to byte[]. This means that there is no checksum, so any integrity checking required had better be done elsewhere.

This routine will not alter the content of the input array.

Throws:

java.io.IOException - in case of difficulty, such as corrupt data

roundUpToFSBlockSize

public static long roundUpToFSBlockSize(long length)

Rounds up a byte length to the next block size. Return is no smaller than input value.

Input value must be non-negative.

An input value of 0 results in a return value of 0.

wrapExhibitAsStream

public static AllExhibitProperties.ExhibitDataSource wrapExhibitAsStream(SimpleExhibitPipelineIF dataSource)

Wrap an exhibit as a stream; never null.

wrapExhibitAsRandomAccessData

public static FileTools.RandomAccessData wrapExhibitAsRandomAccessData(SimpleExhibitPipelineIF dataSource,
                                                                       Name.ExhibitFull exhibitName)

Wrap a single exhibit as RandomAccessData; never null.

wrapBytesAsRandomAccessData

public static FileTools.RandomAccessData wrapBytesAsRandomAccessData(byte[] data)

Wrap a byte[] as RandomAccessData; never null.

readFully

public static void readFully(java.io.InputStream is,
                             byte[] b,
                             int off,
                             int len)
                      throws java.io.IOException

Stand-alone blocking 'readFully()' for arbitrary InputStream. The underlying call to is.read(b, off, len) is assumed to validate all arguments.

Throws:

java.io.IOException

getZIPEntry

public static Tuple.Pair<java.util.zip.ZipEntry,ROByteArray> getZIPEntry(java.io.InputStream is,
                                                                         java.lang.String entryName)
                                                                  throws java.io.IOException

Get ZipEntry and (read-only, uncompressed) data for a named file/entry; null if none. A well-formed ZIP file should contain at least one entry, but this may return an empty result if the input is not well-formed.

See http://infozip.sourceforge.net/ and http://www.zlib.net/ and http://www.pkware.com/documents/casestudies/APPNOTE.TXT

The ZipEntry returned is a private copy.

We do not close the input stream.

Parameters:

is - input stream; must be open and non-null

Returns:

map from file/entry name to offset of start of entry from start of stream; never null

Throws:

java.io.IOException - if the input is malformed

getZIPEntriesLengthAndOffset

public static java.util.SortedMap<java.lang.CharSequence,FileTools.ZE> getZIPEntriesLengthAndOffset(FileTools.RandomAccessData rad)
                                                                                             throws java.io.IOException

Read (immutable) central directory from (end of) random-access 32-bit ZIP file; null if none. This allows for fast random access into a ZIP file (and quick rejection of attempts to fetch files not present at all).

The resultant map may have keys of mixed type (using String keys for non-8-bit filenames, else more-compact representations) but a lookup with any CharSequence key will work.

Returns:

map from entry/file name to (uncompressed) length and offset of entry from start of file, or null if no central directory or otherwise apparently not a well-formed ZIP archive

Throws:

java.io.IOException

estimatedFreeSpaceBelowReserve

public static final long estimatedFreeSpaceBelowReserve(java.io.File target,
                                                        int percentFree)

Compute available remaining usable bytes of space in filesystem containing given file/dir with specified reserve. Estimates remaining free space in bytes if we are to avoid over-filling the target filesystem.

Result may be restricted by the JVM's user ID, quotas, etc.

Returns:

-ve if unknown, 0L if no space, else +ve usable bytes available