Pipeline package¶

Main Package of the data pipeline. Contains the PyTorch data generator that provides the data and files with explicit loading functions.

Pipeline.data_gather¶

Contains functions for collecting the paths to training data.

Pipeline.data_gather.get_filelist_within_folder(root_directory)¶

Parameters:	root_directory (string) – Root directory from which all paths should be collected. –
Returns:	List of all .erfh5 files in the root_directory

Pipeline.data_gather.get_filelist_within_folder_blacklisted(root_directory: str)¶

Parameters:	root_directory (string) – Root directory from which all paths should be collected. –

Applies a blacklist, located in the root directory to the paths and thus does not return them.

Returns:	List of all .erfh5 files in the root_directory

Pipeline.data_gather.get_folders_within_folder(root_directory)¶

Pipeline.data_loader_sensor¶

Contains functions for loading sensor related data.

class Pipeline.data_loader_sensor.DataLoaderSensor(num_samples_sensorgrid=50, frm=0, until=-1)¶

Bases: object

get_sensordata_and_filling_percentage(file)¶

Args:

file (string): File from which the data should be extracted. until: End frame of sequence. frm: Starting frame of sequence.

Returns:	sequence of sensordata as data and the filling percentage at the last sequence state as label
Return type:	[(data, label)]

sensorgrid_simulationsuccess(file)¶

Pipeline.data_loaders¶

Contains functions for loading various data snippets from erfh5-files.

Pipeline.data_loaders.get_all_sequences_for_file(filename)¶

Returns:	All Sequences of a specified length that can be extracted from a file, filling percentage at that timestep.
Return type:	[(data, label)]

Pipeline.data_loaders.get_index_sequence(filename)¶

Returns:	a sequence of simulation steps as data and the filling percentage of the last step as label
Return type:	([data, label)]

Pipeline.data_loaders.get_single_states_and_fillings(filename)¶

Returns:	Returns single simulation states as data and label for an autoencoder.
Return type:	[(data, label)]

Pipeline.data_loaders_IMG¶

Contains functions for loading images as data.

class Pipeline.data_loaders_IMG.DataloaderImageSequences(image_size=(135, 103), wanted_frames=10)¶

Bases: Pipeline.data_loaders_IMG.DataloaderImages

Subclass for dataloader functions that generate sequences of images as samples and only generate one sample per file

get_images_of_flow_front_and_permeability_map(filename)¶

class Pipeline.data_loaders_IMG.DataloaderImages(image_size=(135, 103), ignore_useless_states=True, sensor_indizes=((0, 1), (0, 1)), skip_indizes=(0, None, 1), divide_by_100k=True)¶

Bases: object

get_sensordata_and_flowfront(file: pathlib.Path)¶

Pipeline.data_loader_dryspot¶

Contains the data loader for dryspot detection

class Pipeline.data_loader_dryspot.DataloaderDryspots(image_size=None, ignore_useless_states=True, sensor_indizes=((0, 1), (0, 1)), skip_indizes=(0, None, 1), divide_by_100k=True, aux_info=False)¶

Bases: object

get_flowfront_bool_dryspot(filename, states=None)¶: Load the flow front for the all states or given states. Returns a bool label for dryspots.

get_sensor_bool_dryspot(filename)¶: Load the flow front for the given states or all available states if states is None

Pipeline.TorchDataGeneratorUtils.looping_strategies¶

class Pipeline.TorchDataGeneratorUtils.looping_strategies.ComplexListLoopingStrategy(batch_size)¶

Bases: Pipeline.TorchDataGeneratorUtils.looping_strategies.LoopingStrategy

This strategy stores individual samples and shuffles these between epochs. This is pretty slow compared to the SimpleListLoopingStrategy, but it gives better results in training.

get_new_iterator()¶: This should return a new iterator. The new iterator must yield all samples that were previously stored using store. Yielded objects should be in the same batch form store expects. Also, the strategy is responsible for shuffling samples.

store(batch)¶

Store a new sample into the strategies buffer

Parameters:	batch (tuple of feature-batch and label-batch) –

class Pipeline.TorchDataGeneratorUtils.looping_strategies.DataLoaderListLoopingStrategy(batch_size, sampler=None)¶

Bases: Pipeline.TorchDataGeneratorUtils.looping_strategies.LoopingStrategy, sphinx.ext.autodoc.importer._MockObject

This strategy shuffles on a sample basis like the ComplexListLoopingStrategy, but it relies on the torch DataLoader for shuffling. It seems to have slightly better performance than the ComplexList approach.

dump_content(f)¶: Write the stored samples of this generator into the pickle f

get_new_iterator()¶: This should return a new iterator. The new iterator must yield all samples that were previously stored using store. Yielded objects should be in the same batch form store expects. Also, the strategy is responsible for shuffling samples.

load_content(f)¶: Load the stored samples for this generator from the pickle f

store(batch)¶

Store a new sample into the strategies buffer

Parameters:	batch (tuple of feature-batch and label-batch) –

class Pipeline.TorchDataGeneratorUtils.looping_strategies.LoopingStrategy¶

Bases: abc.ABC

LoopingStrategies are used to repeat samples after the first epoch. The LoopingDataGenerator will pass every loaded batch into the store function. Once a sample iterator is exhausted, a new one will be created using the get_new_iterator function.

dump_content(f)¶: Write the stored samples of this generator into the pickle f

get_new_iterator()¶: This should return a new iterator. The new iterator must yield all samples that were previously stored using store. Yielded objects should be in the same batch form store expects. Also, the strategy is responsible for shuffling samples.

load_content(f)¶: Load the stored samples for this generator from the pickle f

store(batch)¶

Store a new sample into the strategies buffer

Parameters:	batch (tuple of feature-batch and label-batch) –

class Pipeline.TorchDataGeneratorUtils.looping_strategies.NoOpLoopingStrategy¶

Bases: Pipeline.TorchDataGeneratorUtils.looping_strategies.LoopingStrategy

A “do-nothing” strategy that will just forget everything stored in it. This is automatically used if you only run a single epoch and will prevent the huge memory requirements that the other strategies have.

get_new_iterator()¶: This should return a new iterator. The new iterator must yield all samples that were previously stored using store. Yielded objects should be in the same batch form store expects. Also, the strategy is responsible for shuffling samples.

class Pipeline.TorchDataGeneratorUtils.looping_strategies.SimpleListLoopingStrategy¶

Bases: Pipeline.TorchDataGeneratorUtils.looping_strategies.LoopingStrategy

This strategy just stores batches in a list and shuffles that list between epochs. This strategy is really fast, but shuffling on a batch basis instead of samples reduces training performance and overall training results.

get_new_iterator()¶: This should return a new iterator. The new iterator must yield all samples that were previously stored using store. Yielded objects should be in the same batch form store expects. Also, the strategy is responsible for shuffling samples.

store(batch)¶

Store a new sample into the strategies buffer

Parameters:	batch (tuple of feature-batch and label-batch) –

Pipeline.TorchDataGeneratorUtils.looping_strategies.split_aux_dicts(aux)¶

Pipeline.TorchDataGeneratorUtils.looping_strategies.stack_aux_dicts(auxs)¶

Pipeline.TorchDataGeneratorUtils.torch_internal¶

class Pipeline.TorchDataGeneratorUtils.torch_internal.CachingMode¶

Bases: enum.Enum

Enum to specify if and how caching is done.

Nothing: No caching is done FileList: Only the list of files is cached.

FileList = 2¶

Nothing = 1¶

class Pipeline.TorchDataGeneratorUtils.torch_internal.FileDiscovery(gather_data, cache_path=None, cache_mode=<CachingMode.FileList: 2>)¶

Bases: object

A helper class to gather files from a set of base paths This class can be used to discover sample files in a set of directories.

Parameters:	gather_data (function) – A callable that gathers files given a single root directory. data_gather.get_filelist_within_folder is usually used for this. cache_path (str) – A directory to use for caching file lists, if required.

discover(data_paths)¶

Get a list of files for the given set of paths.

Parameters:	data_paths (list of str) – The set of paths to load
Returns:	A list of files that were found

class Pipeline.TorchDataGeneratorUtils.torch_internal.FileSetIterable(files, load_data)¶

Bases: sphinx.ext.autodoc.importer._MockObject

An Iterable meant to be used with the torch DataLoader.

Parameters:	files – A list of (typically HDF5 files) to load load_data (function) – A function that can load a list of samples given a filename MUST return the following format: [(data_1, label_1), … , (data_n, label_n)]

class Pipeline.TorchDataGeneratorUtils.torch_internal.FileSetIterator(files, load_data, worker_id=0)¶

Bases: object

An iterator for samples stored in a set of files. The FileSetIterator provides an iterator over the samples stored in a set of files. These files are usually HDF5-files.

Parameters:	files (list of str) – A list of paths to the files to be loaded load_data (function) – A function that can load a list of samples given a filename MUST return the following format: [(data_1, label_1), … , (data_n, label_n)] cache_path (Path) – A path to cache loaded samples worker_id (int) – The id of this worker for multiprocessing environments

get_remaining_files()¶

Get the list of remaining files

Returns:	A list of remaining files.

class Pipeline.TorchDataGeneratorUtils.torch_internal.SubSetGenerator(load_data, subset_name: str, num_samples: int, load_path=None, save_path=None, data_root=None, dont_care_num_samples=False)¶

Bases: object

This class is responsible for creating and loading test and validation splits. Given a set of filenames, it will load a subset of samples and return unused files.

Parameters:

load_data (function) – A function that can load a list of samples given a filename MUST return the following format: [(data_1, label_1), … , (data_n, label_n)]
subset_name (str) – The name of this subset
num_samples (int) – The number of samples in this subset
load_path (Path) – A path for loading existing splits
save_path (Path) – A path for saving the used splits

get_samples()¶

Get the list of samples for this subset. If it was not yet loaded, this will do so.

Returns:	A list of samples

prepare_subset(file_paths)¶

Prepare the subset from the given file paths. This will either load a subset of the files in file_path or load a stored split.

Parameters:	file_paths (list of Path) – A list of paths to files to load.
Returns:	A list of file paths that can still be used