asaptools.partition module

A module for data partitioning functions.

This provides a collection of ‘partitioning’ functions. A partitioning function is a three-argument function that takes, as the first argument, a given data object and, as the second argument, an index into that object and, as the third argument, a maximum index. The operation of the partitioning function is to return a subset of the data corresponding to the given index.

By design, partitioning functions should keep the data “unchanged” except for subselecting parts of the data.

Copyright 2020 University Corporation for Atmospheric Research

Licensed under the Apache License, Version 2.0 (the “License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an “AS IS” BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

class asaptools.partition.Duplicate

Bases: asaptools.partition.PartitionFunction

Return a copy of the original input data in each partition.

class asaptools.partition.EqualLength

Bases: asaptools.partition.PartitionFunction

Partition an indexable object by striding through the data.

The initial object is “chopped” along its length into roughly equal length sublists. If the partition size is greater than the length of the input data, then it will return an empty list for ‘empty’ partitions. If the data is not indexable, then it will return the data for index=0 only, and an empty list otherwise.

class asaptools.partition.EqualStride

Bases: asaptools.partition.PartitionFunction

Partition an object by chopping the data into roughly equal lengths.

This returns a sublist of an indexable object by “striding” through the data in steps equal to the partition size. If the partition size is greater than the length of the input data, then it will return an empty list for “empty” partitions. If the data is not indexable, then it will return the data for index=0 only, and an empty list otherwise.

class asaptools.partition.PartitionFunction

Bases: object

The abstract base-class for all Partitioning Function objects.

A PartitionFunction object is one with a __call__ method that takes three arguments. The first argument is the data to be partitioned, the second argument is the index of the partition (or part) requested, and third argument is the number of partitions to assume when dividing the data.

class asaptools.partition.SortedStride

Bases: asaptools.partition.PartitionFunction

Partition an indexable list of pairs by striding through sorted data.

The first index of each pair is assumed to be an item of data (which will be partitioned), and the second index in each pair is assumed to be a numeric weight. The pairs are first sorted by weight, and then partitions are returned by striding through the sorted data.

The results are partitions of roughly equal length and roughly equal total weight. However, equal length is prioritized over total weight.

class asaptools.partition.WeightBalanced

Bases: asaptools.partition.PartitionFunction

Partition an indexable list of pairs by balancing the total weight.

The first index of each pair is assumed to be an item of data (which will be partitioned), and the second index in each pair is assumed to be a numeric weight. The data items are grouped via a “greedy” binning algorithm into partitions of roughly equal total weight.

The results are partitions of roughly equal length and roughly equal total weight. However, equal total weight is prioritized over length.