Helpers ¶

curator.helpers.date_ops.get_datetime(index_timestamp, timestring)¶

Parameters:

index_timestamp (str) – The index timestamp
timestring (strftime()) – An strftime pattern

Returns:

The datetime extracted from the index name, which is the index creation time.

Return type:

datetime

curator.helpers.date_ops.get_point_of_reference(unit, count, epoch=None)¶

Parameters:

unit (str) – One of seconds, minutes, hours, days, weeks, months, or years.
unit_count (int) – The number of units. unit_count * unit will be calculated out to the relative number of seconds.
epoch (int) – An epoch timestamp used in conjunction with unit and unit_count to establish a point of reference for calculations.

Returns:

A point-of-reference timestamp in epoch + milliseconds by deriving from a unit and a count, and an optional reference timestamp, epoch

Return type:

int

curator.helpers.date_ops.get_unit_count_from_name(index_name, pattern)¶

Parameters:

index_name (str) – An index name
pattern (str) – A regular expression pattern

Returns:

The unit count, if a match is able to be found in the name

Return type:

int

curator.helpers.date_ops.handle_iso_week_number(mydate, timestring, index_timestamp)¶

Parameters:

mydate (datetime) – A Python datetime
timestring (strftime()) – An strftime pattern
index_timestamp (str) – The index timestamp

Returns:

The date of the previous week based on ISO week number

Return type:

datetime

curator.helpers.date_ops.isdatemath(data)¶

Parameters:: data (str) – An expression to validate as being datemath or not
Returns:: True if data is a valid datemath expression, else False
Return type:: bool

curator.helpers.date_ops.parse_date_pattern(name)¶

Scan and parse name for strftime() strings, replacing them with the associated value when found, but otherwise returning lowercase values, as uppercase snapshot names are not allowed. It will detect if the first character is a <, which would indicate name is going to be using Elasticsearch date math syntax, and skip accordingly.

The strftime() identifiers that Curator currently recognizes as acceptable include:

Y: A 4 digit year
y: A 2 digit year
m: The 2 digit month
W: The 2 digit week of the year
d: The 2 digit day of the month
H: The 2 digit hour of the day, in 24 hour notation
M: The 2 digit minute of the hour
S: The 2 digit number of second of the minute
j: The 3 digit day of the year

Parameters:: name (str) – A name, which can contain strftime() strings
Returns:: The parsed date pattern
Return type:: str

curator.helpers.date_ops.parse_datemath(client, value)¶

Validate that value looks like proper datemath. If it passes this test, then try to ship it to Elasticsearch for real. It may yet fail this test, and if it does, it will raise a ConfigurationError exception. If it passes, return the fully parsed string.

Parameters:

client (Elasticsearch) – A client connection object
value (str) – A string to check for datemath

Returns:

A datemath indexname, fully rendered by Elasticsearch

Return type:

Getters¶

curator.helpers.getters.byte_size(num, suffix='B')¶

Parameters:

num (int) – The number of byte
suffix (str) – An arbitrary suffix, like Bytes

Returns:

A formatted string indicating the size in bytes, with the proper unit, e.g. KB, MB, GB, TB, etc.

Return type:

float

curator.helpers.getters.get_alias_actions(oldidx, newidx, aliases)¶

Parameters:

oldidx (str) – The old index name
newidx (str) – The new index name
aliases (dict) – The aliases

Returns:

A list of actions suitable for update_aliases() actions kwarg.

Return type:

curator.helpers.getters.get_data_tiers(client)¶

Get all valid data tiers from the node roles of each node in the cluster by polling each node

Parameters:: client (Elasticsearch) – A client connection object
Returns:: The available data tiers in tier: bool form.
Return type:: dict

curator.helpers.getters.get_indices(client, search_pattern='_all')¶

Calls indices()

Parameters:: client (Elasticsearch) – A client connection object
Returns:: The current list of indices from the cluster
Return type:: list

curator.helpers.getters.get_repository(client, repository='')¶

Calls get_repository()

Parameters:

client (Elasticsearch) – A client connection object
repository (str) – The Elasticsearch snapshot repository to use

Returns:

Configuration information for repository.

Return type:

dict

curator.helpers.getters.get_snapshot(client, repository=None, snapshot='')¶

Calls get()

Parameters:

client (Elasticsearch) – A client connection object
repository (str) – The Elasticsearch snapshot repository to use
snapshot (str) – The snapshot name, or a comma-separated list of snapshots

Returns:

Information about the provided snapshot, a snapshot (or a comma-separated list of snapshots). If no snapshot specified, it will collect info for all snapshots. If none exist, an empty dict will be returned.

Return type:

dict

curator.helpers.getters.get_snapshot_data(client, repository=None)¶

Get all snapshots from repository and return a list. Calls get()

Parameters:

client (Elasticsearch) – A client connection object
repository (str) – The Elasticsearch snapshot repository to use

Returns:

The list of all snapshots from repository

Return type:

curator.helpers.getters.get_tier_preference(client, target_tier='data_frozen')¶

Do the tier preference thing in reverse order from coldest to hottest Based on the value of target_tier, build out the list to use.

Parameters:

client (Elasticsearch) – A client connection object
target_tier (str) – The target data tier, e.g. data_warm.

Returns:

A suitable tier preference string in csv format

Return type:

curator.helpers.getters.get_write_index(client, alias)¶

Calls get_alias()

Parameters:

client (Elasticsearch) – A client connection object
alias (str) – An alias name

Returns:

The the index name associated with the alias that is designated is_write_index

Return type:

curator.helpers.getters.index_size(client, idx, value='total')¶

Calls stats()

Parameters:

client (Elasticsearch) – A client connection object
idx (str) – An index name
value (str) – One of either primaries or total

Returns:

The sum of either primaries or total shards for index idx

Return type:

integer

curator.helpers.getters.name_to_node_id(client, name)¶

Calls stats()

Parameters:

client (Elasticsearch) – A client connection object
name (str) – The node name

Returns:

The node_id of the node identified by name

Return type:

curator.helpers.getters.node_id_to_name(client, node_id)¶

Calls stats()

Parameters:

client (Elasticsearch) – A client connection object
node_id (str) – The node node_id

Returns:

The name of the node identified by node_id

Return type:

curator.helpers.getters.node_roles(client, node_id)¶

Calls info()

Parameters:

client (Elasticsearch) – A client connection object
node_id (str) – The node node_id

Returns:

The list of roles assigned to the node identified by node_id

Return type:

curator.helpers.getters.single_data_path(client, node_id)¶

In order for a shrink to work, it should be on a single filesystem, as shards cannot span filesystems. Calls stats()

Parameters:

client (Elasticsearch) – A client connection object
node_id (str) – The node node_id

Returns:

True if the node has a single filesystem, else False

Return type:

Testers¶

curator.helpers.testers.ilm_policy_check(client, alias)¶

Test if alias is associated with an ILM policy

Calls get_settings()

Parameters:

client (Elasticsearch) – A client connection object
alias (str) – The alias name

Return type:

curator.helpers.testers.repository_exists(client, repository=None)¶

Calls get_repository()

Parameters:

client (Elasticsearch) – A client connection object
repository (str) – The Elasticsearch snapshot repository to use

Returns:

True if repository exists, else False

Return type:

curator.helpers.testers.rollable_alias(client, alias)¶

Calls get_alias()

Parameters:

client (Elasticsearch) – A client connection object
alias (str) – An Elasticsearch alias

Returns:

True or False depending on whether alias is an alias that points to an index that can be used by the _rollover API.

Return type:

curator.helpers.testers.snapshot_running(client)¶

Calls get_repository()

Return True if a snapshot is in progress, and False if not

Parameters:: client (Elasticsearch) – A client connection object
Return type:: bool

curator.helpers.testers.validate_actions(data)¶

Validate the actions configuration dictionary, as imported from actions.yml, for example.

Parameters:: data (dict) – The configuration dictionary
Returns:: The validated and sanitized configuration dictionary.
Return type:: dict

curator.helpers.testers.validate_filters(action, myfilters)¶

Validate that myfilters are appropriate for the action type, e.g. no index filters applied to a snapshot list.

Parameters:

action (str) – An action name
myfilters (list) – A list of filters to test.

Returns:

Validated list of filters

Return type:

curator.helpers.testers.verify_client_object(test)¶

Parameters:: test (Elasticsearch) – The variable or object to test
Returns:: True if test is a proper Elasticsearch client object and raise a TypeError exception if it is not.
Return type:: bool

curator.helpers.testers.verify_repository(client, repository=None)¶

Do verify_repository() call. If it fails, raise a RepositoryException.

Parameters:

client (Elasticsearch) – A client connection object
repository (str) – A repository name

Return type:

None

curator.helpers.testers.verify_snapshot_list(test)¶

Parameters:: test (SnapshotList) – The variable or object to test
Returns:: None if test is a proper SnapshotList object, else raise a TypeError exception.
Return type:: None

Utils¶

curator.helpers.utils.chunk_index_list(indices)¶

This utility chunks very large index lists into 3KB chunks. It measures the size as a csv string, then converts back into a list for the return value.

Parameters:: indices (list) – The list of indices
Returns:: A list of lists (each a piece of the original indices)
Return type:: list

curator.helpers.utils.report_failure(exception)¶

Raise a FailedExecution exception and include the original error message.

Parameters:: exception (:py:exc:Exception) – The upstream exception.
Return type:: None

curator.helpers.utils.show_dry_run(ilo, action, **kwargs)¶

Log dry run output with the action which would have been executed.

Parameters:

ilo (IndexList) – An IndexList Object
action (str) – The action to be performed.
kwargs (dict) – Any other args to show in the log output

Return type:

None

curator.helpers.utils.to_csv(indices)¶

Parameters:: indices (list) – A list of indices to act on, or a single value, which could be in the format of a csv string already.
Returns:: A csv string from a list of indices, or a single value if only one value is present
Return type:: str

Waiters¶

curator.helpers.waiters.health_check(client, **kwargs)¶

This function calls client.cluster. health() and, based on the params provided, will return True or False depending on whether that particular keyword appears in the output, and has the expected value.

If multiple keys are provided, all must match for a True response.

Parameters:: client (Elasticsearch) – A client connection object
Return type:: bool

curator.helpers.waiters.relocate_check(client, index)¶

This function calls client.cluster. state() with a given index to check if all of the shards for that index are in the STARTED state. It will return True if all primary and replica shards are in the STARTED state, and it will return False if any shard is in a different state.

Parameters:

client (Elasticsearch) – A client connection object
index (str) – The index name

Return type:

curator.helpers.waiters.restore_check(client, index_list)¶

This function calls client.indices. recovery() with the list of indices to check for complete recovery. It will return True if recovery of those indices is complete, and False otherwise. It is designed to fail fast: if a single shard is encountered that is still recovering (not in DONE stage), it will immediately return False, rather than complete iterating over the rest of the response.

Parameters:

client (Elasticsearch) – A client connection object
index_list (list) – The list of indices to verify having been restored.

Return type:

curator.helpers.waiters.snapshot_check(client, snapshot=None, repository=None)¶

This function calls client.snapshot. get() and tests to see whether the snapshot is complete, and if so, with what status. It will log errors according to the result. If the snapshot is still IN_PROGRESS, it will return False. SUCCESS will be an INFO level message, PARTIAL nets a WARNING message, FAILED is an ERROR, message, and all others will be a WARNING level message.

Parameters:

client (Elasticsearch) – A client connection object
snapshot (str) – The snapshot name
repository (str) – The repository name

Return type:

curator.helpers.waiters.task_check(client, task_id=None)¶

This function calls client.tasks. get() with the provided task_id. If the task data contains 'completed': True, then it will return True. If the task is not completed, it will log some information about the task and return False

Parameters:

client (Elasticsearch) – A client connection object
task_id (str) – The task id

Return type: