1CURATOR(1) Elasticsearch Curator CURATOR(1)
2
6 curator - Elasticsearch Curator Documentation
7 The Elasticsearch Curator Python API helps you manage your indices and
9 snapshots.
10 NOTE:
12 This documentation is for the Elasticsearch Curator Python API.
13 Documentation for the Elasticsearch Curator CLI -- which uses this
14 API and is installed as an entry_point as part of the package -- is
15 available in the Elastic guide.
16
18 The Elasticsearch Curator Python API is compatible with Elasticsearch
19 versions 2.x through 5.0, and supports Python versions 2.6 and later.
20
22 import elasticsearch
23 import curator
24 client = elasticsearch.Elasticsearch()
26 ilo = curator.IndexList(client)
28 ilo.filter_by_regex(kind='prefix', value='logstash-')
29 ilo.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d', unit='days', unit_count=30)
30 delete_indices = curator.DeleteIndices(ilo)
31 delete_indices.do_action()
32 TIP:
34 See more examples in the Examples page.
35
37 The API methods fall into the following categories:
38 · Object Classes build and filter index list or snapshot list objects.
40 · Action Classes act on object classes.
42 · Utilities are helper methods.
44 Logging
46 The Elasticsearch Curator Python API uses the standard logging library
47 from Python. It inherits two loggers from elasticsearch-py: elastic‐
48 search and elasticsearch.trace. Clients use the elasticsearch logger to
49 log standard activity, depending on the log level. The elastic‐
50 search.trace logger logs requests to the server in JSON format as
51 pretty-printed curl commands that you can execute from the command
52 line. The elasticsearch.trace logger is not inherited from the base
53 logger and must be activated separately.
54
56 Object Classes
57 · IndexList
58 · SnapshotList
60 IndexList
62 class curator.indexlist.IndexList(client)
63 all_indices = None
65 Instance variable. All indices in the cluster at
66 instance creation time. Type: list()
67 client = None
69 An Elasticsearch Client object Also accessible as an
70 instance variable.
71 empty_list_check()
73 Raise exception if indices is empty
74 filter_allocated(key=None, value=None, alloca‐
76 tion_type='require', exclude=True)
77 Match indices that have the routing allocation rule of
78 key=value from indices
79 Parameters
81 · key -- The allocation attribute to check for
83 · value -- The value to check for
85 · allocation_type -- Type of allocation to apply
87 · exclude -- If exclude is True, this filter will
89 remove matching indices from indices. If exclude
90 is False, then only matching indices will be
91 kept in indices. Default is True
92 filter_by_age(source='name', direction=None, timestring=None,
94 unit=None, unit_count=None, field=None,
95 stats_result='min_value', epoch=None, exclude=False)
96 Match indices by relative age calculations.
97 Parameters
99 · source -- Source of index age. Can be one of
101 'name', 'creation_date', or 'field_stats'
102 · direction -- Time to filter, either older or
104 younger
105 · timestring -- An strftime string to match the
107 datestamp in an index name. Only used for index
108 filtering by name.
109 · unit -- One of seconds, minutes, hours, days,
111 weeks, months, or years.
112 · unit_count -- The number of unit``s.
114 ``unit_count * unit will be calculated out to
115 the relative number of seconds.
116 · field -- A timestamp field name. Only used for
118 field_stats based calculations.
119 · stats_result -- Either min_value or max_value.
121 Only used in conjunction with
122 source`=``field_stats` to choose whether to ref‐
123 erence the minimum or maximum result value.
124 · epoch -- An epoch timestamp used in conjunction
126 with unit and unit_count to establish a point of
127 reference for calculations. If not provided,
128 the current time will be used.
129 · exclude -- If exclude is True, this filter will
131 remove matching indices from indices. If exclude
132 is False, then only matching indices will be
133 kept in indices. Default is False
134 filter_by_alias(aliases=None, exclude=False)
136 Match indices which are associated with the alias identi‐
137 fied by name
138 Parameters
140 · aliases (list) -- A list of alias names.
142 · exclude -- If exclude is True, this filter will
144 remove matching indices from indices. If exclude
145 is False, then only matching indices will be
146 kept in indices. Default is False
147 filter_by_count(count=None, reverse=True, use_age=False,
149 source='creation_date', timestring=None, field=None,
150 stats_result='min_value', exclude=True)
151 Remove indices from the actionable list beyond the number
152 count, sorted reverse-alphabetically by default. If you
153 set reverse to False, it will be sorted alphabetically.
154 The default is usually what you will want. If only one
156 kind of index is provided--for example, indices matching
157 logstash-%Y.%m.%d--then reverse alphabetical sorting will
158 mean the oldest will remain in the list, because lower
159 numbers in the dates mean older indices.
160 By setting reverse to False, then index3 will be deleted
162 before index2, which will be deleted before index1
163 use_age allows ordering indices by age. Age is determined
165 by the index creation date by default, but you can spec‐
166 ify an source of name, max_value, or min_value. The name
167 source requires the timestring argument.
168 Parameters
170 · count -- Filter indices beyond count.
172 · reverse -- The filtering direction. (default:
174 True).
175 · use_age -- Sort indices by age. source is
177 required in this case.
178 · source -- Source of index age. Can be one of
180 name, creation_date, or field_stats. Default:
181 creation_date
182 · timestring -- An strftime string to match the
184 datestamp in an index name. Only used if source
185 name is selected.
186 · field -- A timestamp field name. Only used if
188 source field_stats is selected.
189 · stats_result -- Either min_value or max_value.
191 Only used if source field_stats is selected. It
192 determines whether to reference the minimum or
193 maximum value of field in each index.
194 · exclude -- If exclude is True, this filter will
196 remove matching indices from indices. If exclude
197 is False, then only matching indices will be
198 kept in indices. Default is True
199 filter_by_regex(kind=None, value=None, exclude=False)
201 Match indices by regular expression (pattern).
202 Parameters
204 · kind -- Can be one of: suffix, prefix, regex, or
206 timestring. This option defines what kind of
207 filter you will be building.
208 · value -- Depends on kind. It is the strftime
210 string if kind is timestring. It's used to build
211 the regular expression for other kinds.
212 · exclude -- If exclude is True, this filter will
214 remove matching indices from indices. If exclude
215 is False, then only matching indices will be
216 kept in indices. Default is False
217 filter_by_space(disk_space=None, reverse=True, use_age=False,
219 source='creation_date', timestring=None, field=None,
220 stats_result='min_value', exclude=False)
221 Remove indices from the actionable list based on space
222 consumed, sorted reverse-alphabetically by default. If
223 you set reverse to False, it will be sorted alphabeti‐
224 cally.
225 The default is usually what you will want. If only one
227 kind of index is provided--for example, indices matching
228 logstash-%Y.%m.%d--then reverse alphabetical sorting will
229 mean the oldest will remain in the list, because lower
230 numbers in the dates mean older indices.
231 By setting reverse to False, then index3 will be deleted
233 before index2, which will be deleted before index1
234 use_age allows ordering indices by age. Age is determined
236 by the index creation date by default, but you can spec‐
237 ify an source of name, max_value, or min_value. The name
238 source requires the timestring argument.
239 Parameters
241 · disk_space -- Filter indices over n gigabytes
243 · reverse -- The filtering direction. (default:
245 True). Ignored if use_age is True
246 · use_age -- Sort indices by age. source is
248 required in this case.
249 · source -- Source of index age. Can be one of
251 name, creation_date, or field_stats. Default:
252 creation_date
253 · timestring -- An strftime string to match the
255 datestamp in an index name. Only used if source
256 name is selected.
257 · field -- A timestamp field name. Only used if
259 source field_stats is selected.
260 · stats_result -- Either min_value or max_value.
262 Only used if source field_stats is selected. It
263 determines whether to reference the minimum or
264 maximum value of field in each index.
265 · exclude -- If exclude is True, this filter will
267 remove matching indices from indices. If exclude
268 is False, then only matching indices will be
269 kept in indices. Default is False
270 filter_closed(exclude=True)
272 Filter out closed indices from indices
273 Parameters
275 exclude -- If exclude is True, this filter will
276 remove matching indices from indices. If exclude
277 is False, then only matching indices will be kept
278 in indices. Default is True
279 filter_forceMerged(max_num_segments=None, exclude=True)
281 Match any index which has max_num_segments per shard or
282 fewer in the actionable list.
283 Parameters
285 · max_num_segments -- Cutoff number of segments
287 per shard.
288 · exclude -- If exclude is True, this filter will
290 remove matching indices from indices. If exclude
291 is False, then only matching indices will be
292 kept in indices. Default is True
293 filter_kibana(exclude=True)
295 Match any index named .kibana, kibana-int, .mar‐
296 vel-kibana, or .marvel-es-data in indices.
297 Parameters
299 exclude -- If exclude is True, this filter will
300 remove matching indices from indices. If exclude
301 is False, then only matching indices will be kept
302 in indices. Default is True
303 filter_opened(exclude=True)
305 Filter out opened indices from indices
306 Parameters
308 exclude -- If exclude is True, this filter will
309 remove matching indices from indices. If exclude
310 is False, then only matching indices will be kept
311 in indices. Default is True
312 index_info = None
314 Instance variable. Information extracted from indices,
315 such as segment count, age, etc. Populated at instance
316 creation time, and by other private helper methods, as
317 needed. Type: dict()
318 indices = None
320 Instance variable. The running list of indices which
321 will be used by an Action class. Populated at instance
322 creation time. Type: list()
323 iterate_filters(filter_dict)
325 Iterate over the filters defined in config and execute
326 them.
327 Parameters
329 filter_dict -- The configuration dictionary
330 NOTE:
332 filter_dict should be a dictionary with the following
333 form:
334 { 'filters' : [
336 {
337 'filtertype': 'the_filter_type',
338 'key1' : 'value1',
339 ...
340 'keyN' : 'valueN'
341 }
342 ]
343 }
344 working_list()
346 Return the current value of indices as copy-by-value to
347 prevent list stomping during iterations
348 SnapshotList
350 class curator.snapshotlist.SnapshotList(client, repository=None)
351 client = None
353 An Elasticsearch Client object. Also accessible as an
354 instance variable.
355 empty_list_check()
357 Raise exception if snapshots is empty
358 filter_by_age(source='creation_date', direction=None,
360 timestring=None, unit=None, unit_count=None, epoch=None,
361 exclude=False)
362 Remove snapshots from snapshots by relative age calcula‐
363 tions.
364 Parameters
366 · source -- Source of snapshot age. Can be 'name',
368 or 'creation_date'.
369 · direction -- Time to filter, either older or
371 younger
372 · timestring -- An strftime string to match the
374 datestamp in an snapshot name. Only used for
375 snapshot filtering by name.
376 · unit -- One of seconds, minutes, hours, days,
378 weeks, months, or years.
379 · unit_count -- The number of unit``s.
381 ``unit_count * unit will be calculated out to
382 the relative number of seconds.
383 · epoch -- An epoch timestamp used in conjunction
385 with unit and unit_count to establish a point of
386 reference for calculations. If not provided,
387 the current time will be used.
388 · exclude -- If exclude is True, this filter will
390 remove matching snapshots from snapshots. If
391 exclude is False, then only matching snapshots
392 will be kept in snapshots. Default is False
393 filter_by_count(count=None, reverse=True, use_age=False,
395 source='creation_date', timestring=None, exclude=True)
396 Remove snapshots from the actionable list beyond the num‐
397 ber count, sorted reverse-alphabetically by default. If
398 you set reverse to False, it will be sorted alphabeti‐
399 cally.
400 The default is usually what you will want. If only one
402 kind of snapshot is provided--for example, snapshots
403 matching curator-%Y%m%d%H%M%S-- then reverse alphabetical
404 sorting will mean the oldest will remain in the list,
405 because lower numbers in the dates mean older snapshots.
406 By setting reverse to False, then snapshot3 will be acted
408 on before snapshot2, which will be acted on before snap‐
409 shot1
410 use_age allows ordering snapshots by age. Age is deter‐
412 mined by the snapshot creation date (as identified by
413 start_time_in_millis) by default, but you can also spec‐
414 ify a source of name. The name source requires the
415 timestring argument.
416 Parameters
418 · count -- Filter snapshots beyond count.
420 · reverse -- The filtering direction. (default:
422 True).
423 · use_age -- Sort snapshots by age. source is
425 required in this case.
426 · source -- Source of snapshot age. Can be one of
428 name, or creation_date. Default: creation_date
429 · timestring -- An strftime string to match the
431 datestamp in a snapshot name. Only used if
432 source name is selected.
433 · exclude -- If exclude is True, this filter will
435 remove matching snapshots from snapshots. If
436 exclude is False, then only matching snapshots
437 will be kept in snapshots. Default is True
438 filter_by_regex(kind=None, value=None, exclude=False)
440 Filter out snapshots not matching the pattern, or in the
441 case of exclude, filter those matching the pattern.
442 Parameters
444 · kind -- Can be one of: suffix, prefix, regex, or
446 timestring. This option defines what kind of
447 filter you will be building.
448 · value -- Depends on kind. It is the strftime
450 string if kind is timestring. It's used to build
451 the regular expression for other kinds.
452 · exclude -- If exclude is True, this filter will
454 remove matching snapshots from snapshots. If
455 exclude is False, then only matching snapshots
456 will be kept in snapshots. Default is False
457 filter_by_state(state=None, exclude=False)
459 Filter out snapshots not matching state, or in the case
460 of exclude, filter those matching state.
461 Parameters
463 · state -- The snapshot state to filter for. Must
465 be one of SUCCESS, PARTIAL, FAILED, or
466 IN_PROGRESS.
467 · exclude -- If exclude is True, this filter will
469 remove matching snapshots from snapshots. If
470 exclude is False, then only matching snapshots
471 will be kept in snapshots. Default is False
472 iterate_filters(config)
474 Iterate over the filters defined in config and execute
475 them.
476 Parameters
478 config -- A dictionary of filters, as extracted
479 from the YAML configuration file.
480 NOTE:
482 config should be a dictionary with the following form:
483 { 'filters' : [
485 {
486 'filtertype': 'the_filter_type',
487 'key1' : 'value1',
488 ...
489 'keyN' : 'valueN'
490 }
491 ]
492 }
493 most_recent()
495 Return the most recent snapshot based on
496 start_time_in_millis.
497 repository = None
499 An Elasticsearch repository. Also accessible as an
500 instance variable.
501 snapshot_info = None
503 Instance variable. Information extracted from snapshots,
504 such as age, etc. Populated by internal method
505 __get_snapshots at instance creation time. Type: dict()
506 snapshots = None
508 Instance variable. The running list of snapshots which
509 will be used by an Action class. Populated by internal
510 methods __get_snapshots at instance creation time. Type:
511 list()
512 working_list()
514 Return the current value of snapshots as copy-by-value to
515 prevent list stomping during iterations
516 Action Classes
518 SEE ALSO:
519 It is important to note that each action has a do_action() method,
520 which accepts no arguments. This is the means by which all actions
521 are executed.
522 · Alias
524 · Allocation
526 · Close
528 · ClusterRouting
530 · DeleteIndices
532 · DeleteSnapshots
534 · ForceMerge
536 · Open
538 · Replicas
540 · Snapshot
542 Alias
544 class curator.actions.Alias(name=None, extra_settings={})
545 Define the Alias object.
546 Parameters
548 · name -- The alias name
550 · extra_settings (dict, representing the settings.) --
552 Extra settings, including filters and routing. For more
553 information see
554 https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-aliases.html
555 actions = None
557 The list of actions to perform. Populated by
558 curator.actions.Alias.add and
559 curator.actions.Alias.remove
560 add(ilo)
562 Create add statements for each index in ilo for alias,
563 then append them to actions. Add any extras that may be
564 there.
565 Parameters
567 ilo -- A curator.indexlist.IndexList object
568 body() Return a body string suitable for use with the
570 update_aliases API call.
571 client = None
573 Instance variable. The Elasticsearch Client object
574 derived from ilo
575 do_action()
577 Run the API call update_aliases with the results of
578 body()
579 do_dry_run()
581 Log what the output would be, but take no action.
582 extra_settings = None
584 Instance variable. Any extra things to add to the alias,
585 like filters, or routing.
586 name = None
588 Instance variable The strftime parsed version of name.
589 remove(ilo)
591 Create remove statements for each index in ilo for alias,
592 then append them to actions.
593 Parameters
595 ilo -- A curator.indexlist.IndexList object
596 Allocation
598 class curator.actions.Allocation(ilo, key=None, value=None, alloca‐
599 tion_type='require', wait_for_completion=False, timeout=30)
600 Parameters
602 · ilo -- A curator.indexlist.IndexList object
604 · key -- An arbitrary metadata attribute key. Must match
606 the key assigned to at least some of your nodes to have
607 any effect.
608 · value -- An arbitrary metadata attribute value. Must
610 correspond to values associated with key assigned to at
611 least some of your nodes to have any effect.
612 · allocation_type -- Type of allocation to apply. Default
614 is require
615 · wait_for_completion (bool) -- Wait (or not) for the
617 operation to complete before returning. (default:
618 False)
619 · timeout -- Number of seconds to wait_for_completion
621 NOTE:
623 See:
624 https://www.elastic.co/guide/en/elasticsearch/reference/current/shard-allocation-filtering.html
625 client = None
627 Instance variable. The Elasticsearch Client object
628 derived from ilo
629 do_action()
631 Change allocation settings for indices in
632 index_list.indices with the settings in body.
633 do_dry_run()
635 Log what the output would be, but take no action.
636 index_list = None
638 Instance variable. Internal reference to ilo
639 timeout = None
641 Instance variable. How long in seconds to wait_for_com‐
642 pletion before returning with an exception
643 wfc = None
645 Instance variable. Internal reference to wait_for_com‐
646 pletion
647 Close
649 class curator.actions.Close(ilo, delete_aliases=False)
650 Parameters
652 · ilo -- A curator.indexlist.IndexList object
654 · delete_aliases (bool) -- If True, will delete any asso‐
656 ciated aliases before closing indices.
657 client = None
659 Instance variable. The Elasticsearch Client object
660 derived from ilo
661 delete_aliases = None
663 Instance variable. Internal reference to delete_aliases
664 do_action()
666 Close open indices in index_list.indices
667 do_dry_run()
669 Log what the output would be, but take no action.
670 index_list = None
672 Instance variable. Internal reference to ilo
673 ClusterRouting
675 class curator.actions.ClusterRouting(client, routing_type=None, set‐
676 ting=None, value=None, wait_for_completion=False, timeout=30)
677 For now, the cluster routing settings are hardcoded to be tran‐
678 sient
679 Parameters
681 · client -- An elasticsearch.Elasticsearch client object
683 · routing_type -- Type of routing to apply. Either allo‐
685 cation or rebalance
686 · setting -- Currently, the only acceptable value for
688 setting is enable. This is here in case that changes.
689 · value -- Used only if setting is enable. Semi-dependent
691 on routing_type. Acceptable values for allocation and
692 rebalance are all, primaries, and none (string, not
693 NoneType). If routing_type is allocation, this can
694 also be new_primaries, and if rebalance, it can be
695 replicas.
696 · wait_for_completion (bool) -- Wait (or not) for the
698 operation to complete before returning. (default:
699 False)
700 · timeout -- Number of seconds to wait_for_completion
702 client = None
704 Instance variable. An elasticsearch.Elasticsearch client
705 object
706 do_action()
708 Change cluster routing settings with the settings in
709 body.
710 do_dry_run()
712 Log what the output would be, but take no action.
713 timeout = None
715 Instance variable. How long in seconds to wait_for_com‐
716 pletion before returning with an exception
717 wfc = None
719 Instance variable. Internal reference to wait_for_com‐
720 pletion
721 DeleteIndices
723 class curator.actions.DeleteIndices(ilo, master_timeout=30)
724 Parameters
726 · ilo -- A curator.indexlist.IndexList object
728 · master_timeout -- Number of seconds to wait for master
730 node response
731 client = None
733 Instance variable. The Elasticsearch Client object
734 derived from ilo
735 do_action()
737 Delete indices in index_list.indices
738 do_dry_run()
740 Log what the output would be, but take no action.
741 index_list = None
743 Instance variable. Internal reference to ilo
744 master_timeout = None
746 Instance variable. String value of master_timeout + 's',
747 for seconds.
748 DeleteSnapshots
750 class curator.actions.DeleteSnapshots(slo, retry_interval=120,
751 retry_count=3)
752 Parameters
754 · slo -- A curator.snapshotlist.SnapshotList object
756 · retry_interval -- Number of seconds to delay betwen
758 retries. Default: 120 (seconds)
759 · retry_count -- Number of attempts to make. Default: 3
761 client = None
763 Instance variable. The Elasticsearch Client object
764 derived from slo
765 do_action()
767 Delete snapshots in slo Retry up to retry_count times,
768 pausing retry_interval seconds between retries.
769 do_dry_run()
771 Log what the output would be, but take no action.
772 repository = None
774 Instance variable. The repository name derived from slo
775 retry_count = None
777 Instance variable. Internally accessible copy of
778 retry_count
779 retry_interval = None
781 Instance variable. Internally accessible copy of
782 retry_interval
783 snapshot_list = None
785 Instance variable. Internal reference to slo
786 ForceMerge
788 class curator.actions.ForceMerge(ilo, max_num_segments=None, delay=0)
789 Parameters
791 · ilo -- A curator.indexlist.IndexList object
793 · max_num_segments -- Number of segments per shard to
795 forceMerge
796 · delay -- Number of seconds to delay between forceMerge
798 operations
799 client = None
801 Instance variable. The Elasticsearch Client object
802 derived from ilo
803 delay = None
805 Instance variable. Internally accessible copy of delay
806 do_action()
808 forcemerge indices in index_list.indices
809 do_dry_run()
811 Log what the output would be, but take no action.
812 index_list = None
814 Instance variable. Internal reference to ilo
815 max_num_segments = None
817 Instance variable. Internally accessible copy of
818 max_num_segments
819 Open
821 class curator.actions.Open(ilo)
822 Parameters
824 ilo -- A curator.indexlist.IndexList object
825 client = None
827 Instance variable. The Elasticsearch Client object
828 derived from ilo
829 do_action()
831 Open closed indices in index_list.indices
832 do_dry_run()
834 Log what the output would be, but take no action.
835 index_list = None
837 Instance variable. Internal reference to ilo
838 Replicas
840 class curator.actions.Replicas(ilo, count=None, wait_for_comple‐
841 tion=False, timeout=30)
842 Parameters
844 · ilo -- A curator.indexlist.IndexList object
846 · count -- The count of replicas per shard
848 · wait_for_completion (bool) -- Wait (or not) for the
850 operation to complete before returning. (default:
851 False)
852 client = None
854 Instance variable. The Elasticsearch Client object
855 derived from ilo
856 count = None
858 Instance variable. Internally accessible copy of count
859 do_action()
861 Update the replica count of indices in index_list.indices
862 do_dry_run()
864 Log what the output would be, but take no action.
865 index_list = None
867 Instance variable. Internal reference to ilo
868 timeout = None
870 Instance variable. How long in seconds to wait_for_com‐
871 pletion before returning with an exception
872 wfc = None
874 Instance variable. Internal reference to wait_for_com‐
875 pletion
876 Snapshot
878 class curator.actions.Snapshot(ilo, repository=None, name=None,
879 ignore_unavailable=False, include_global_state=True, partial=False,
880 wait_for_completion=True, skip_repo_fs_check=False)
881 Parameters
883 · ilo -- A curator.indexlist.IndexList object
885 · repository -- The Elasticsearch snapshot repository to
887 use
888 · name -- What to name the snapshot.
890 · wait_for_completion (bool) -- Wait (or not) for the
892 operation to complete before returning. (default:
893 True)
894 · ignore_unavailable (bool) -- Ignore unavailable
896 shards/indices. (default: False)
897 · include_global_state (bool) -- Store cluster global
899 state with snapshot. (default: True)
900 · partial (bool) -- Do not fail if primary shard is
902 unavailable. (default: False)
903 · skip_repo_fs_check (bool) -- Do not validate write
905 access to repository on all cluster nodes before pro‐
906 ceeding. (default: False). Useful for shared filesys‐
907 tems where intermittent timeouts can affect validation,
908 but won't likely affect snapshot success.
909 body = None
911 Instance variable. Populated at instance creation time
912 by calling curator.utils.create_snapshot_body with
913 ilo.indices and the provided arguments: ignore_unavail‐
914 able, include_global_state, partial
915 client = None
917 Instance variable. The Elasticsearch Client object
918 derived from ilo
919 do_action()
921 Snapshot indices in index_list.indices, with options
922 passed.
923 do_dry_run()
925 Log what the output would be, but take no action.
926 get_state()
928 Get the state of the snapshot
929 index_list = None
931 Instance variable. Internal reference to ilo
932 name = None
934 Instance variable. The parsed version of name
935 report_state()
937 Log the state of the snapshot
938 repository = None
940 Instance variable. Internally accessible copy of reposi‐
941 tory
942 skip_repo_fs_check = None
944 Instance variable. Internally accessible copy of
945 skip_repo_fs_check
946 wait_for_completion = None
948 Instance variable. Internally accessible copy of
949 wait_for_completion
950 Restore
952 class curator.actions.Restore(slo, name=None, indices=None,
953 include_aliases=False, ignore_unavailable=False,
954 include_global_state=True, partial=False, rename_pattern=None,
955 rename_replacement=None, extra_settings={}, wait_for_completion=True,
956 skip_repo_fs_check=False)
957 Parameters
959 · slo -- A curator.snapshotlist.SnapshotList object
961 · name (str) -- Name of the snapshot to restore. If no
963 name is provided, it will restore the most recent snap‐
964 shot by age.
965 · indices (list) -- A list of indices to restore. If no
967 indices are provided, it will restore all indices in
968 the snapshot.
969 · include_aliases (bool) -- If set to True, restore
971 aliases with the indices. (default: False)
972 · ignore_unavailable (bool) -- Ignore unavailable
974 shards/indices. (default: False)
975 · include_global_state (bool) -- Store cluster global
977 state with snapshot. (default: True)
978 · partial (bool) -- Do not fail if primary shard is
980 unavailable. (default: False)
981 · rename_pattern (str) -- A regular expression pattern
983 with one or more captures, e.g. index_(.+)
984 · rename_replacement (str) -- A target index name pattern
986 with $# numbered references to the captures in
987 rename_pattern, e.g. restored_index_$1
988 · extra_settings (dict, representing the settings.) --
990 Extra settings, including shard count and settings to
991 omit. For more information see
992 https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html#_changing_index_settings_during_restore
993 · wait_for_completion (bool) -- Wait (or not) for the
995 operation to complete before returning. (default:
996 True)
997 · skip_repo_fs_check (bool) -- Do not validate write
999 access to repository on all cluster nodes before pro‐
1000 ceeding. (default: False). Useful for shared filesys‐
1001 tems where intermittent timeouts can affect validation,
1002 but won't likely affect snapshot success.
1003 body = None
1005 Instance variable. Populated at instance creation time
1006 from the other options
1007 client = None
1009 Instance variable. The Elasticsearch Client object
1010 derived from slo
1011 do_action()
1013 Restore indices with options passed.
1014 do_dry_run()
1016 Log what the output would be, but take no action.
1017 name = None
1019 Instance variable. Will use a provided snapshot name, or
1020 the most recent snapshot in slo
1021 py_rename_replacement = None
1023 Also an instance variable version of rename_replacement
1024 but with Java regex group designations of $# converted to
1025 Python's \\# style.
1026 rename_pattern = None
1028 Instance variable version of rename_pattern
1029 rename_replacement = None
1031 Instance variable version of rename_replacement
1032 report_state()
1034 Log the state of the restore This should only be done if
1035 wait_for_completion is True, and only after completing
1036 the restore.
1037 repository = None
1039 Instance variable. repository derived from slo
1040 skip_repo_fs_check = None
1042 Instance variable. Internally accessible copy of
1043 skip_repo_fs_check
1044 snapshot_list = None
1046 Instance variable. Internal reference to slo
1047 Filter Methods
1049 · IndexList
1050 · SnapshotList
1052 IndexList
1054 IndexList.filter_allocated(key=None, value=None, alloca‐
1055 tion_type='require', exclude=True)
1056 Match indices that have the routing allocation rule of key=value
1057 from indices
1058 Parameters
1060 · key -- The allocation attribute to check for
1062 · value -- The value to check for
1064 · allocation_type -- Type of allocation to apply
1066 · exclude -- If exclude is True, this filter will remove
1068 matching indices from indices. If exclude is False,
1069 then only matching indices will be kept in indices.
1070 Default is True
1071 IndexList.filter_by_age(source='name', direction=None, timestring=None,
1073 unit=None, unit_count=None, field=None, stats_result='min_value',
1074 epoch=None, exclude=False)
1075 Match indices by relative age calculations.
1076 Parameters
1078 · source -- Source of index age. Can be one of 'name',
1080 'creation_date', or 'field_stats'
1081 · direction -- Time to filter, either older or younger
1083 · timestring -- An strftime string to match the datestamp
1085 in an index name. Only used for index filtering by
1086 name.
1087 · unit -- One of seconds, minutes, hours, days, weeks,
1089 months, or years.
1090 · unit_count -- The number of unit``s. ``unit_count *
1092 unit will be calculated out to the relative number of
1093 seconds.
1094 · field -- A timestamp field name. Only used for
1096 field_stats based calculations.
1097 · stats_result -- Either min_value or max_value. Only
1099 used in conjunction with source`=``field_stats` to
1100 choose whether to reference the minimum or maximum
1101 result value.
1102 · epoch -- An epoch timestamp used in conjunction with
1104 unit and unit_count to establish a point of reference
1105 for calculations. If not provided, the current time
1106 will be used.
1107 · exclude -- If exclude is True, this filter will remove
1109 matching indices from indices. If exclude is False,
1110 then only matching indices will be kept in indices.
1111 Default is False
1112 IndexList.filter_by_regex(kind=None, value=None, exclude=False)
1114 Match indices by regular expression (pattern).
1115 Parameters
1117 · kind -- Can be one of: suffix, prefix, regex, or
1119 timestring. This option defines what kind of filter you
1120 will be building.
1121 · value -- Depends on kind. It is the strftime string if
1123 kind is timestring. It's used to build the regular
1124 expression for other kinds.
1125 · exclude -- If exclude is True, this filter will remove
1127 matching indices from indices. If exclude is False,
1128 then only matching indices will be kept in indices.
1129 Default is False
1130 IndexList.filter_by_space(disk_space=None, reverse=True, use_age=False,
1132 source='creation_date', timestring=None, field=None,
1133 stats_result='min_value', exclude=False)
1134 Remove indices from the actionable list based on space consumed,
1135 sorted reverse-alphabetically by default. If you set reverse to
1136 False, it will be sorted alphabetically.
1137 The default is usually what you will want. If only one kind of
1139 index is provided--for example, indices matching
1140 logstash-%Y.%m.%d--then reverse alphabetical sorting will mean
1141 the oldest will remain in the list, because lower numbers in the
1142 dates mean older indices.
1143 By setting reverse to False, then index3 will be deleted before
1145 index2, which will be deleted before index1
1146 use_age allows ordering indices by age. Age is determined by the
1148 index creation date by default, but you can specify an source of
1149 name, max_value, or min_value. The name source requires the
1150 timestring argument.
1151 Parameters
1153 · disk_space -- Filter indices over n gigabytes
1155 · reverse -- The filtering direction. (default: True).
1157 Ignored if use_age is True
1158 · use_age -- Sort indices by age. source is required in
1160 this case.
1161 · source -- Source of index age. Can be one of name, cre‐
1163 ation_date, or field_stats. Default: creation_date
1164 · timestring -- An strftime string to match the datestamp
1166 in an index name. Only used if source name is selected.
1167 · field -- A timestamp field name. Only used if source
1169 field_stats is selected.
1170 · stats_result -- Either min_value or max_value. Only
1172 used if source field_stats is selected. It determines
1173 whether to reference the minimum or maximum value of
1174 field in each index.
1175 · exclude -- If exclude is True, this filter will remove
1177 matching indices from indices. If exclude is False,
1178 then only matching indices will be kept in indices.
1179 Default is False
1180 IndexList.filter_closed(exclude=True)
1182 Filter out closed indices from indices
1183 Parameters
1185 exclude -- If exclude is True, this filter will remove
1186 matching indices from indices. If exclude is False, then
1187 only matching indices will be kept in indices. Default
1188 is True
1189 IndexList.filter_forceMerged(max_num_segments=None, exclude=True)
1191 Match any index which has max_num_segments per shard or fewer in
1192 the actionable list.
1193 Parameters
1195 · max_num_segments -- Cutoff number of segments per
1197 shard.
1198 · exclude -- If exclude is True, this filter will remove
1200 matching indices from indices. If exclude is False,
1201 then only matching indices will be kept in indices.
1202 Default is True
1203 IndexList.filter_kibana(exclude=True)
1205 Match any index named .kibana, kibana-int, .marvel-kibana, or
1206 .marvel-es-data in indices.
1207 Parameters
1209 exclude -- If exclude is True, this filter will remove
1210 matching indices from indices. If exclude is False, then
1211 only matching indices will be kept in indices. Default
1212 is True
1213 IndexList.filter_opened(exclude=True)
1215 Filter out opened indices from indices
1216 Parameters
1218 exclude -- If exclude is True, this filter will remove
1219 matching indices from indices. If exclude is False, then
1220 only matching indices will be kept in indices. Default
1221 is True
1222 IndexList.filter_none()
1224 IndexList.filter_by_alias(aliases=None, exclude=False)
1226 Match indices which are associated with the alias identified by
1227 name
1228 Parameters
1230 · aliases (list) -- A list of alias names.
1232 · exclude -- If exclude is True, this filter will remove
1234 matching indices from indices. If exclude is False,
1235 then only matching indices will be kept in indices.
1236 Default is False
1237 IndexList.filter_by_count(count=None, reverse=True, use_age=False,
1239 source='creation_date', timestring=None, field=None,
1240 stats_result='min_value', exclude=True)
1241 Remove indices from the actionable list beyond the number count,
1242 sorted reverse-alphabetically by default. If you set reverse to
1243 False, it will be sorted alphabetically.
1244 The default is usually what you will want. If only one kind of
1246 index is provided--for example, indices matching
1247 logstash-%Y.%m.%d--then reverse alphabetical sorting will mean
1248 the oldest will remain in the list, because lower numbers in the
1249 dates mean older indices.
1250 By setting reverse to False, then index3 will be deleted before
1252 index2, which will be deleted before index1
1253 use_age allows ordering indices by age. Age is determined by the
1255 index creation date by default, but you can specify an source of
1256 name, max_value, or min_value. The name source requires the
1257 timestring argument.
1258 Parameters
1260 · count -- Filter indices beyond count.
1262 · reverse -- The filtering direction. (default: True).
1264 · use_age -- Sort indices by age. source is required in
1266 this case.
1267 · source -- Source of index age. Can be one of name, cre‐
1269 ation_date, or field_stats. Default: creation_date
1270 · timestring -- An strftime string to match the datestamp
1272 in an index name. Only used if source name is selected.
1273 · field -- A timestamp field name. Only used if source
1275 field_stats is selected.
1276 · stats_result -- Either min_value or max_value. Only
1278 used if source field_stats is selected. It determines
1279 whether to reference the minimum or maximum value of
1280 field in each index.
1281 · exclude -- If exclude is True, this filter will remove
1283 matching indices from indices. If exclude is False,
1284 then only matching indices will be kept in indices.
1285 Default is True
1286 SnapshotList
1288 SnapshotList.filter_by_age(source='creation_date', direction=None,
1289 timestring=None, unit=None, unit_count=None, epoch=None, exclude=False)
1290 Remove snapshots from snapshots by relative age calculations.
1291 Parameters
1293 · source -- Source of snapshot age. Can be 'name', or
1295 'creation_date'.
1296 · direction -- Time to filter, either older or younger
1298 · timestring -- An strftime string to match the datestamp
1300 in an snapshot name. Only used for snapshot filtering
1301 by name.
1302 · unit -- One of seconds, minutes, hours, days, weeks,
1304 months, or years.
1305 · unit_count -- The number of unit``s. ``unit_count *
1307 unit will be calculated out to the relative number of
1308 seconds.
1309 · epoch -- An epoch timestamp used in conjunction with
1311 unit and unit_count to establish a point of reference
1312 for calculations. If not provided, the current time
1313 will be used.
1314 · exclude -- If exclude is True, this filter will remove
1316 matching snapshots from snapshots. If exclude is False,
1317 then only matching snapshots will be kept in snapshots.
1318 Default is False
1319 SnapshotList.filter_by_regex(kind=None, value=None, exclude=False)
1321 Filter out snapshots not matching the pattern, or in the case of
1322 exclude, filter those matching the pattern.
1323 Parameters
1325 · kind -- Can be one of: suffix, prefix, regex, or
1327 timestring. This option defines what kind of filter you
1328 will be building.
1329 · value -- Depends on kind. It is the strftime string if
1331 kind is timestring. It's used to build the regular
1332 expression for other kinds.
1333 · exclude -- If exclude is True, this filter will remove
1335 matching snapshots from snapshots. If exclude is False,
1336 then only matching snapshots will be kept in snapshots.
1337 Default is False
1338 SnapshotList.filter_by_state(state=None, exclude=False)
1340 Filter out snapshots not matching state, or in the case of
1341 exclude, filter those matching state.
1342 Parameters
1344 · state -- The snapshot state to filter for. Must be one
1346 of SUCCESS, PARTIAL, FAILED, or IN_PROGRESS.
1347 · exclude -- If exclude is True, this filter will remove
1349 matching snapshots from snapshots. If exclude is False,
1350 then only matching snapshots will be kept in snapshots.
1351 Default is False
1352 SnapshotList.filter_none()
1354 SnapshotList.filter_by_count(count=None, reverse=True, use_age=False,
1356 source='creation_date', timestring=None, exclude=True)
1357 Remove snapshots from the actionable list beyond the number
1358 count, sorted reverse-alphabetically by default. If you set
1359 reverse to False, it will be sorted alphabetically.
1360 The default is usually what you will want. If only one kind of
1362 snapshot is provided--for example, snapshots matching cura‐
1363 tor-%Y%m%d%H%M%S-- then reverse alphabetical sorting will mean
1364 the oldest will remain in the list, because lower numbers in the
1365 dates mean older snapshots.
1366 By setting reverse to False, then snapshot3 will be acted on
1368 before snapshot2, which will be acted on before snapshot1
1369 use_age allows ordering snapshots by age. Age is determined by
1371 the snapshot creation date (as identified by start_time_in_mil‐
1372 lis) by default, but you can also specify a source of name. The
1373 name source requires the timestring argument.
1374 Parameters
1376 · count -- Filter snapshots beyond count.
1378 · reverse -- The filtering direction. (default: True).
1380 · use_age -- Sort snapshots by age. source is required
1382 in this case.
1383 · source -- Source of snapshot age. Can be one of name,
1385 or creation_date. Default: creation_date
1386 · timestring -- An strftime string to match the datestamp
1388 in a snapshot name. Only used if source name is
1389 selected.
1390 · exclude -- If exclude is True, this filter will remove
1392 matching snapshots from snapshots. If exclude is False,
1393 then only matching snapshots will be kept in snapshots.
1394 Default is True
1395 Utility & Helper Methods
1397 class curator.utils.TimestringSearch(timestring)
1398 An object to allow repetitive search against a string, searchme,
1399 without having to repeatedly recreate the regex.
1400 Parameters
1402 timestring -- An strftime pattern
1403 get_epoch(searchme)
1405 Return the epoch timestamp extracted from the timestring
1406 appearing in searchme.
1407 Parameters
1409 searchme -- A string to be searched for a date
1410 pattern that matches timestring
1411 Return type
1413 int
1414 curator.utils.byte_size(num, suffix='B')
1416 Return a formatted string indicating the size in bytes, with the
1417 proper unit, e.g. KB, MB, GB, TB, etc.
1418 Parameters
1420 · num -- The number of byte
1422 · suffix -- An arbitrary suffix, like Bytes
1424 Return type
1426 float
1427 curator.utils.check_csv(value)
1429 Some of the curator methods should not operate against multiple
1430 indices at once. This method can be used to check if a list or
1431 csv has been sent.
1432 Parameters
1434 value -- The value to test, if list or csv string
1435 Return type
1437 bool
1438 curator.utils.check_master(client, master_only=False)
1440 Check if connected client is the elected master node of the
1441 cluster. If not, cleanly exit with a log message.
1442 Parameters
1444 client -- An elasticsearch.Elasticsearch client object
1445 Return type
1447 None
1448 curator.utils.check_version(client)
1450 Verify version is within acceptable range. Raise an exception
1451 if it is not.
1452 Parameters
1454 client -- An elasticsearch.Elasticsearch client object
1455 Return type
1457 None
1458 curator.utils.chunk_index_list(indices)
1460 This utility chunks very large index lists into 3KB chunks It
1461 measures the size as a csv string, then converts back into a
1462 list for the return value.
1463 Parameters
1465 indices -- A list of indices to act on.
1466 Return type
1468 list
1469 curator.utils.create_repo_body(repo_type=None, compress=True,
1471 chunk_size=None, max_restore_bytes_per_sec=None, max_snap‐
1472 shot_bytes_per_sec=None, location=None, bucket=None, region=None,
1473 base_path=None, access_key=None, secret_key=None, **kwargs)
1474 Build the 'body' portion for use in creating a repository.
1475 Parameters
1477 · repo_type -- The type of repository (presently only fs
1479 and s3)
1480 · compress -- Turn on compression of the snapshot files.
1482 Compression is applied only to metadata files (index
1483 mapping and settings). Data files are not compressed.
1484 (Default: True)
1485 · chunk_size -- The chunk size can be specified in bytes
1487 or by using size value notation, i.e. 1g, 10m, 5k.
1488 Defaults to null (unlimited chunk size).
1489 · max_restore_bytes_per_sec -- Throttles per node restore
1491 rate. Defaults to 20mb per second.
1492 · max_snapshot_bytes_per_sec -- Throttles per node snap‐
1494 shot rate. Defaults to 20mb per second.
1495 · location -- Location of the snapshots. Required.
1497 · bucket -- S3 only. The name of the bucket to be used
1499 for snapshots. Required.
1500 · region -- S3 only. The region where bucket is located.
1502 Defaults to US Standard
1503 · base_path -- S3 only. Specifies the path within bucket
1505 to repository data. Defaults to value of reposito‐
1506 ries.s3.base_path or to root directory if not set.
1507 · access_key -- S3 only. The access key to use for
1509 authentication. Defaults to value of
1510 cloud.aws.access_key.
1511 · secret_key -- S3 only. The secret key to use for
1513 authentication. Defaults to value of
1514 cloud.aws.secret_key.
1515 Returns
1517 A dictionary suitable for creating a repository from the
1518 provided arguments.
1519 Return type
1521 dict
1522 curator.utils.create_repository(client, **kwargs)
1524 Create repository with repository and body settings
1525 Parameters
1527 · client -- An elasticsearch.Elasticsearch client object
1529 · repo_type -- The type of repository (presently only fs
1531 and s3)
1532 · compress -- Turn on compression of the snapshot files.
1534 Compression is applied only to metadata files (index
1535 mapping and settings). Data files are not compressed.
1536 (Default: True)
1537 · chunk_size -- The chunk size can be specified in bytes
1539 or by using size value notation, i.e. 1g, 10m, 5k.
1540 Defaults to null (unlimited chunk size).
1541 · max_restore_bytes_per_sec -- Throttles per node restore
1543 rate. Defaults to 20mb per second.
1544 · max_snapshot_bytes_per_sec -- Throttles per node snap‐
1546 shot rate. Defaults to 20mb per second.
1547 · location -- Location of the snapshots. Required.
1549 · bucket -- S3 only. The name of the bucket to be used
1551 for snapshots. Required.
1552 · region -- S3 only. The region where bucket is located.
1554 Defaults to US Standard
1555 · base_path -- S3 only. Specifies the path within bucket
1557 to repository data. Defaults to value of reposito‐
1558 ries.s3.base_path or to root directory if not set.
1559 · access_key -- S3 only. The access key to use for
1561 authentication. Defaults to value of
1562 cloud.aws.access_key.
1563 · secret_key -- S3 only. The secret key to use for
1565 authentication. Defaults to value of
1566 cloud.aws.secret_key.
1567 Returns
1569 A boolean value indicating success or failure.
1570 Return type
1572 bool
1573 curator.utils.create_snapshot_body(indices, ignore_unavailable=False,
1575 include_global_state=True, partial=False)
1576 Create the request body for creating a snapshot from the pro‐
1577 vided arguments.
1578 Parameters
1580 · indices -- A single index, or list of indices to snap‐
1582 shot.
1583 · ignore_unavailable (bool) -- Ignore unavailable
1585 shards/indices. (default: False)
1586 · include_global_state (bool) -- Store cluster global
1588 state with snapshot. (default: True)
1589 · partial (bool) -- Do not fail if primary shard is
1591 unavailable. (default: False)
1592 Return type
1594 dict
1595 curator.utils.ensure_list(indices)
1597 Return a list, even if indices is a single value
1598 Parameters
1600 indices -- A list of indices to act upon
1601 Return type
1603 list
1604 curator.utils.fix_epoch(epoch)
1606 Fix value of epoch to be epoch, which should be 10 or fewer dig‐
1607 its long.
1608 Parameters
1610 epoch -- An epoch timestamp, in epoch + milliseconds, or
1611 microsecond, or even nanoseconds.
1612 Return type
1614 int
1615 curator.utils.get_client(**kwargs)
1617 NOTE: AWS IAM parameters aws_key, aws_secret_key, and aws_region
1618 are provided for future compatibility, should AWS ES support the
1619 /_cluster/state/metadata endpoint. So long as this endpoint
1620 does not function in AWS ES, the client will not be able to use
1621 curator.indexlist.IndexList, which is the backbone of Curator 4
1622 Return an elasticsearch.Elasticsearch client object using the
1624 provided parameters. Any of the keyword arguments the elastic‐
1625 search.Elasticsearch client object can receive are valid, such
1626 as:
1627 Parameters
1629 · hosts (list) -- A list of one or more Elasticsearch
1631 client hostnames or IP addresses to connect to. Can
1632 send a single host.
1633 · port (int) -- The Elasticsearch client port to connect
1635 to.
1636 · url_prefix (str) -- Optional url prefix, if needed to
1638 reach the Elasticsearch API (i.e., it's not at the root
1639 level)
1640 · use_ssl (bool) -- Whether to connect to the client via
1642 SSL/TLS
1643 · certificate -- Path to SSL/TLS certificate
1645 · client_cert -- Path to SSL/TLS client certificate (pub‐
1647 lic key)
1648 · client_key -- Path to SSL/TLS private key
1650 · aws_key -- AWS IAM Access Key (Only used if the
1652 requests-aws4auth python module is installed)
1653 · aws_secret_key -- AWS IAM Secret Access Key (Only used
1655 if the requests-aws4auth python module is installed)
1656 · aws_region -- AWS Region (Only used if the
1658 requests-aws4auth python module is installed)
1659 · ssl_no_validate (bool) -- If True, do not validate the
1661 certificate chain. This is an insecure option and you
1662 will see warnings in the log output.
1663 · http_auth (str) -- Authentication credentials in
1665 user:pass format.
1666 · timeout (int) -- Number of seconds before the client
1668 will timeout.
1669 · master_only (bool) -- If True, the client will only
1671 connect if the endpoint is the elected master node of
1672 the cluster. This option does not work if `hosts` has
1673 more than one value. It will raise an Exception in
1674 that case.
1675 Return type
1677 elasticsearch.Elasticsearch
1678 curator.utils.get_date_regex(timestring)
1680 Return a regex string based on a provided strftime timestring.
1681 Parameters
1683 timestring -- An strftime pattern
1684 Return type
1686 str
1687 curator.utils.get_datetime(index_timestamp, timestring)
1689 Return the datetime extracted from the index name, which is the
1690 index creation time.
1691 Parameters
1693 · index_timestamp -- The timestamp extracted from an
1695 index name
1696 · timestring -- An strftime pattern
1698 Return type
1700 datetime.datetime
1701 curator.utils.get_indices(client)
1703 Get the current list of indices from the cluster.
1704 Parameters
1706 client -- An elasticsearch.Elasticsearch client object
1707 Return type
1709 list
1710 curator.utils.get_point_of_reference(unit, count, epoch=None)
1712 Get a point-of-reference timestamp in epoch + milliseconds by
1713 deriving from a unit and a count, and an optional reference
1714 timestamp, epoch
1715 Parameters
1717 · unit -- One of seconds, minutes, hours, days, weeks,
1719 months, or years.
1720 · unit_count -- The number of units. unit_count * unit
1722 will be calculated out to the relative number of sec‐
1723 onds.
1724 · epoch -- An epoch timestamp used in conjunction with
1726 unit and unit_count to establish a point of reference
1727 for calculations.
1728 Return type
1730 int
1731 curator.utils.get_repository(client, repository='')
1733 Return configuration information for the indicated repository.
1734 Parameters
1736 · client -- An elasticsearch.Elasticsearch client object
1738 · repository -- The Elasticsearch snapshot repository to
1740 use
1741 Return type
1743 dict
1744 curator.utils.get_snapshot(client, repository=None, snapshot='')
1746 Return information about a snapshot (or a comma-separated list
1747 of snapshots) If no snapshot specified, it will return all snap‐
1748 shots. If none exist, an empty dictionary will be returned.
1749 Parameters
1751 · client -- An elasticsearch.Elasticsearch client object
1753 · repository -- The Elasticsearch snapshot repository to
1755 use
1756 · snapshot -- The snapshot name, or a comma-separated
1758 list of snapshots
1759 Return type
1761 dict
1762 curator.utils.get_snapshot_data(client, repository=None)
1764 Get _all snapshots from repository and return a list.
1765 Parameters
1767 · client -- An elasticsearch.Elasticsearch client object
1769 · repository -- The Elasticsearch snapshot repository to
1771 use
1772 Return type
1774 list
1775 curator.utils.get_version(client)
1777 Return the ES version number as a tuple. Omits trailing tags
1778 like -dev, or Beta
1779 Parameters
1781 client -- An elasticsearch.Elasticsearch client object
1782 Return type
1784 tuple
1785 curator.utils.get_yaml(path)
1787 Read the file identified by path and import its YAML contents.
1788 Parameters
1790 path -- The path to a YAML configuration file.
1791 Return type
1793 dict
1794 curator.utils.is_master_node(client)
1796 Return True if the connected client node is the elected master
1797 node in the Elasticsearch cluster, otherwise return False.
1798 Parameters
1800 client -- An elasticsearch.Elasticsearch client object
1801 Return type
1803 bool
1804 curator.utils.override_timeout(timeout, action)
1806 Override the default timeout for forcemerge, snapshot, and
1807 sync_flush operations if the default value of 30 is provided.
1808 Parameters
1810 · timeout -- Number of seconds before the client will
1812 timeout.
1813 · action -- The action to be performed.
1815 curator.utils.parse_date_pattern(name)
1817 Scan and parse name for time.strftime() strings, replacing them
1818 with the associated value when found, but otherwise returning
1819 lowercase values, as uppercase snapshot names are not allowed.
1820 The time.strftime() identifiers that Curator currently recog‐
1822 nizes as acceptable include:
1823 · Y: A 4 digit year
1825 · y: A 2 digit year
1827 · m: The 2 digit month
1829 · W: The 2 digit week of the year
1831 · d: The 2 digit day of the month
1833 · H: The 2 digit hour of the day, in 24 hour notation
1835 · M: The 2 digit minute of the hour
1837 · S: The 2 digit number of second of the minute
1839 · j: The 3 digit day of the year
1841 Parameters
1843 name -- A name, which can contain time.strftime() strings
1844 curator.utils.prune_nones(mydict)
1846 Remove keys from mydict whose values are None
1847 Parameters
1849 mydict -- The dictionary to act on
1850 Return type
1852 dict
1853 curator.utils.read_file(myfile)
1855 Read a file and return the resulting data.
1856 Parameters
1858 myfile -- A file to read.
1859 Return type
1861 str
1862 curator.utils.report_failure(exception)
1864 Raise a FailedExecution exception and include the original error
1865 message.
1866 Parameters
1868 exception -- The upstream exception.
1869 Return type
1871 None
1872 curator.utils.repository_exists(client, repository=None)
1874 Verify the existence of a repository
1875 Parameters
1877 · client -- An elasticsearch.Elasticsearch client object
1879 · repository -- The Elasticsearch snapshot repository to
1881 use
1882 Return type
1884 bool
1885 curator.utils.safe_to_snap(client, repository=None, retry_interval=120,
1887 retry_count=3)
1888 Ensure there are no snapshots in progress. Pause and retry
1889 accordingly
1890 Parameters
1892 · client -- An elasticsearch.Elasticsearch client object
1894 · repository -- The Elasticsearch snapshot repository to
1896 use
1897 · retry_interval -- Number of seconds to delay betwen
1899 retries. Default: 120 (seconds)
1900 · retry_count -- Number of attempts to make. Default: 3
1902 Return type
1904 bool
1905 curator.utils.show_dry_run(ilo, action, **kwargs)
1907 Log dry run output with the action which would have been exe‐
1908 cuted.
1909 Parameters
1911 · ilo -- A curator.indexlist.IndexList
1913 · action -- The action to be performed.
1915 · kwargs -- Any other args to show in the log output
1917 curator.utils.snapshot_in_progress(client, repository=None, snap‐
1919 shot=None)
1920 Determine whether the provided snapshot in repository is
1921 IN_PROGRESS. If no value is provided for snapshot, then check
1922 all of them. Return snapshot if it is found to be in progress,
1923 or False
1924 Parameters
1926 · client -- An elasticsearch.Elasticsearch client object
1928 · repository -- The Elasticsearch snapshot repository to
1930 use
1931 · snapshot -- The snapshot name
1933 curator.utils.snapshot_running(client)
1935 Return True if a snapshot is in progress, and False if not
1936 Parameters
1938 client -- An elasticsearch.Elasticsearch client object
1939 Return type
1941 bool
1942 curator.utils.test_client_options(config)
1944 Test whether a SSL/TLS files exist. Will raise an exception if
1945 the files cannot be read.
1946 Parameters
1948 config -- A client configuration file data dictionary
1949 Return type
1951 None
1952 curator.utils.test_repo_fs(client, repository=None)
1954 Test whether all nodes have write access to the repository
1955 Parameters
1957 · client -- An elasticsearch.Elasticsearch client object
1959 · repository -- The Elasticsearch snapshot repository to
1961 use
1962 curator.utils.to_csv(indices)
1964 Return a csv string from a list of indices, or a single value if
1965 only one value is present
1966 Parameters
1968 indices -- A list of indices to act on, or a single
1969 value, which could be in the format of a csv string
1970 already.
1971 Return type
1973 str
1974 curator.utils.validate_actions(data)
1976 Validate an Action configuration dictionary, as imported from
1977 actions.yml, for example.
1978 The method returns a validated and sanitized configuration dic‐
1980 tionary.
1981 Parameters
1983 data -- The configuration dictionary
1984 Return type
1986 dict
1987 curator.utils.validate_filters(action, filters)
1989 Validate that the filters are appropriate for the action type,
1990 e.g. no index filters applied to a snapshot list.
1991 Parameters
1993 · action -- An action name
1995 · filters -- A list of filters to test.
1997 curator.utils.verify_client_object(test)
1999 Test if test is a proper elasticsearch.Elasticsearch client
2000 object and raise an exception if it is not.
2001 Parameters
2003 test -- The variable or object to test
2004 Return type
2006 None
2007 curator.utils.verify_index_list(test)
2009 Test if test is a proper curator.indexlist.IndexList object and
2010 raise an exception if it is not.
2011 Parameters
2013 test -- The variable or object to test
2014 Return type
2016 None
2017 curator.utils.verify_snapshot_list(test)
2019 Test if test is a proper curator.snapshotlist.SnapshotList
2020 object and raise an exception if it is not.
2021 Parameters
2023 test -- The variable or object to test
2024 Return type
2026 None
2027 class curator.SchemaCheck(config, schema, test_what, location)
2029 Validate config with the provided voluptuous schema. test_what
2030 and location are for reporting the results, in case of failure.
2031 If validation is successful, the method returns config as valid.
2032 Parameters
2034 · config (dict) -- A configuration dictionary.
2036 · schema (voluptuous.Schema) -- A voluptuous schema defi‐
2038 nition
2039 · test_what (str) -- which configuration block is being
2041 validated
2042 · location (str) -- An string to report which configura‐
2044 tion sub-block is being tested.
2045 Examples
2047 Each of these examples presupposes that the requisite modules have been
2048 imported and an instance of the Elasticsearch client object has been
2049 created:
2050 import elasticsearch
2052 import curator
2053 client = elasticsearch.Elasticsearch()
2055 Filter indices by prefix
2057 ilo = curator.IndexList(client)
2058 ilo.filter_by_regex(kind='prefix', value='logstash-')
2059 The contents of ilo.indices would then only be indices matching the
2061 prefix.
2062 Filter indices by suffix
2064 ilo = curator.IndexList(client)
2065 ilo.filter_by_regex(kind='suffix', value='-prod')
2066 The contents of ilo.indices would then only be indices matching the
2068 suffix.
2069 Filter indices by age (name)
2071 This example will match indices with the following criteria:
2072 · Have a date string of %Y.%m.%d
2074 · Use days as the unit of time measurement
2076 · Filter indices older than 5 days
2078 ilo = curator.IndexList(client)
2080 ilo.filter_by_age(source='name', direction='older', timestring='%Y.%m.%d',
2081 unit='days', unit_count=5
2082 )
2083 The contents of ilo.indices would then only be indices matching these
2085 criteria.
2086 Filter indices by age (creation_date)
2088 This example will match indices with the following criteria:
2089 · Use months as the unit of time measurement
2091 · Filter indices where the index creation date is older than 2 months
2093 from this moment.
2094 ilo = curator.IndexList(client)
2096 ilo.filter_by_age(source='creation_date', direction='older',
2097 unit='months', unit_count=2
2098 )
2099 The contents of ilo.indices would then only be indices matching these
2101 criteria.
2102 Filter indices by age (field_stats)
2104 This example will match indices with the following criteria:
2105 · Use days as the unit of time measurement
2107 · Filter indices where the timestamp field's min_value is a date older
2109 than 3 weeks from this moment.
2110 ilo = curator.IndexList(client)
2112 ilo.filter_by_age(source='field_stats', direction='older',
2113 unit='weeks', unit_count=3, field='timestamp', stats_result='min_value'
2114 )
2115 The contents of ilo.indices would then only be indices matching these
2117 criteria.
2118 Changelog
2120 4.2.5 (22 December 2016)
2121 General
2122 · Add and increment test versions for Travis CI. #839 (untergeek)
2124 · Make filter_list optional in snapshot, show_snapshot and
2126 show_indices singleton actions. #853 (alexef)
2127 Bug Fixes
2129 · Fix cli integration test when different host/port are specified.
2131 Reported in #843 (untergeek)
2132 · Catch empty list condition during filter iteration in singleton
2134 actions. Reported in #848 (untergeek)
2135 Documentation
2137 · Add docs regarding how filters are ANDed together, and how to do
2139 an OR with the regex pattern filter type. Requested in #842
2140 (untergeek)
2141 · Fix typo in Click version in docs. #850 (breml)
2143 · Where applicable, replace [source,text] with [source,yaml] for
2145 better formatting in the resulting docs.
2146 4.2.4 (7 December 2016)
2148 Bug Fixes
2149 · --wait_for_completion should be True by default for Snapshot sin‐
2151 gleton action. Reported in #829 (untergeek)
2152 · Increase version_max to 5.1.99. Prematurely reported in #832
2154 (untergeek)
2155 · Make the '.security' index visible for snapshots so long as proper
2157 credentials are used. Reported in #826 (untergeek)
2158 4.2.3.post1 (22 November 2016)
2160 This fix is only going in for pip-based installs. There are no other
2161 code changes.
2162 Bug Fixes
2164 · Fixed incorrect assumption of PyPI picking up dependency for cer‐
2166 tifi. It is still a dependency, but should not affect pip
2167 installs with an error any more. Reported in #821 (untergeek)
2168 4.2.3 (21 November 2016)
2170 4.2.2 was pulled immediately after release after it was discovered that
2171 the Windows binary distributions were still not including the cer‐
2172 tifi-provided certificates. This has now been remedied.
2173 General
2175 · certifi is now officially a requirement.
2177 · setup.py now forcibly includes the certifi certificate PEM file in
2179 the "frozen" distributions (i.e., the compiled versions). The
2180 get_client method was updated to reflect this and catch it for
2181 both the Linux and Windows binary distributions. This should
2182 finally put to rest #810
2183 4.2.2 (21 November 2016)
2185 Bug Fixes
2186 · The certifi-provided certificates were not propagating to the com‐
2188 piled RPM/DEB packages. This has been corrected. Reported in
2189 #810 (untergeek)
2190 General
2192 · Added missing --ignore_empty_list option to singleton actions.
2194 Requested in #812 (untergeek)
2195 Documentation
2197 · Add a FAQ entry regarding the click module's need for Unicode when
2199 using Python 3. Kind of a bug fix too, as the entry_points were
2200 altered to catch this omission and report a potential solution on
2201 the command-line. Reported in #814 (untergeek)
2202 · Change the "Command-Line" documentation header to be "Running
2204 Curator"
2205 4.2.1 (8 November 2016)
2207 Bug Fixes
2208 · In the course of package release testing, an undesirable scenario
2210 was caught where boolean flags default values for curator_cli were
2211 improperly overriding values from a yaml config file.
2212 General
2214 · Adding in direct download URLs for the RPM, DEB, tarball and zip
2216 packages.
2217 4.2.0 (4 November 2016)
2219 New Features
2220 · Shard routing allocation enable/disable. This will allow you to
2222 disable shard allocation routing before performing one or more
2223 actions, and then re-enable after it is complete. Requested in
2224 #446 (untergeek)
2225 · Curator 3.x-style command-line. This is now curator_cli, to dif‐
2227 ferentiate between the current binary. Not all actions are avail‐
2228 able, but the most commonly used ones are. With the addition in
2229 4.1.0 of schema and configuration validation, there's even a way
2230 to still do filter chaining on the command-line! Requested in
2231 #767, and by many other users (untergeek)
2232 General
2234 · Update testing to the most recent versions.
2236 · Lock elasticsearch-py module version at >= 2.4.0 and <= 3.0.0.
2238 There are API changes in the 5.0 release that cause tests to fail.
2239 Bug Fixes
2241 · Guarantee that binary packages are built from the latest Python +
2243 libraries. This ensures that SSL/TLS will work without warning
2244 messages about insecure connections, unless they actually are
2245 insecure. Reported in #780, though the reported problem isn't what
2246 was fixed. The fix is needed based on what was discovered while
2247 troubleshooting the problem. (untergeek)
2248 4.1.2 (6 October 2016)
2250 This release does not actually add any new code to Curator, but instead
2251 improves documentation and includes new linux binary packages.
2252 General
2254 · New Curator binary packages for common Linux systems! These will
2256 be found in the same repositories that the python-based packages
2257 are in, but have no dependencies. All necessary libraries/modules
2258 are bundled with the binary, so everything should work out of the
2259 box. This feature doesn't change any other behavior, so it's not
2260 a major release.
2261 These binaries have been tested in:
2263 · CentOS 6 & 7
2265 · Ubuntu 12.04, 14.04, 16.04
2267 · Debian 8
2269 They do not work in Debian 7 (library mismatch). They may work in
2271 other systems, but that is untested.
2272 The script used is in the unix_packages directory. The Vagrant‐
2274 files for the various build systems are in the Vagrant directory.
2275 Bug Fixes
2277 · The only bug that can be called a bug is actually a stray .exe
2279 suffix in the binary package creation section (cx_freeze) of set‐
2280 up.py. The Windows binaries should have .exe extensions, but not
2281 unix variants.
2282 · Elasticsearch 5.0.0-beta1 testing revealed that a document ID is
2284 required during document creation in tests. This has been fixed,
2285 and a redundant bit of code in the forcemerge integration test was
2286 removed.
2287 Documentation
2289 · The documentation has been updated and improved. Examples and
2291 installation are now top-level events, with the sub-sections each
2292 having their own link. They also now show how to install and use
2293 the binary packages, and the section on installation from source
2294 has been improved. The missing section on installing the volup‐
2295 tuous schema verification module has been written and included.
2296 #776 (untergeek)
2297 4.1.1 (27 September 2016)
2299 Bug Fixes
2300 · String-based booleans are now properly coerced. This fixes an
2302 issue where True/False were used in environment variables, but not
2303 recognized. #765 (untergeek)
2304 · Fix missing count method in __map_method in SnapshotList. Reported
2306 in #766 (untergeek)
2307 General
2309 · Update es_repo_mgr to use the same client/logging YAML config
2311 file. Requested in #752 (untergeek)
2312 Schema Validation
2314 · Cases where source was not defined in a filter (but should have
2316 been) were informing users that a timestring field was there that
2317 shouldn't have been. This edge case has been corrected.
2318 Documentation
2320 · Added notifications and FAQ entry to explain that AWS ES is not
2322 supported.
2323 4.1.0 (6 September 2016)
2325 New Features
2326 · Configuration and Action file schema validation. Requested in
2328 #674 (untergeek)
2329 · Alias filtertype! With this filter, you can select indices based
2331 on whether they are part of an alias. Merged in #748 (untergeek)
2332 · Count filtertype! With this filter, you can now configure Curator
2334 to only keep the most recent _n_ indices (or snapshots!). Merged
2335 in #749 (untergeek)
2336 · Experimental! Use environment variables in your YAML configuration
2338 files. This was a popular request, #697. (untergeek)
2339 General
2341 · New requirement! voluptuous Python schema validation module
2343 · Requirement version bump: Now requires elasticsearch-py 2.4.0
2345 Bug Fixes
2347 · delete_aliases option in close action no longer results in an
2349 error if not all selected indices have an alias. Add test to con‐
2350 firm expected behavior. Reported in #736 (untergeek)
2351 Documentation
2353 · Add information to FAQ regarding indices created before Elastic‐
2355 search 1.4. Merged in #747
2356 4.0.6 (15 August 2016)
2358 Bug Fixes
2359 · Update old calls used with ES 1.x to reflect changes in 2.x+. This
2361 was necessary to work with Elasticsearch 5.0.0-alpha5. Fixed in
2362 #728 (untergeek)
2363 Doc Fixes
2365 · Add section detailing that the value of a value filter element
2367 should be encapsulated in single quotes. Reported in #726.
2368 (untergeek)
2369 4.0.5 (3 August 2016)
2371 Bug Fixes
2372 · Fix incorrect variable name for AWS Region reported in #679
2374 (basex)
2375 · Fix filter_by_space() to not fail when index age metadata is not
2377 present. Indices without the appropriate age metadata will
2378 instead be excluded, with a debug-level message. Reported in #724
2379 (untergeek)
2380 Doc Fixes
2382 · Fix documentation for the space filter and the source filter ele‐
2384 ment.
2385 4.0.4 (1 August 2016)
2387 Bug Fixes
2388 · Fix incorrect variable name in Allocation action. #706 (lukewaite)
2390 · Incorrect error message in create_snapshot_body reported in #711
2392 (untergeek)
2393 · Test for empty index list object should happen in action initial‐
2395 ization for snapshot action. Discovered in #711. (untergeek)
2396 Doc Fixes
2398 · Add menus to asciidoc chapters #704 (untergeek)
2400 · Add pyyaml dependency #710 (dtrv)
2402 4.0.3 (22 July 2016)
2404 General
2405 · 4.0.2 didn't work for pip installs due to an omission in the MANI‐
2407 FEST.in file. This came up during release testing, but before the
2408 release was fully published. As the release was never fully pub‐
2409 lished, this should not have actually affected anyone.
2410 Bug Fixes
2412 · These are the same as 4.0.2, but it was never fully released.
2414 · All default settings are now values returned from functions
2416 instead of constants. This was resulting in settings getting
2417 stomped on. New test addresses the original complaint. This
2418 removes the need for deepcopy. See issue #687 (untergeek)
2419 · Fix host vs. hosts issue in get_client() rather than the non-func‐
2421 tional function in repomgrcli.py.
2422 · Update versions being tested.
2424 · Community contributed doc fixes.
2426 · Reduced logging verbosity by making most messages debug level.
2428 #684 (untergeek)
2429 · Fixed log whitelist behavior (and switched to blacklisting
2431 instead). Default behavior will now filter traffic from the elas‐
2432 ticsearch and urllib3 modules.
2433 · Fix Travis CI testing to accept some skipped tests, as needed.
2435 #695 (untergeek)
2436 · Fix missing empty index test in snapshot action. #682 (sherzberg)
2438 4.0.2 (22 July 2016)
2440 Bug Fixes
2441 · All default settings are now values returned from functions
2443 instead of constants. This was resulting in settings getting
2444 stomped on. New test addresses the original complaint. This
2445 removes the need for deepcopy. See issue #687 (untergeek)
2446 · Fix host vs. hosts issue in get_client() rather than the non-func‐
2448 tional function in repomgrcli.py.
2449 · Update versions being tested.
2451 · Community contributed doc fixes.
2453 · Reduced logging verbosity by making most messages debug level.
2455 #684 (untergeek)
2456 · Fixed log whitelist behavior (and switched to blacklisting
2458 instead). Default behavior will now filter traffic from the elas‐
2459 ticsearch and urllib3 modules.
2460 · Fix Travis CI testing to accept some skipped tests, as needed.
2462 #695 (untergeek)
2463 · Fix missing empty index test in snapshot action. #682 (sherzberg)
2465 4.0.1 (1 July 2016)
2467 Bug Fixes
2468 · Coerce Logstash/JSON logformat type timestamp value to always use
2470 UTC. #661 (untergeek)
2471 · Catch and remove indices from the actionable list if they do not
2473 have a creation_date field in settings. This field was introduced
2474 in ES v1.4, so that indicates a rather old index. #663 (untergeek)
2475 · Replace missing state filter for snapshotlist. #665 (untergeek)
2477 · Restore es_repo_mgr as a stopgap until other CLI scripts are
2479 added. It will remain undocumented for now, as I am debating
2480 whether to make repository creation its own action in the API.
2481 #668 (untergeek)
2482 · Fix dry run results for snapshot action. #673 (untergeek)
2484 4.0.0 (24 June 2016)
2486 It's official! Curator 4.0.0 is released!
2487 Breaking Changes
2489 · New and improved API!
2491 · Command-line changes. No more command-line args, except for
2493 --config, --actions, and --dry-run:
2494 · --config points to a YAML client and logging configuration
2496 file. The default location is ~/.curator/curator.yml
2497 · --actions arg points to a YAML action configuration file
2499 · --dry-run will simulate the action(s) which would have taken
2501 place, but not actually make any changes to the cluster or
2502 its indices.
2503 New Features
2505 · Snapshot restore is here!
2507 · YAML configuration files. Now a single file can define an entire
2509 batch of commands, each with their own filters, to be performed in
2510 sequence.
2511 · Sort by index age not only by index name (as with previous ver‐
2513 sions of Curator), but also by index creation_date, or by calcula‐
2514 tions from the Field Stats API on a timestamp field.
2515 · Atomically add/remove indices from aliases! This is possible by
2517 way of the new IndexList class and YAML configuration files.
2518 · State of indices pulled and stored in IndexList instance. Fewer
2520 API calls required to serially test for open/close, size_in_bytes,
2521 etc.
2522 · Filter by space now allows sorting by age!
2524 · Experimental! Use AWS IAM credentials to sign requests to Elastic‐
2526 search. This requires the end user to manually install the
2527 requests_aws4auth python module.
2528 · Optionally delete aliases from indices before closing.
2530 · An empty index or snapshot list no longer results in an error if
2532 you set ignore_empty_list to True. If True it will still log that
2533 the action was not performed, but will continue to the next
2534 action. If 'False' it will log an ERROR and exit with code 1.
2535 API
2537 · Updated API documentation
2539 · Class: IndexList. This pulls all indices at instantiation, and you
2541 apply filters, which are class methods. You can iterate over as
2542 many filters as you like, in fact, due to the YAML config file.
2543 · Class: SnapshotList. This pulls all snapshots from the given
2545 repository at instantiation, and you apply filters, which are
2546 class methods. You can iterate over as many filters as you like,
2547 in fact, due to the YAML config file.
2548 · Add wait_for_completion to Allocation and Replicas actions. These
2550 will use the client timeout, as set by default or timeout_over‐
2551 ride, to determine how long to wait for timeout. These are han‐
2552 dled in batches of indices for now.
2553 · Allow timeout_override option for all actions. This allows for
2555 different timeout values per action.
2556 · Improve API by giving each action its own do_dry_run() method.
2558 General
2560 · Updated use documentation for Elastic main site.
2562 · Include example files for --config and --actions.
2564 4.0.0b2 (16 June 2016)
2566 Second beta release of the 4.0 branch
2567 New Feature
2569 · An empty index or snapshot list no longer results in an error if
2571 you set ignore_empty_list to True. If True it will still log that
2572 the action was not performed, but will continue to the next
2573 action. If 'False' it will log an ERROR and exit with code 1.
2574 (untergeek)
2575 4.0.0b1 (13 June 2016)
2577 First beta release of the 4.0 branch!
2578 The release notes will be rehashing the new features in 4.0, rather
2580 than the bug fixes done during the alphas.
2581 Breaking Changes
2583 · New and improved API!
2585 · Command-line changes. No more command-line args, except for
2587 --config, --actions, and --dry-run:
2588 · --config points to a YAML client and logging configuration
2590 file. The default location is ~/.curator/curator.yml
2591 · --actions arg points to a YAML action configuration file
2593 · --dry-run will simulate the action(s) which would have taken
2595 place, but not actually make any changes to the cluster or
2596 its indices.
2597 New Features
2599 · Snapshot restore is here!
2601 · YAML configuration files. Now a single file can define an entire
2603 batch of commands, each with their own filters, to be performed in
2604 sequence.
2605 · Sort by index age not only by index name (as with previous ver‐
2607 sions of Curator), but also by index creation_date, or by calcula‐
2608 tions from the Field Stats API on a timestamp field.
2609 · Atomically add/remove indices from aliases! This is possible by
2611 way of the new IndexList class and YAML configuration files.
2612 · State of indices pulled and stored in IndexList instance. Fewer
2614 API calls required to serially test for open/close, size_in_bytes,
2615 etc.
2616 · Filter by space now allows sorting by age!
2618 · Experimental! Use AWS IAM credentials to sign requests to Elastic‐
2620 search. This requires the end user to manually install the
2621 requests_aws4auth python module.
2622 · Optionally delete aliases from indices before closing.
2624 API
2626 · Updated API documentation
2628 · Class: IndexList. This pulls all indices at instantiation, and you
2630 apply filters, which are class methods. You can iterate over as
2631 many filters as you like, in fact, due to the YAML config file.
2632 · Class: SnapshotList. This pulls all snapshots from the given
2634 repository at instantiation, and you apply filters, which are
2635 class methods. You can iterate over as many filters as you like,
2636 in fact, due to the YAML config file.
2637 · Add wait_for_completion to Allocation and Replicas actions. These
2639 will use the client timeout, as set by default or timeout_over‐
2640 ride, to determine how long to wait for timeout. These are han‐
2641 dled in batches of indices for now.
2642 · Allow timeout_override option for all actions. This allows for
2644 different timeout values per action.
2645 · Improve API by giving each action its own do_dry_run() method.
2647 General
2649 · Updated use documentation for Elastic main site.
2651 · Include example files for --config and --actions.
2653 4.0.0a10 (10 June 2016)
2655 New Features
2656 · Snapshot restore is here!
2658 · Optionally delete aliases from indices before closing. Fixes #644
2660 (untergeek)
2661 General
2663 · Add wait_for_completion to Allocation and Replicas actions. These
2665 will use the client timeout, as set by default or timeout_over‐
2666 ride, to determine how long to wait for timeout. These are han‐
2667 dled in batches of indices for now.
2668 · Allow timeout_override option for all actions. This allows for
2670 different timeout values per action.
2671 Bug Fixes
2673 · Disallow use of master_only if multiple hosts are used. Fixes #615
2675 (untergeek)
2676 · Fix an issue where arguments weren't being properly passed and
2678 populated.
2679 · ForceMerge replaced Optimize in ES 2.1.0.
2681 · Fix prune_nones to work with Python 2.6. Fixes #619 (untergeek)
2683 · Fix TimestringSearch to work with Python 2.6. Fixes #622
2685 (untergeek)
2686 · Add language classifiers to setup.py. Fixes #640 (untergeek)
2688 · Changed references to readthedocs.org to be readthedocs.io.
2690 4.0.0a9 (27 Apr 2016)
2692 General
2693 · Changed create_index API to use kwarg extra_settings instead of
2695 body
2696 · Normalized Alias action to use name instead of alias. This sim‐
2698 plifies documentation by reducing the number of option elements.
2699 · Streamlined some code
2701 · Made exclude a filter element setting for all filters. Updated all
2703 examples to show this.
2704 · Improved documentation
2706 New Features
2708 · Alias action can now accept extra_settings to allow adding fil‐
2710 ters, and/or routing.
2711 4.0.0a8 (26 Apr 2016)
2713 Bug Fixes
2714 · Fix to use optimize with versions of Elasticsearch < 5.0
2716 · Fix missing setting in testvars
2718 4.0.0a7 (25 Apr 2016)
2720 Bug Fixes
2721 · Fix AWS4Auth error.
2723 4.0.0a6 (25 Apr 2016)
2725 General
2726 · Documentation updates.
2728 · Improve API by giving each action its own do_dry_run() method.
2730 Bug Fixes
2732 · Do not escape characters other than . and - in timestrings. Fixes
2734 #602 (untergeek)
2735 ** New Features**
2737 · Added CreateIndex action.
2739 4.0.0a4 (21 Apr 2016)
2741 Bug Fixes
2742 · Require pyyaml 3.10 or better.
2744 · In the case that no options are in an action, apply the defaults.
2746 4.0.0a3 (21 Apr 2016)
2748 It's time for Curator 4.0 alpha!
2749 Breaking Changes
2751 · New API! (again?!)
2753 · Command-line changes. No more command-line args, except for
2755 --config, --actions, and --dry-run:
2756 · --config points to a YAML client and logging configuration
2758 file. The default location is ~/.curator/curator.yml
2759 · --actions arg points to a YAML action configuration file
2761 · --dry-run will simulate the action(s) which would have taken
2763 place, but not actually make any changes to the cluster or
2764 its indices.
2765 General
2767 · Updated API documentation
2769 · Updated use documentation for Elastic main site.
2771 · Include example files for --config and --actions.
2773 New Features
2775 · Sort by index age not only by index name (as with previous ver‐
2777 sions of Curator), but also by index creation_date, or by calcula‐
2778 tions from the Field Stats API on a timestamp field.
2779 · Class: IndexList. This pulls all indices at instantiation, and you
2781 apply filters, which are class methods. You can iterate over as
2782 many filters as you like, in fact, due to the YAML config file.
2783 · Class: SnapshotList. This pulls all snapshots from the given
2785 repository at instantiation, and you apply filters, which are
2786 class methods. You can iterate over as many filters as you like,
2787 in fact, due to the YAML config file.
2788 · YAML configuration files. Now a single file can define an entire
2790 batch of commands, each with their own filters, to be performed in
2791 sequence.
2792 · Atomically add/remove indices from aliases! This is possible by
2794 way of the new IndexList class and YAML configuration files.
2795 · State of indices pulled and stored in IndexList instance. Fewer
2797 API calls required to serially test for open/close, size_in_bytes,
2798 etc.
2799 · Filter by space now allows sorting by age!
2801 · Experimental! Use AWS IAM credentials to sign requests to Elastic‐
2803 search. This requires the end user to manually install the
2804 requests_aws4auth python module.
2805 3.5.1 (21 March 2016)
2807 Bug fixes
2808 · Add more logging information to snapshot delete method #582
2810 (untergeek)
2811 · Improve default timeout, logging, and exception handling for seal
2813 command #583 (untergeek)
2814 · Fix use of default snapshot name. #584 (untergeek)
2816 3.5.0 (16 March 2016)
2818 General
2819 · Add support for the --client-cert and --client-key command line
2821 parameters and client_cert and client_key parameters to the
2822 get_client() call. #520 (richm)
2823 Bug fixes
2825 · Disallow users from creating snapshots with upper-case letters,
2827 which is not permitted by Elasticsearch. #562 (untergeek)
2828 · Remove print() command from setup.py as it causes issues with com‐
2830 mand- line retrieval of --url, etc. #568 (thib-ack)
2831 · Remove unnecessary argument from build_filter() #530 (zzugg)
2833 · Allow day of year filter to be made up with 1, 2 or 3 digits #578
2835 (petitout)
2836 3.4.1 (10 February 2016)
2838 General
2839 · Update license copyright to 2016
2841 · Use slim python version with Docker #527 (xaka)
2843 · Changed --master-only exit code to 0 when connected to non-master
2845 node #540 (wkruse)
2846 · Add cx_Freeze capability to setup.py, plus a binary_release.py
2848 script to simplify binary package creation. #554 (untergeek)
2849 · Set Elastic as author. #555 (untergeek)
2851 · Put repository creation methods into API and document them.
2853 Requested in #550 (untergeek)
2854 Bug fixes
2856 · Fix sphinx documentation build error #506 (hydrapolic)
2858 · Ensure snapshots are found before iterating #507 (garyelephant)
2860 · Fix a doc inconsistency #509 (pmoust)
2862 · Fix a typo in show documentation #513 (pbamba)
2864 · Default to trying the cluster state for checking whether indices
2866 are closed, and then fall back to using the _cat API (for Amazon
2867 ES instances). #519 (untergeek)
2868 · Improve logging to show time delay between optimize runs, if
2870 selected. #525 (untergeek)
2871 · Allow elasticsearch-py module versions through 2.3.0 (a presump‐
2873 tion at this point) #524 (untergeek)
2874 · Improve logging in snapshot api method to reveal when a repository
2876 appears to be missing. Reported in #551 (untergeek)
2877 · Test that --timestring has the correct variable for --time-unit.
2879 Reported in #544 (untergeek)
2880 · Allocation will exit with exit_code 0 now when there are no
2882 indices to work on. Reported in #531 (untergeek)
2883 3.4.0 (28 October 2015)
2885 General
2886 · API change in elasticsearch-py 1.7.0 prevented alias operations.
2888 Fixed in #486 (HonzaKral)
2889 · During index selection you can now select only closed indices with
2891 --closed-only. Does not impact --all-indices Reported in #476.
2892 Fixed in #487 (Basster)
2893 · API Changes in Elasticsearch 2.0.0 required some refactoring. All
2895 tests pass for ES versions 1.0.3 through 2.0.0-rc1. Fixed in #488
2896 (untergeek)
2897 · es_repo_mgr now has access to the same SSL options from #462. #489
2899 (untergeek)
2900 · Logging improvements requested in #475. (untergeek)
2902 · Added --quiet flag. #494 (untergeek)
2904 · Fixed index_closed to work with AWS Elasticsearch. #499 (univerio)
2906 · Acceptable versions of Elasticsearch-py module are 1.8.0 up to
2908 2.1.0 (untergeek)
2909 3.3.0 (31 August 2015)
2911 Announcement
2912 · Curator is tested in Jenkins. Each commit to the master branch is
2914 tested with both Python versions 2.7.6 and 3.4.0 against each of
2915 the following Elasticsearch versions: * 1.7_nightly * 1.6_nightly
2916 * 1.7.0 * 1.6.1 * 1.5.1 * 1.4.4 * 1.3.9 * 1.2.4 * 1.1.2 * 1.0.3
2917 · If you are using a version different from this, your results may
2919 vary.
2920 General
2922 · Allocation type can now also be include or exclude, in addition to
2924 the the existing default require type. Add --type to the alloca‐
2925 tion command to specify the type. #443 (steffo)
2926 · Bump elasticsearch python module dependency to 1.6.0+ to enable
2928 synced_flush API call. Reported in #447 (untergeek)
2929 · Add SSL features, --ssl-no-validate and certificate to provide
2931 other ways to validate SSL connections to Elasticsearch. #436
2932 (untergeek)
2933 Bug fixes
2935 · Delete by space was only reporting space used by primary shards.
2937 Fixed to show all space consumed. Reported in #455 (untergeek)
2938 · Update exit codes and messages for snapshot selection. Reported
2940 in #452 (untergeek)
2941 · Fix potential int/float casting issues. Reported in #465
2943 (untergeek)
2944 3.2.3 (16 July 2015)
2946 Bug fix
2947 · In order to address customer and community issues with bulk
2949 deletes, the master_timeout is now invoked for delete operations.
2950 This should address 503s with 30s timeouts in the debug log, even
2951 when --timeout is set to a much higher value. The master_timeout
2952 is tied to the --timeout flag value, but will not exceed 300 sec‐
2953 onds. #420 (untergeek)
2954 General
2956 · Mixing it up a bit here by putting General second! The only other
2958 changes are that logging has been improved for deletes so you
2959 won't need to have the --debug flag to see if you have error codes
2960 >= 400, and some code documentation improvements.
2961 3.2.2 (13 July 2015)
2963 General
2964 · This is a very minor change. The mock library recently removed
2966 support for Python 2.6. As many Curator users are using RHEL/Cen‐
2967 tOS 6, which is pinned to Python 2.6, this requires the mock ver‐
2968 sion referenced by Curator to also be pinned to a supported ver‐
2969 sion (mock==1.0.1).
2970 3.2.1 (10 July 2015)
2972 General
2973 · Added delete verification & retry (fixed at 3x) to potentially
2975 cover an edge case in #420 (untergeek)
2976 · Since GitHub allows rST (reStructuredText) README documents, and
2978 that's what PyPI wants also, the README has been rebuilt in rST.
2979 (untergeek)
2980 Bug fixes
2982 · If closing indices with ES 1.6+, and all indices are closed,
2984 ensure that the seal command does not try to seal all indices.
2985 Reported in #426 (untergeek)
2986 · Capture AttributeError when sealing indices if a non-TransportEr‐
2988 ror occurs. Reported in #429 (untergeek)
2989 3.2.0 (25 June 2015)
2991 New!
2992 · Added support to manually seal, or perform a [synced flush](‐
2994 http://www.elastic.co/guide/en/elasticsearch/reference/cur‐
2995 rent/indices-synced-flush.html) on indices with the seal command.
2996 #394 (untergeek)
2997 · Added experimental support for SSL certificate validation. In
2999 order for this to work, you must install the certifi python mod‐
3000 ule: pip install certifi This feature should automatically work if
3001 the certifi module is installed. Please report any issues.
3002 General
3004 · Changed logging to go to stdout rather than stderr. Reopened #121
3006 and figured they were right. This is better. (untergeek)
3007 · Exit code 99 was unpopular. It has been removed. Reported in #371
3009 and #391 (untergeek)
3010 · Add --skip-repo-validation flag for snapshots. Do not validate
3012 write access to repository on all cluster nodes before proceeding.
3013 Useful for shared filesystems where intermittent timeouts can
3014 affect validation, but won't likely affect snapshot success.
3015 Requested in #396 (untergeek)
3016 · An alias no longer needs to be pre-existent in order to use the
3018 alias command. #317 (untergeek)
3019 · es_repo_mgr now passes through upstream errors in the event a
3021 repository fails to be created. Requested in #405 (untergeek)
3022 Bug fixes
3024 · In rare cases, * wildcard would not expand. Replaced with _all.
3026 Reported in #399 (untergeek)
3027 · Beginning with Elasticsearch 1.6, closed indices cannot have their
3029 replica count altered. Attempting to do so results in this error:
3030 org.elasticsearch.ElasticsearchIllegalArgumentException: Can't
3031 update [index.number_of_replicas] on closed indices [[test_index]]
3032 - can leave index in an unopenable state As a result, the
3033 change_replicas method has been updated to prune closed indices.
3034 This change will apply to all versions of Elasticsearch. Reported
3035 in #400 (untergeek)
3036 · Fixed es_repo_mgr repository creation verification error. Reported
3038 in #389 (untergeek)
3039 3.1.0 (21 May 2015)
3041 General
3042 · If wait_for_completion is true, snapshot success is now tested and
3044 logged. Reported in #253 (untergeek)
3045 · Log & return false if a snapshot is already in progress
3047 (untergeek)
3048 · Logs individual deletes per index, even though they happen in
3050 batch mode. Also log individual snapshot deletions. Reported in
3051 #372 (untergeek)
3052 · Moved chunk_index_list from cli to api utils as it's now also used
3054 by filter.py
3055 · Added a warning and 10 second timer countdown if you use
3057 --timestring to filter indices, but do not use --older-than or
3058 --newer-than in conjunction with it. This is to address #348,
3059 which behavior isn't a bug, but prevents accidental action against
3060 all of your time-series indices. The warning and timer are not
3061 displayed for show and --dry-run operations.
3062 · Added tests for es_repo_mgr in #350
3064 · Doc fixes
3066 Bug fixes
3068 · delete-by-space needed the same fix used for #245. Fixed in #353
3070 (untergeek)
3071 · Increase default client timeout for es_repo_mgr as node discovery
3073 and availability checks for S3 repositories can take a bit. Fixed
3074 in #352 (untergeek)
3075 · If an index is closed, indicate in show and --dry-run output.
3077 Reported in #327. (untergeek)
3078 · Fix issue where CLI parameters were not being passed to the
3080 es_repo_mgr create sub-command. Reported in #337. (feltnerm)
3081 3.0.3 (27 Mar 2015)
3083 Announcement
3084 This is a bug fix release. #319 and #320 are affecting a few users, so
3086 this release is being expedited.
3087 Test count: 228 Code coverage: 99%
3089 General
3091 · Documentation for the CLI converted to Asciidoc and moved to
3093 http://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html
3094 · Improved logging, and refactored a few methods to help with this.
3096 · Dry-run output is now more like v2, with the index or snapshot in
3098 the log line, along with the command. Several tests needed refac‐
3099 toring with this change, along with a bit of documentation.
3100 Bug fixes
3102 · Fix links to repository in setup.py. Reported in #318 (untergeek)
3104 · No more --delay with optimized indices. Reported in #319
3106 (untergeek)
3107 · --request_timeout not working as expected. Reinstate the version
3109 2 timeout override feature to prevent default timeouts for opti‐
3110 mize and snapshot operations. Reported in #320 (untergeek)
3111 · Reduce index count to 200 for test.integration.test_cli_com‐
3113 mands.TestCLISnapshot.test_cli_snapshot_huge_list in order to
3114 reduce or eliminate Jenkins CI test timeouts. Reported in #324
3115 (untergeek)
3116 · --dry-run no longer calls show, but will show output in the log,
3118 as in v2. This was a recurring complaint. See #328 (untergeek)
3119 3.0.2 (23 Mar 2015)
3121 Announcement
3122 This is a bug fix release. #307 and #309 were big enough to warrant an
3124 expedited release.
3125 Bug fixes
3127 · Purge unneeded constants, and clean up config options for snap‐
3129 shot. Reported in #303 (untergeek)
3130 · Don't split large index list if performing snapshots. Reported in
3132 #307 (untergeek)
3133 · Act correctly if a zero value for --older-than or --newer-than is
3135 provided. #309 (untergeek)
3136 3.0.1 (16 Mar 2015)
3138 Announcement
3139 The regex_iterate method was horribly named. It has been renamed to
3141 apply_filter. Methods have been added to allow API users to build a
3142 filtered list of indices similarly to how the CLI does. This was an
3143 oversight. Props to @SegFaultAX for pointing this out.
3144 General
3146 · In conjunction with the rebrand to Elastic, URLs and documentation
3148 were updated.
3149 · Renamed horribly named regex_iterate method to apply_filter #298
3151 (untergeek)
3152 · Added build_filter method to mimic CLI calls. #298 (untergeek)
3154 · Added Examples page in the API documentation. #298 (untergeek)
3156 Bug fixes
3158 · Refactored to show --dry-run info for --disk-space calls. Reported
3160 in #290 (untergeek)
3161 · Added list chunking so acting on huge lists of indices won't
3163 result in a URL bigger than 4096 bytes (Elasticsearch's default
3164 limit.) Reported in
3165 https://github.com/elastic/curator/issues/245#issuecomment-77916081
3166 · Refactored to_csv() method to be simpler.
3168 · Added and removed tests according to changes. Code coverage still
3170 at 99%
3171 3.0.0 (9 March 2015)
3173 Release Notes
3174 The full release of Curator 3.0 is out! Check out all of the changes
3176 here!
3177 Note: This release is _not_ reverse compatible with any previous ver‐
3179 sion.
3180 Because 3.0 is a major point release, there have been some major
3182 changes to both the API as well as the CLI arguments and structure.
3183 Be sure to read the updated command-line specific docs in the [wiki](‐
3185 https://github.com/elasticsearch/curator/wiki) and change your com‐
3186 mand-line arguments accordingly.
3187 The API docs are still at http://curator.readthedocs.io. Be sure to
3189 read the latest docs, or select the docs for 3.0.0.
3190 General
3192 · Breaking changes to the API. Because this is a major point revi‐
3194 sion, changes to the API have been made which are non-reverse com‐
3195 patible. Before upgrading, be sure to update your scripts and
3196 test them thoroughly.
3197 · Python 3 support Somewhere along the line, Curator would no longer
3199 work with curator. All tests now pass for both Python2 and
3200 Python3, with 99% code coverage in both environments.
3201 · New CLI library. Using Click now. http://click.pocoo.org/3/ This
3203 change is especially important as it allows very easy CLI integra‐
3204 tion testing.
3205 · Pipelined filtering! You can now use --older-than & --newer-than
3207 in the same command! You can also provide your own regex via the
3208 --regex parameter. You can use multiple instances of the
3209 --exclude flag.
3210 · Manually include indices! With the --index paramter, you can add
3212 an index to the working list. You can provide multiple instances
3213 of the --index parameter as well!
3214 · Tests! So many tests now. Test coverage of the API methods is at
3216 100% now, and at 99% for the CLI methods. This doesn't mean that
3217 all of the tests are perfect, or that I haven't missed some sce‐
3218 narios. It does mean, however, that it will be much easier to
3219 write tests if something turns up missed. It also means that any
3220 new functionality will now need to have tests.
3221 · Iteration changes Methods now only iterate through each index when
3223 appropriate! In fact, the only commands that iterate are alias
3224 and optimize. The bloom command will iterate, but only if you
3225 have added the --delay flag with a value greater than zero.
3226 · Improved packaging! Methods have been moved into categories of
3228 api and cli, and further broken out into individual modules to
3229 help them be easier to find and read.
3230 · Check for allocation before potentially re-applying an allocation
3232 rule. #273 (ferki)
3233 · Assigning replica count and routing allocation rules _can_ be done
3235 to closed indices. #283 (ferki)
3236 Bug fixes
3238 · Don't accidentally delete .kibana index. #261 (malagoli)
3240 · Fix segment count for empty indices. #265 (untergeek)
3242 · Change bloom filter cutoff Elasticsearch version to 1.4. Reported
3244 in #267 (untergeek)
3245 3.0.0rc1 (5 March 2015)
3247 Release Notes
3248 RC1 is here! I'm re-releasing the Changes from all betas here, minus
3250 the intra-beta code fixes. Barring any show stoppers, the official
3251 release will be soon.
3252 General
3254 · Breaking changes to the API. Because this is a major point revi‐
3256 sion, changes to the API have been made which are non-reverse com‐
3257 patible. Before upgrading, be sure to update your scripts and
3258 test them thoroughly.
3259 · Python 3 support Somewhere along the line, Curator would no longer
3261 work with curator. All tests now pass for both Python2 and
3262 Python3, with 99% code coverage in both environments.
3263 · New CLI library. Using Click now. http://click.pocoo.org/3/ This
3265 change is especially important as it allows very easy CLI integra‐
3266 tion testing.
3267 · Pipelined filtering! You can now use --older-than & --newer-than
3269 in the same command! You can also provide your own regex via the
3270 --regex parameter. You can use multiple instances of the
3271 --exclude flag.
3272 · Manually include indices! With the --index paramter, you can add
3274 an index to the working list. You can provide multiple instances
3275 of the --index parameter as well!
3276 · Tests! So many tests now. Test coverage of the API methods is at
3278 100% now, and at 99% for the CLI methods. This doesn't mean that
3279 all of the tests are perfect, or that I haven't missed some sce‐
3280 narios. It does mean, however, that it will be much easier to
3281 write tests if something turns up missed. It also means that any
3282 new functionality will now need to have tests.
3283 · Methods now only iterate through each index when appropriate!
3285 · Improved packaging! Hopefully the entry_point issues some users
3287 have had will be addressed by this. Methods have been moved into
3288 categories of api and cli, and further broken out into individual
3289 modules to help them be easier to find and read.
3290 · Check for allocation before potentially re-applying an allocation
3292 rule. #273 (ferki)
3293 · Assigning replica count and routing allocation rules _can_ be done
3295 to closed indices. #283 (ferki)
3296 Bug fixes
3298 · Don't accidentally delete .kibana index. #261 (malagoli)
3300 · Fix segment count for empty indices. #265 (untergeek)
3302 · Change bloom filter cutoff Elasticsearch version to 1.4. Reported
3304 in #267 (untergeek)
3305 3.0.0b4 (5 March 2015)
3307 Notes
3308 Integration testing! Because I finally figured out how to use the
3310 Click Testing API, I now have a good collection of command-line simula‐
3311 tions, complete with a real back-end. This testing found a few bugs
3312 (this is why testing exists, right?), and fixed a few of them.
3313 Bug fixes
3315 · HUGE! curator show snapshots would _delete_ snapshots. This is
3317 fixed.
3318 · Return values are now being sent from the commands.
3320 · scripttest is no longer necessary (click.Test works!)
3322 · Calling get_snapshot without a snapshot name returns all snapshots
3324 3.0.0b3 (4 March 2015)
3326 Bug fixes
3327 · setup.py was lacking the new packages "curator.api" and "cura‐
3329 tor.cli" The package works now.
3330 · Python3 suggested I had to normalize the beta tag to just b3, so
3332 that's also changed.
3333 · Cleaned out superfluous imports and logger references from the
3335 __init__.py files.
3336 3.0.0-beta2 (3 March 2015)
3338 Bug fixes
3339 · Python3 issues resolved. Tests now pass on both Python2 and
3341 Python3
3342 3.0.0-beta1 (3 March 2015)
3344 General
3345 · Breaking changes to the API. Because this is a major point revi‐
3347 sion, changes to the API have been made which are non-reverse com‐
3348 patible. Before upgrading, be sure to update your scripts and
3349 test them thoroughly.
3350 · New CLI library. Using Click now. http://click.pocoo.org/3/
3352 · Pipelined filtering! You can now use --older-than & --newer-than
3354 in the same command! You can also provide your own regex via the
3355 --regex parameter. You can use multiple instances of the
3356 --exclude flag.
3357 · Manually include indices! With the --index paramter, you can add
3359 an index to the working list. You can provide multiple instances
3360 of the --index parameter as well!
3361 · Tests! So many tests now. Unit test coverage of the API methods
3363 is at 100% now. This doesn't mean that all of the tests are per‐
3364 fect, or that I haven't missed some scenarios. It does mean that
3365 any new functionality will need to also have tests, now.
3366 · Methods now only iterate through each index when appropriate!
3368 · Improved packaging! Hopefully the entry_point issues some users
3370 have had will be addressed by this. Methods have been moved into
3371 categories of api and cli, and further broken out into individual
3372 modules to help them be easier to find and read.
3373 · Check for allocation before potentially re-applying an allocation
3375 rule. #273 (ferki)
3376 Bug fixes
3378 · Don't accidentally delete .kibana index. #261 (malagoli)
3380 · Fix segment count for empty indices. #265 (untergeek)
3382 · Change bloom filter cutoff Elasticsearch version to 1.4. Reported
3384 in #267 (untergeek)
3385 2.1.2 (22 January 2015)
3387 Bug fixes
3388 · Do not try to set replica count if count matches provided argu‐
3390 ment. #247 (bobrik)
3391 · Fix JSON logging (Logstash format). #250 (magnusbaeck)
3393 · Fix bug in filter_by_space() which would match all indices if the
3395 provided patterns found no matches. Reported in #254 (untergeek)
3396 2.1.1 (30 December 2014)
3398 Bug fixes
3399 · Renamed unnecessarily redundant --replicas to --count in args for
3401 curator_script.py
3402 2.1.0 (30 December 2014)
3404 General
3405 · Snapshot name now appears in log output or STDOUT. #178
3407 (untergeek)
3408 · Replicas! You can now change the replica count of indices.
3410 Requested in #175 (untergeek)
3411 · Delay option added to Bloom Filter functionality. #206 (untergeek)
3413 · Add 2-digit years as acceptable pattern (y vs. Y). Reported in
3415 #209 (untergeek)
3416 · Add Docker container definition #226 (christianvozar)
3418 · Allow the use of 0 with --older-than, --most-recent and
3420 --delete-older-than. See #208. #211 (bobrik)
3421 Bug fixes
3423 · Edge case where 1.4.0.Beta1-SNAPSHOT would break version check.
3425 Reported in #183 (untergeek)
3426 · Typo fixed. #193 (ferki)
3428 · Type fixed. #204 (gheppner)
3430 · Shows proper error in the event of concurrent snapshots. #177
3432 (untergeek)
3433 · Fixes erroneous index display of _, a, l, l when --all-indices
3435 selected. Reported in #222 (untergeek)
3436 · Use json.dumps() to escape exceptions. Reported in #210
3438 (untergeek)
3439 · Check if index is closed before adding to alias. Reported in #214
3441 (bt5e)
3442 · No longer force-install argparse if pre-installed #216 (whyscream)
3444 · Bloom filters have been removed from Elasticsearch 1.5.0. Update
3446 methods and tests to act accordingly. #233 (untergeek)
3447 2.0.2 (8 October 2014)
3449 Bug fixes
3450 · Snapshot name not displayed in log or STDOUT #185 (untergeek)
3452 · Variable name collision in delete_snapshot() #186 (untergeek)
3454 2.0.1 (1 October 2014)
3456 Bug fix
3457 · Override default timeout when snapshotting --all-indices #179
3459 (untergeek)
3460 2.0.0 (25 September 2014)
3462 General
3463 · New! Separation of Elasticsearch Curator Python API and cura‐
3465 tor_script.py (untergeek)
3466 · New! --delay after optimize to allow cluster to quiesce #131
3468 (untergeek)
3469 · New! --suffix option in addition to --prefix #136 (untergeek)
3471 · New! Support for wildcards in prefix & suffix #136 (untergeek)
3473 · Complete refactor of snapshots. Now supporting incrementals!
3475 (untergeek)
3476 Bug fix
3478 · Incorrect error msg if no indices sent to create_snapshot
3480 (untergeek)
3481 · Correct for API change coming in ES 1.4 #168 (untergeek)
3483 · Missing " in Logstash log format #143 (cassianoleal)
3485 · Change non-master node test to exit code 0, log as INFO. #145
3487 (untergeek)
3488 · months option missing from validate_timestring() (untergeek)
3490 1.2.2 (29 July 2014)
3492 Bug fix
3493 · Updated README.md to briefly explain what curator does #117
3495 (untergeek)
3496 · Fixed es_repo_mgr logging whitelist #119 (untergeek)
3498 · Fixed absent months time-unit #120 (untergeek)
3500 · Filter out .marvel-kibana when prefix is .marvel- #120 (untergeek)
3502 · Clean up arg parsing code where redundancy exists #123 (untergeek)
3504 · Properly divide debug from non-debug logging #125 (untergeek)
3506 · Fixed show command bug caused by changes to command structure #126
3508 (michaelweiser)
3509 1.2.1 (24 July 2014)
3511 Bug fix
3512 · Fixed the new logging when called by curator entrypoint.
3514 1.2.0 (24 July 2014)
3516 General
3517 · New! Allow user-specified date patterns: --timestring #111
3519 (untergeek)
3520 · New! Curate weekly indices (must use week of year) #111
3522 (untergeek)
3523 · New! Log output in logstash format --logformat logstash #111
3525 (untergeek)
3526 · Updated! Cleaner default logs (debug still shows everything)
3528 (untergeek)
3529 · Improved! Dry runs are more visible in log output (untergeek)
3531 Errata
3533 · The --separator option was removed in lieu of user-specified date
3535 patterns.
3536 · Default --timestring for days: %Y.%m.%d (Same as before)
3538 · Default --timestring for hours: %Y.%m.%d.%H (Same as before)
3540 · Default --timestring for weeks: %Y.%W
3542 1.1.3 (18 July 2014)
3544 Bug fix
3545 · Prefix not passed in get_object_list() #106 (untergeek)
3547 · Use os.devnull instead of /dev/null for Windows #102 (untergeek)
3549 · The http auth feature was erroneously omitted #100 (bbuchacher)
3551 1.1.2 (13 June 2014)
3553 Bug fix
3554 · This was a showstopper bug for anyone using RHEL/CentOS with a
3556 Python 2.6 dependency for yum
3557 · Python 2.6 does not like format calls without an index. #96 via
3559 #95 (untergeek)
3560 · We won't talk about what happened to 1.1.1. No really. I hate
3562 git today :(
3563 1.1.0 (12 June 2014)
3565 General
3566 · Updated! New command structure
3568 · New! Snapshot to fs or s3 #82 (untergeek)
3570 · New! Add/Remove indices to alias #82 via #86 (cschellenger)
3572 · New! --exclude-pattern #80 (ekamil)
3574 · New! (sort of) Restored --log-level support #73 (xavier-calland)
3576 · New! show command-line options #82 via #68 (untergeek)
3578 · New! Shard Allocation Routing #82 via #62 (nickethier)
3580 Bug fix
3582 · Fix --max_num_segments not being passed correctly #74 (untergeek)
3584 · Change BUILD_NUMBER to CURATOR_BUILD_NUMBER in setup.py #60
3586 (mohabusama)
3587 · Fix off-by-one error in time calculations #66 (untergeek)
3589 · Fix testing with python3 #92 (untergeek)
3591 Errata
3593 · Removed optparse compatibility. Now requires argparse.
3595 1.0.0 (25 Mar 2014)
3597 General
3598 · compatible with elasticsearch-py 1.0 and Elasticsearch 1.0 (honza‐
3600 kral)
3601 · Lots of tests! (honzakral)
3603 · Streamline code for 1.0 ES versions (honzakral)
3605 Bug fix
3607 · Fix find_expired_indices() to not skip closed indices (honzakral)
3609 0.6.2 (18 Feb 2014)
3611 General
3612 · Documentation fixes #38 (dharrigan)
3614 · Add support for HTTPS URI scheme and optparse compatibility for
3616 Python 2.6 (gelim)
3617 · Add elasticsearch module version checking for future compatibility
3619 checks (untergeek)
3620 0.6.1 (08 Feb 2014)
3622 General
3623 · Added tarball versioning to setup.py (untergeek)
3625 Bug fix
3627 · Fix long_description by including README.md in MANIFEST.in
3629 (untergeek)
3630 · Incorrect version number in curator.py (untergeek)
3632 0.6.0 (08 Feb 2014)
3634 General
3635 · Restructured repository to a be a proper python package. (arieb)
3637 · Added setup.py file. (arieb)
3639 · Removed the deprecated file logstash_index_cleaner.py (arieb)
3641 · Updated README.md to fit the new package, most importantly the
3643 usage and installation. (arieb)
3644 · Fixes and package push to PyPI (untergeek)
3646 0.5.2 (26 Jan 2014)
3648 General
3649 · Fix boolean logic determining hours or days for time selection
3651 (untergeek)
3652 0.5.1 (20 Jan 2014)
3654 General
3655 · Fix can_bloom to compare numbers (HonzaKral)
3657 · Switched find_expired_indices() to use datetime and timedelta
3659 · Do not try and catch unrecoverable exceptions. (HonzaKral)
3661 · Future proofing the use of the elasticsearch client (i.e. work
3663 with version 1.0+ of Elasticsearch) (HonzaKral) Needs more test‐
3664 ing, but should work.
3665 · Add tests for these scenarios (HonzaKral)
3667 0.5.0 (17 Jan 2014)
3669 General
3670 · Deprecated logstash_index_cleaner.py Use new curator.py instead
3672 (untergeek)
3673 · new script change: curator.py (untergeek)
3675 · new add index optimization (Lucene forceMerge) to reduce segments
3677 and therefore memory usage. (untergeek)
3678 · update refactor of args and several functions to streamline opera‐
3680 tion and make it more readable (untergeek)
3681 · update refactor further to clean up and allow immediate (and
3683 future) portability (HonzaKral)
3684 0.4.0
3686 General
3687 · First version logged in CHANGELOG
3689 · new --disable-bloom-days feature requires 0.90.9+
3691 http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/index-modules-codec.html#bloom-postings
3693 This can save a lot of heap space on cold indexes (i.e. not
3695 actively indexing documents)
3696
3698 Copyright (c) 2012–2016 Elasticsearch <http://www.elastic.co>
3699 Licensed under the Apache License, Version 2.0 (the "License"); you may
3701 not use this file except in compliance with the License. You may
3702 obtain a copy of the License at
3703 http://www.apache.org/licenses/LICENSE-2.0
3704 Unless required by applicable law or agreed to in writing, software
3706 distributed under the License is distributed on an "AS IS" BASIS, WITH‐
3707 OUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
3708 See the License for the specific language governing permissions and
3709 limitations under the License.
3710
3712 · genindex
3713 · search
3715
3717 Aaron Mildenstein
3718
3720 2011-2016, Elasticsearch
37214.2 Jul 13, 2018 CURATOR(1)