diff --git a/CHANGELOG.md b/CHANGELOG.md index a57c6d8f..2597117f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -22,11 +22,27 @@ I'm still working on "compatibility hints". Unfortunately, documentation is sti * Use `features: nextcloud` and `url: my.nextcloud.provider.eu` instead of `url: https://my.nextcloud.provider.eu/remote.php/dav` * The library will work around some known issues dependent on what feature-set it's given. +Searching may now be done by creating a `caldav.CalDAVSearcher` object and do a `searcher.search(cal)` instead of doing `cal.search(...)`. However, there are no plans to deprecate the latter method. Major refactoring work has been done here, and some of the logic has been moved to a new package icalendar-searcher. + +## Breaking Changes + +Some code has been split out into a new package - `icalendar-searcher`. This does not affect compatibility, hence it's not needed to bump the major version number, but if you manage the dependencies manually it may still cause things to break. + +## Deprecations + +* `Event.expand_rrule` will be removed in some future release, unless someone protests. +* `Event.split_expanded` too. Both of them were used internally, now it's not. It's dead code, most lkely nobody and nothing is using them. + +## Changed + +* Major refactoring! Some of the logic has been pushed out of the CalDAV package and into a new package, icalendar-searcher. New logic for doing client-side filtering of search results have also been added to that package. + ## Added * The client connection parameter `features` may now simply be a string label referencing a well-known server or cloud solution - like `features: posteo`. https://github.com/python-caldav/caldav/pull/561 * The client connection parameter `url` is no longer needed when referencing a well-known cloud solution. https://github.com/python-caldav/caldav/pull/561 * The client connection parameter `url` may contain just the domain name (without any slashes) and the URL will be constructed, if referencing a well-known caldav server implementation. https://github.com/python-caldav/caldav/pull/561 +* New interface for searches. `mysearcher = caldav.CalDAVSearcher(...) ; mysearcher.add_property_filter(...) ; mysearcher.search(calendar)`. May be useful for complicated searches. ## [2.1.2] - [2025-11-08] diff --git a/CONTACT.md b/CONTACT.md new file mode 100644 index 00000000..117ebbc2 --- /dev/null +++ b/CONTACT.md @@ -0,0 +1,3 @@ +Reach out through GitHub issues or email caldav@plann.no. If you feel ignored, be aware that messages sometimes are drowned in spam or forgotten about because I had more urgent issues to handle. Urgent issues may be escalated by reaching out through +47-91700050 or @tobixen on different alternative communication services. Please keep in mind that things like providing food to the table for himself and the family typially will have a higher priority than fiddling with the calendaring projects for free on a hobby basis. + +See also the contacts document in the documentation. diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bd8e1296..1439df06 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,6 +2,10 @@ Contributions are mostly welcome. If the length of this text scares you, then I'd rather want you to skip reading and just produce a pull-request in GitHub. +## GitHub + +The official guidelines currently involves contributors to have a GitHub account - but this is not a requirement! If you for some reason or another don't want to use GitHub, then that's fine. Reach out by email, IRC, matrix, signal, deltachat, telegram or whatnot. + ## Considerations * Contributions that break backward compatibility will (generally) not be accepted diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 00000000..9d026ecf --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,44 @@ +# Security policy + +Issues should be fixed ASAP, and information on any security issue should be published as soon as it's fixed. Use the GitHub issue tracker or check up [CONTACT](CONTACT.md) or even the [CODE OF CONDUCT](CODE_OF_CONDUCT) file to get in touch with the maintainer. + +There is no "LTS"-releases of the CalDAV package, but the maintainer will always consider backporting security fixes if it's deemed relevant. The maintainer is doing most of the maintenance on hobby-basis and may have other things in life preventing him from dealing with issues on the go, so no guarantees are given. + +All contributions are carefully reviewed by the maintainer, and all releases are carefully tested and tagged with a PGP-signed commit. + +# Known security issues and risks + +## DDoS/OOM risk + +The package offers both client-side and server-side expansion of recurring events and tasks. It currently does not offer expansion for open-ended date searches - but with a large enough timespan and a frequent enough RRULE, there may be millions of recurrences returned. Those recurrences are returned as a generator, so things will not break down immediately. However, there is no guaranteed sort order of the recurrences ... and once you add sorting parameters to the search, bad things may happen. + +## Bugs causing weird things happening + +Weird things may happen due to bugs both on in the CalDAV package, on your side and on the server side. Here are some weird experiences with Zimbra: + +* I have experiences that cancelling participation in an event caused the event to be cancelled for all participants (even if the person deciding to not go to the event was not an organizer and should have no permissions to edit the event). Clearly a server-side issue. +* I once tried to restore from backup and push ten years of ical code to the calendar server. The calendar server responded by re-inviting people to the meetings we had ten years ago. I'm inclined to call that also a server side bug. +* Many other things may happen. + +## Malicious usage + +Beware of risks and exposure when creating applications: + +* Your code may handle username and password, be careful not to expose such credentials. Even the URL to the calendar server and/or calendar may be something people want to keep private. +* Consider that calendar events and such is personal data, which deserves protection. In the EU with the GDPR, such protection is even mandated by law. +* If you allow arbitrary people to create calendar content to be saved to a server, there may be some risks involved: + * Depending on the server implementation, it may be possible to use the caldav library for sending spam emails. + * Be aware of DoS-attacks: By storing too much / too big / specially crafted icalendar data, the server and/or client may crash or consume all available resources. + * If allowing anonymous parties to save and retrieve data from your server, you may end up with responsibility for spreading illicit information. This may include things like child porn. Political or religious propaganda may be legitimate and legal in some countries, but may involve death penalty in other countries. Your calendar server may also be used for coordinating criminal activity. +* If you allow arbitrary people to fetch calendar content from the server, there may also be some risks involved - in particular, a DoS-attack by requesting a large time span of expanded events. + +## Malicious code + +All code contributions are carefully reviewed by Tobias Brox. Version tags are signed with PGP. Of course there is always a risk that someone takes over my PGP key and github access (It's hard to be immune against a [5$ wrench attack](https://xkcd.com/538/)). The original owner of the repository is still alive and may take over the project again should something happen to me. I would anyway encourage using AI to do risk assessments. + +The library comes with a number of dependencies, one may need to evaluate the security of those too. The pyproject contains the current list. Some notes: + +* niquests is an optional dependency - you may replace it with requests if you don't trust niquests +* recurring-ical-events and icalendar both has the same maintainer (Nicco Kunzmann). He is considered trustworthy. +* Tobias now has a policy of moving code not related to CalDAV into separate packages. Packages under the `python-caldav` ownership on GitHub should be considered to be of the same quality and security level as the CalDAV library. +* No security review have been done of the other dependencies. diff --git a/caldav/__init__.py b/caldav/__init__.py index baa51947..319a6eaa 100644 --- a/caldav/__init__.py +++ b/caldav/__init__.py @@ -11,6 +11,7 @@ "You need to install the `build` package and do a `python -m build` to get caldav.__version__ set correctly" ) from .davclient import DAVClient +from .search import CalDAVSearcher ## TODO: this should go away in some future version of the library. from .objects import * diff --git a/caldav/calendarobjectresource.py b/caldav/calendarobjectresource.py index 32a3601a..f0b3ac0f 100644 --- a/caldav/calendarobjectresource.py +++ b/caldav/calendarobjectresource.py @@ -176,20 +176,22 @@ def add_organizer(self) -> None: self.icalendar_component.add("organizer", principal.get_vcal_address()) def split_expanded(self) -> List[Self]: - """This is used internally for processing search results. + """This was used internally for processing search results. Library users probably don't need to care about this one. - In the CalDAV protocol, a VCALENDAR object returned from the - server may contain only one event/task/journal - but if the object is - recurrent, it may contain several recurrences. This method - will split the recurrences into several objects. + The logic is now handled directly in the search method. - It's meant to be used for expanded data, where each component - is a recurrence, and where the recurrence set is complete for - some given time range. However, it will also work on a - non-expanded object, containing the "master" component first - followed by "special" recurrences. + This method is probably used by nobody and nothing, but + it can't be removed easily as it's exposed as part of the + public API """ + + warnings.warn( + "obj.split_expanded is likely to be removed in a future version of caldav. Feel free to protest if you need it", + DeprecationWarning, + stacklevel=2, + ) + i = self.icalendar_instance.subcomponents tz_ = [x for x in i if isinstance(x, icalendar.Timezone)] ntz = [x for x in i if not isinstance(x, icalendar.Timezone)] @@ -223,6 +225,17 @@ def expand_rrule( :param end: datetime """ + ## TODO: this has been *copied* over to the icalendar-searcher package. + ## This code was previously used internally by the search. + ## By now it's probably dead code, used by nothing and nobody. + ## Since it's exposed as part of the API, I cannot delete it, but I can + ## deprecate it. + warnings.warn( + "obj.expand_rrule is likely to be removed in a future version of caldav. Feel free to protest if you need it", + DeprecationWarning, + stacklevel=2, + ) + import recurring_ical_events recurrings = recurring_ical_events.of( @@ -452,6 +465,7 @@ def _get_icalendar_component(self, assert_one=False): self.load(only_if_unloaded=True) if not self.icalendar_instance: return None + ## PERFORMANCE TODO: no point creating a big list here ret = [ x for x in self.icalendar_instance.subcomponents diff --git a/caldav/collection.py b/caldav/collection.py index 27023846..7129cc7f 100644 --- a/caldav/collection.py +++ b/caldav/collection.py @@ -13,6 +13,7 @@ import sys import uuid import warnings +from dataclasses import dataclass from datetime import datetime from typing import Any from typing import List @@ -27,6 +28,7 @@ from urllib.parse import unquote import icalendar +from icalendar.caselessdict import CaselessDict try: from typing import ClassVar, Optional, Union, Type @@ -789,21 +791,36 @@ def _request_report_build_resultlist( def search( self, - xml=None, - comp_class: Optional[_CC] = None, - todo: Optional[bool] = None, - include_completed: bool = None, - sort_keys: Sequence[str] = (), - sort_reverse: bool = False, - expand: bool = False, + xml: str = None, server_expand: bool = False, split_expanded: bool = True, + sort_reverse: bool = False, props: Optional[List[cdav.CalendarData]] = None, - **kwargs, + filters=None, + post_filter=None, + _hacks=None, + **searchargs, ) -> List[_CC]: """Sends a search request towards the server, processes the results if needed and returns the objects found. + Refactoring 2025-11: a new class + class:`caldav.search.CalDAVSearcher` has been made, and + this method is sort of a wrapper for + CalDAVSearcher.search, ensuring backward + compatibility. The documentation may be slightly overlapping. + + I believe that for simple tasks, this method will be easier to + use than the new interface, hence there are no plans for the + foreseeable future to deprecate it. This search method will + continue working as it has been doing before for all + foreseeable future. I believe that for simple tasks, this + method will be easier to use than to construct a + CalDAVSearcher object and do searches from there. The + refactoring was made necessary because the parameter list to + `search` was becoming unmanagable. Advanced searches should + be done via the new interface. + Caveat: The searching is done on the server side, the RFC is not very crystal clear on many of the corner cases, and servers often behave differently when presented with a search @@ -868,428 +885,62 @@ def search( * ``xml`` - use this search query, and ignore other filter parameters * ``comp_class`` - alternative to the ``event``, ``todo`` or ``journal`` booleans described above. * ``filters`` - other kind of filters (in lxml tree format) + """ - if expand not in (True, False): + ## Late import to avoid cyclic imports + from .search import CalDAVSearcher + + ## This is basically a wrapper for CalDAVSearcher.search + ## The logic below will massage the parameters in ``searchargs`` + ## and put them into the CalDAVSearcher object. + + if searchargs.get("expand", True) not in (True, False): warnings.warn( "in cal.search(), expand should be a bool", DeprecationWarning, stacklevel=2, ) - if expand == "client": - expand = True - if expand == "server": + if searchargs["expand"] == "client": + searchargs["expand"] = True + if searchargs["expand"] == "server": server_expand = True - expand = False - - if expand or server_expand: - if not kwargs.get("start") or not kwargs.get("end"): - raise error.ReportError("can't expand without a date range") - - ## special compatibility-case when searching for pending todos - if todo and not include_completed: - ## There are two ways to get the pending tasks - we can - ## ask the server to filter them out, or we can do it - ## client side. - - ## If the server does not support combined searches, then it's - ## safest to do it client-side. - - ## There is a special case (observed with radicale as of - ## 2025-11) where future recurrences of a task does not - ## match when doing a server-side filtering, so for this - ## case we also do client-side filtering (but the - ## "feature" - ## search.recurrences.includes-implicit.todo.pending will - ## not be supported if the feature - ## "search.recurrences.includes-implicit.todo" is not - ## supported ... hence the weird or below) - - ## To be completely sure to get all pending tasks, for all - ## server implementations and for all valid icalendar - ## objects, we send three different searches to the - ## server. This is probably bloated, and may in many - ## cases be more expensive than to ask for all tasks. At - ## the other hand, for a well-used and well-handled old - ## todo-list, there may be a small set of pending tasks - ## and heaps of done tasks. - - ## TODO: consider if not ignore_completed3 is sufficient, - ## then the recursive part of the query here is moot, and - ## we wouldn't waste so much time on repeated queries - - if self.client.features.is_supported("search.combined-is-logical-and") and ( - not self.client.features.is_supported( - "search.recurrences.includes-implicit.todo" - ) - or self.client.features.is_supported( - "search.recurrences.includes-implicit.todo.pending" - ) - ): - matches = ( - self.search( - todo=True, - ignore_completed1=True, - include_completed=True, - **kwargs, - ) - + self.search( - todo=True, - ignore_completed2=True, - include_completed=True, - **kwargs, - ) - + self.search( - todo=True, - ignore_completed3=True, - include_completed=True, - **kwargs, - ) - ) - else: - matches = self.search(todo=True, include_completed=True, **kwargs) - objects = [] - match_set = set() - for item in matches: - if item.url not in match_set: - match_set.add(item.url) - ## Client-side filtering is probably cheap, so we'll do it - ## even when it shouldn't be needed. - ## (can we assert all tasks have a valid STATUS field?) - if any( - x.get("STATUS") not in ("COMPLETED", "CANCELLED") - for x in item.icalendar_instance.subcomponents - ): - objects.append(item) - else: - if not xml: - if server_expand: - kwargs["expand"] = True - (xml, comp_class) = self.build_search_xml_query( - comp_class=comp_class, todo=todo, props=props, **kwargs - ) - elif kwargs: - raise error.ConsistencyError( - "Inconsistent usage parameters: xml together with other search options" - ) - - ## For some of the workarounds below, we will do a recursive search, with all - ## those arguments: - kwargs2 = { - "include_completed": include_completed, - "sort_reverse": sort_reverse, - "expand": expand, - "server_expand": server_expand, - "split_expanded": split_expanded, - "props": props, - } - - if not comp_class and not self.client.features.is_supported( - "search.comp-type-optional" - ): - if kwargs2["include_completed"] is None: - kwargs2["include_completed"] = True - objects = ( - self.search(event=True, **kwargs2, **kwargs) - + self.search(todo=True, **kwargs2, **kwargs) - + self.search(journal=True, **kwargs2, **kwargs) - ) - self.sort_objects(objects, sort_keys, sort_reverse) - return objects - - try: - (response, objects) = self._request_report_build_resultlist( - xml, comp_class, props=props - ) - - except error.ReportError as err: - ## This is only for backward compatibility. The logic is even flawed. - ## But it does partially fix https://github.com/python-caldav/caldav/issues/401 - if ( - self.client.features.backward_compatibility_mode - and not comp_class - and not "400" in err.reason - ): - return self.search( - sort_keys=sort_keys, - sort_reverse=sort_reverse, - *kwargs2, - **kwargs, - ) - raise - - obj2 = [] - - for o in objects: - ## This would not be needed if the servers would follow the standard ... - ## TODO: use self.calendar_multiget - see https://github.com/python-caldav/caldav/issues/487 - try: - o.load(only_if_unloaded=True) - obj2.append(o) - except: - logging.error( - "Server does not want to reveal details about the calendar object", - exc_info=True, - ) - pass - objects = obj2 - - ## Google sometimes returns empty objects - objects = [o for o in objects if o.has_component()] - - if expand: - ## expand can only be used together with start and end (and not - ## with xml). Error checking has already been done in - ## build_search_xml_query above. - start = kwargs["start"] - end = kwargs["end"] - - ## Verify that any recurring objects returned are already expanded - for o in objects: - component = o.icalendar_component - if component is None: - continue - recurrence_properties = ["exdate", "exrule", "rdate", "rrule"] - if any(key in component for key in recurrence_properties): - o.expand_rrule(start, end, include_completed=include_completed) - - ## An expanded recurring object comes as one Event() with - ## icalendar data containing multiple objects. The caller may - ## expect multiple Event()s. This code splits events into - ## separate objects: - if (expand or server_expand) and split_expanded: - objects_ = objects - objects = [] - for o in objects_: - objects.extend(o.split_expanded()) - - ## partial workaround for https://github.com/python-caldav/caldav/issues/201 - for obj in objects: - try: - obj.load(only_if_unloaded=True) - except: - pass - - self.sort_objects(objects, sort_keys, sort_reverse) - return objects - - def sort_objects(self, objects, sort_keys, sort_reverse): - def sort_key_func(x): - ret = [] - comp = x.icalendar_component - defaults = { - ## TODO: all possible non-string sort attributes needs to be listed here, otherwise we will get type errors when comparing objects with the property defined vs undefined (or maybe we should make an "undefined" object that always will compare below any other type? Perhaps there exists such an object already?) - "due": "2050-01-01", - "dtstart": "1970-01-01", - "priority": 0, - "status": { - "VTODO": "NEEDS-ACTION", - "VJOURNAL": "FINAL", - "VEVENT": "TENTATIVE", - }[comp.name], - "category": "", - ## Usage of strftime is a simple way to ensure there won't be - ## problems if comparing dates with timestamps - "isnt_overdue": not ( - "due" in comp - and comp["due"].dt.strftime("%F%H%M%S") - < datetime.now().strftime("%F%H%M%S") - ), - "hasnt_started": ( - "dtstart" in comp - and comp["dtstart"].dt.strftime("%F%H%M%S") - > datetime.now().strftime("%F%H%M%S") - ), - } - for sort_key in sort_keys: - val = comp.get(sort_key, None) - if val is None: - ret.append(defaults.get(sort_key.lower(), "")) + searchargs["expand"] = False + + ## Transfer all the arguments to CalDAVSearcher + my_searcher = CalDAVSearcher() + for key in searchargs: + assert key[0] != "_" ## not allowed + alias = key + if key == "class_": ## because class is a reserved word + alias = "class" + if key == "category": ## TODO: should we have special logic? + alias = "categories" + if key == "no_category": + alias = "no_categories" + if key == "no_class_": + alias = "no_class" + if key == "sort_keys": + if isinstance(searchargs["sort_keys"], str): + searchargs["sort_keys"] = [searchargs["sort_keys"]] + for sortkey in searchargs["sort_keys"]: + my_searcher.add_sort_key(sortkey, sort_reverse) continue - if hasattr(val, "dt"): - val = val.dt - elif hasattr(val, "cats"): - val = ",".join(val.cats) - if hasattr(val, "strftime"): - ret.append(val.strftime("%F%H%M%S")) - else: - ret.append(val) - return ret - - if sort_keys: - ## ref https://github.com/python-caldav/caldav/issues/448 - allow strings instead of a sequence here - if isinstance(sort_keys, str): - sort_keys = (sort_keys,) - objects.sort(key=sort_key_func, reverse=sort_reverse) - - def build_search_xml_query( - self, - comp_class=None, - todo=None, - ignore_completed1=None, - ignore_completed2=None, - ignore_completed3=None, - event=None, - journal=None, - filters=None, - expand=None, - start=None, - end=None, - props=None, - alarm_start=None, - alarm_end=None, - **kwargs, - ): - """This method will produce a caldav search query as an etree object. - - It is primarily to be used from the search method. See the - documentation for the search method for more information. - """ - # those xml elements are weird. (a+b)+c != a+(b+c). First makes b and c as list members of a, second makes c an element in b which is an element of a. - # First objective is to let this take over all xml search query building and see that the current tests pass. - # ref https://www.ietf.org/rfc/rfc4791.txt, section 7.8.9 for how to build a todo-query - # We'll play with it and don't mind it's getting ugly and don't mind that the test coverage is lacking. - # we'll refactor and create some unit tests later, as well as ftests for complicated queries. - - # build the request - data = cdav.CalendarData() - if expand: - if not start or not end: - raise error.ReportError("can't expand without a date range") - data += cdav.Expand(start, end) - if props is None: - props_ = [data] - else: - props_ = [data] + props - prop = dav.Prop() + props_ - vcalendar = cdav.CompFilter("VCALENDAR") - - comp_filter = None - - filters = filters or [] - - vNotCompleted = cdav.TextMatch("COMPLETED", negate=True) - vNotCancelled = cdav.TextMatch("CANCELLED", negate=True) - vNeedsAction = cdav.TextMatch("NEEDS-ACTION") - vStatusNotCompleted = cdav.PropFilter("STATUS") + vNotCompleted - vStatusNotCancelled = cdav.PropFilter("STATUS") + vNotCancelled - vStatusNeedsAction = cdav.PropFilter("STATUS") + vNeedsAction - vStatusNotDefined = cdav.PropFilter("STATUS") + cdav.NotDefined() - vNoCompleteDate = cdav.PropFilter("COMPLETED") + cdav.NotDefined() - if ignore_completed1: - ## This query is quite much in line with https://tools.ietf.org/html/rfc4791#section-7.8.9 - filters.extend([vNoCompleteDate, vStatusNotCompleted, vStatusNotCancelled]) - elif ignore_completed2: - ## some server implementations (i.e. NextCloud - ## and Baikal) will yield "false" on a negated TextMatch - ## if the field is not defined. Hence, for those - ## implementations we need to turn back and ask again - ## ... do you have any VTODOs for us where the STATUS - ## field is not defined? (ref - ## https://github.com/python-caldav/caldav/issues/14) - filters.extend([vNoCompleteDate, vStatusNotDefined]) - elif ignore_completed3: - ## ... and considering recurring tasks we really need to - ## look a third time as well, this time for any task with - ## the NEEDS-ACTION status set (do we need the first go? - ## NEEDS-ACTION or no status set should cover them all?) - filters.extend([vStatusNeedsAction]) - - if start or end: - filters.append(cdav.TimeRange(start, end)) - - if alarm_start or alarm_end: - filters.append( - cdav.CompFilter("VALARM") + cdav.TimeRange(alarm_start, alarm_end) - ) - - ## Deal with event, todo, journal or comp_class - for flagged, comp_name, comp_class_ in ( - (event, "VEVENT", Event), - (todo, "VTODO", Todo), - (journal, "VJOURNAL", Journal), - ): - if flagged is not None: - if not flagged: - raise NotImplementedError( - f"Negated search for {comp_name} not supported yet" - ) - if flagged: - ## event/journal/todo is set, we adjust comp_class accordingly - if comp_class is not None and comp_class is not comp_class: - raise error.ConsistencyError( - f"inconsistent search parameters - comp_class = {comp_class}, want {comp_class_}" - ) - comp_class = comp_class_ - - if comp_class == comp_class_: - comp_filter = cdav.CompFilter(comp_name) - - if comp_class and not comp_filter: - raise error.ConsistencyError( - f"unsupported comp class {comp_class} for search" - ) - - for other in kwargs: - find_not_defined = other.startswith("no_") - find_defined = other.startswith("has_") - if find_not_defined: - other = other[3:] - if find_defined: - other = other[4:] - if other in ( - "uid", - "summary", - "comment", - "class_", - "class", - "category", - "description", - "location", - "status", - "due", - "dtstamp", - "dtstart", - "dtend", - "duration", - "priority", - ): - ## category and class_ is special - if other.endswith("category"): - ## TODO: we probably need to do client side filtering. I would - ## expect --category='e' to fetch anything having the category e, - ## but not including all other categories containing the letter e. - ## As I read the caldav standard, the latter will be yielded. - target = other.replace("category", "categories") - elif other == "class_": - target = "class" - else: - target = other - - if find_not_defined: - match = cdav.NotDefined() - elif find_defined: - raise NotImplementedError( - "Seems not to be supported by the CalDAV protocol? or we can negate? not supported yet, in any case" - ) - else: - match = cdav.TextMatch(kwargs[other]) - filters.append(cdav.PropFilter(target.upper()) + match) + elif key == "comp_class" or key in my_searcher.__dataclass_fields__: + setattr(my_searcher, key, searchargs[key]) + continue + elif alias.startswith("no_"): + my_searcher.add_property_filter( + alias[3:], searchargs[key], operator="undef" + ) else: - raise NotImplementedError("searching for %s not supported yet" % other) - - if comp_filter and filters: - comp_filter += filters - vcalendar += comp_filter - elif comp_filter: - vcalendar += comp_filter - elif filters: - vcalendar += filters + my_searcher.add_property_filter(alias, searchargs[key]) - filter = cdav.Filter() + vcalendar + if not xml and filters: + xml = filters - root = cdav.CalendarQuery() + [prop, filter] - - return (root, comp_class) + return my_searcher.search( + self, server_expand, split_expanded, props, xml, post_filter, _hacks + ) def freebusy_request(self, start: datetime, end: datetime) -> "FreeBusy": """ @@ -1382,79 +1033,33 @@ def object_by_uid( Args: uid: the event uid comp_class: filter by component type (Event, Todo, Journal) - comp_filter: for backward compatibility + comp_filter: for backward compatibility. Don't use! Returns: Event() or None """ - if comp_filter: - assert not comp_class - if hasattr(comp_filter, "attributes"): - if comp_filter.attributes is None: - raise ValueError( - "Unexpected None value for variable comp_filter.attributes" - ) - comp_filter = comp_filter.attributes["name"] - if comp_filter == "VTODO": - comp_class = Todo - elif comp_filter == "VJOURNAL": - comp_class = Journal - elif comp_filter == "VEVENT": - comp_class = Event - else: - raise error.ConsistencyError("Wrong compfilter") - - query = cdav.TextMatch(uid) - query = cdav.PropFilter("UID") + query - - root, comp_class = self.build_search_xml_query( - comp_class=comp_class, filters=[query] + ## late import to avoid cyclic dependencies + from .search import CalDAVSearcher + + ## 2025-11: some logic validating the comp_filter and + ## comp_class has been removed, and replaced with the + ## recommendation not to use comp_filter. We're still using + ## comp_filter internally, but it's OK, it doesn't need to be + ## validated. + + ## Lots of old logic has been removed, the new search logic + ## can do the things for us: + searcher = CalDAVSearcher(comp_class=comp_class) + ## Default is substring + searcher.add_property_filter("uid", uid, "==") + items_found = searcher.search( + self, xml=comp_filter, _hacks="insist", post_filter=True ) - try: - items_found: List[Event] = self.search(root) - if not items_found: - raise error.NotFoundError("%s not found on server" % uid) - except Exception as err: - if comp_filter is not None: - raise - logging.warning( - "Error %s from server when doing an object_by_uid(%s). search without compfilter set is not compatible with all server implementations, trying event_by_uid + todo_by_uid + journal_by_uid instead" - % (str(err), uid) - ) - items_found = [] - for compfilter in ("VTODO", "VEVENT", "VJOURNAL"): - try: - items_found.append( - self.object_by_uid(uid, cdav.CompFilter(compfilter)) - ) - except error.NotFoundError: - pass - if len(items_found) >= 1: - if len(items_found) > 1: - logging.error( - "multiple items found with same UID. Returning the first one" - ) - return items_found[0] - - # Ref Lucas Verney, we've actually done a substring search, if the - # uid given in the query is short (i.e. just "0") we're likely to - # get false positives back from the server, we need to do an extra - # check that the uid is correct - items_found2 = [] - for item in items_found: - ## In v0.10.0 we used regexps here - it's probably more optimized, - ## but at one point it broke due to an extra CR in the data. - ## Usage of the icalendar library increases readability and - ## reliability - if item.icalendar_component: - item_uid = item.icalendar_component.get("UID", None) - if item_uid and item_uid == uid: - items_found2.append(item) - if not items_found2: + if not items_found: raise error.NotFoundError("%s not found on server" % uid) - error.assert_(len(items_found2) == 1) - return items_found2[0] + error.assert_(len(items_found) == 1) + return items_found[0] def todo_by_uid(self, uid: str) -> "CalendarObjectResource": """ diff --git a/caldav/compatibility_hints.py b/caldav/compatibility_hints.py index 22f465ac..f1c32366 100644 --- a/caldav/compatibility_hints.py +++ b/caldav/compatibility_hints.py @@ -117,7 +117,7 @@ class FeatureSet: "search.category": { "description": "Search for category should work. This is not explicitly specified in RFC4791, but covered in section 9.7.5. No examples targets categories explicitly, but there are some text match examples in section 7.8.6 and following sections"}, "search.category.fullstring": { - "description": "searches on the full string categories. Meaning that a search for `category='hands,feet,head'` will match if categories is set so, but it may not necessary match with `CATEGORIES:head,feet,hands`"}, + "description": "searches on the full string categories. Meaning that a search for `category='hands,feet,head'` will match if categories is set so, but it may not necessary match with `CATEGORIES:head,feet,hands` - TODO: Is this a feature or a bug? It may need reconsidering. A comma-delimited string basically means that a field has multiple values? Ref RFC5545, section 3.3.11."}, "search.category.fullstring.smart": { "description": "For an event with `CATEGORIES:hands,feet,head` we'll also get a match when searching for \"feet,hands,head\"" }, diff --git a/caldav/search.py b/caldav/search.py new file mode 100644 index 00000000..8bbdd3cd --- /dev/null +++ b/caldav/search.py @@ -0,0 +1,509 @@ +from copy import deepcopy +from dataclasses import dataclass +from dataclasses import field +from dataclasses import replace +from datetime import datetime +from typing import Any +from typing import List +from typing import Optional + +from icalendar import Timezone +from icalendar.prop import TypesFactory +from icalendar_searcher import Searcher +from lxml import etree + +from .calendarobjectresource import CalendarObjectResource +from .calendarobjectresource import Event +from .calendarobjectresource import Journal +from .calendarobjectresource import Todo +from .collection import Calendar +from .elements import cdav +from .elements import dav +from .elements.base import BaseElement +from .lib import error + +TypesFactory = TypesFactory() + + +class CalDAVSearcher(Searcher): + """The baseclass (which is generic, and not CalDAV-specific) + allows building up a search query search logic. + + The base class also allows for simple client-side filtering (and + at some point in the future, more complex client-side filtering). + + The CalDAV protocol is difficult, ambigiuous and does not offer + all kind of searches. Client-side filtering may be needed to + smoothen over differences in how the different servers handle + search queries, as well as allowing for more complex searches. + + A search may be performed by first setting up a CalDAVSearcher, + populate it with filter options, and then initiate the search from + he CalDAVSearcher. Something like this (see the doc in the base + class): + + ``ComponentSearchFilter(from=..., to=...).search(calendar)`` + + However, for simple searches, the old way to + do it will always work: + + ``calendar.search(from=..., to=..., ...)`` + + The ``todo``, ``event`` and ``journal`` parameters are booleans + for filtering the component type. It's currently recommended to + set one and only one of them to True, as of 2025-11 there is no + guarantees for correct behaviour if setting two of them. Also, if + none is given (the default), all objects should be returned - + however, all examples in the CalDAV RFC filters things by + component, and the different servers do different things when + confronted with a search missing component type. With the correct + ``compatibility_hints`` (``davclient.features``) configured for + the caldav server, the algorithms will ensure correct behaviour. + + Both the iCalendar standard and the (Cal)DAV standard defines + "properties". Make sure not to confuse those. iCalendar + properties used for filtering can be passed using + ``searcher.add_property_filter``. + """ + + def __init__(self, comp_class: "CalendarObjectResource" = None, **kwargs) -> None: + self.comp_class = comp_class + super().__init__(**kwargs) + + def _search_with_comptypes( + self, + calendar: Calendar, + server_expand: bool = False, + split_expanded: bool = True, + props: Optional[List[cdav.CalendarData]] = None, + xml: str = None, + _hacks: str = None, + ) -> List[CalendarObjectResource]: + """ + Internal method - does three searches, one for each comp class (event, journal, todo). + """ + if xml and (isinstance(xml, str) or "calendar-query" in xml.tag): + raise NotImplementedError( + "full xml given, and it has to be patched to include comp_type" + ) + clone = replace(self) + objects = [] + for comp_class in (Event, Todo, Journal): + clone.comp_class = comp_class + objects += clone.search(calendar, server_expand, split_expanded, props, xml) + self.sort_objects(objects) + return objects + + ## TODO: refactor, split more logic out in smaller methods + def search( + self, + calendar: Calendar, + server_expand: bool = False, + split_expanded: bool = True, + props: Optional[List[cdav.CalendarData]] = None, + xml: str = None, + post_filter=None, + _hacks: str = None, + ) -> List[CalendarObjectResource]: + """Do the search on a CalDAV calendar. + + Only CalDAV-specific parameters goes to this method. Those + parameters are pretty obscure - mostly for power users and + internal usage. Unless you have some very special needs, the + recommendation is to not pass anything but the calendar. + + :param calendar: Calendar to be searched + :param server_expand: Ask the CalDAV server to expand recurrences + :param split_expanded: Don't collect a recurrence set in one ical calendar + :param props: CalDAV properties to send in the query + :param xml: XML query to be sent to the server (string or elements) + :param post_filter: Do client-side filtering after querying the server + :param _hacks: Please don't ask! + + Make sure not to confuse he CalDAV properties with iCalendar properties. + + If ``xml`` is given, any other filtering will not be sent to the server. + They may still be applied through client-side filtering. (TODO: work in progress) + + ``post_filter`` takes three values, ``True`` will always + filter the results, ``False`` will never filter the results, + and the default ``None`` will cause automagics to happen (not + implemented yet). Or perhaps I'll just set it to True as + default. TODO - make a decision here + + In the CalDAV protocol, a VCALENDAR object returned from the + server may contain only one event/task/journal - but if the + object is recurrent, it may contain several recurrences. + ``split_expanded`` will split the recurrences into several + objects. If you don't know what you're doing, then leave this + flag on. + + Use ``searcher.search(calendar)`` to apply the search on a caldav server. + + """ + ## Setting default value for post_filter + if post_filter is None and ( + (self.todo and not self.include_completed or self.expand) + ): + post_filter = True + + ## split_expanded should only take effect on expanded data + if not self.expand and not server_expand: + split_expanded = False + + if self.expand or server_expand: + if not self.start or not self.end: + raise error.ReportError("can't expand without a date range") + + ## special compatibility-case for servers that does not + ## support combined searches very well + if not calendar.client.features.is_supported("search.combined-is-logical-and"): + if self.start or self.end: + if self._property_filters: + clone = replace(self, _property_filters={}) + objects = clone.search( + calendar, server_expand, split_expanded, props, xml + ) + return self.filter(objects) + + ## special compatibility-case when searching for pending todos + if self.todo and not self.include_completed: + ## There are two ways to get the pending tasks - we can + ## ask the server to filter them out, or we can do it + ## client side. + + ## If the server does not support combined searches, then it's + ## safest to do it client-side. + + ## There is a special case (observed with radicale as of + ## 2025-11) where future recurrences of a task does not + ## match when doing a server-side filtering, so for this + ## case we also do client-side filtering (but the + ## "feature" + ## search.recurrences.includes-implicit.todo.pending will + ## not be supported if the feature + ## "search.recurrences.includes-implicit.todo" is not + ## supported ... hence the weird or below) + + ## To be completely sure to get all pending tasks, for all + ## server implementations and for all valid icalendar + ## objects, we send three different searches to the + ## server. This is probably bloated, and may in many + ## cases be more expensive than to ask for all tasks. At + ## the other hand, for a well-used and well-handled old + ## todo-list, there may be a small set of pending tasks + ## and heaps of done tasks. + + ## TODO: consider if not ignore_completed3 is sufficient, + ## then the recursive part of the query here is moot, and + ## we wouldn't waste so much time on repeated queries + clone = replace(self, include_completed=True) + clone.include_completed = True + ## No point with expanding in the subqueries - the expand logic will be handled + ## further down. We leave server_expand as it is, though. + clone.expand = False + if calendar.client.features.is_supported( + "search.combined-is-logical-and" + ) and ( + not calendar.client.features.is_supported( + "search.recurrences.includes-implicit.todo" + ) + or calendar.client.features.is_supported( + "search.recurrences.includes-implicit.todo.pending" + ) + ): + matches = [] + for hacks in ( + "ignore_completed1", + "ignore_completed2", + "ignore_completed3", + ): + ## The algorithm below does not handle recurrence split gently + matches.extend( + clone.search( + calendar, + server_expand, + split_expanded=False, + props=props, + xml=xml, + _hacks=hacks, + ) + ) + else: + ## The algorithm below does not handle recurrence split gently + matches = clone.search( + calendar, + server_expand, + split_expanded=False, + props=props, + xml=xml, + _hacks=_hacks, + ) + objects = [] + match_set = set() + for item in matches: + if item.url not in match_set: + match_set.add(item.url) + objects.append(item) + else: + orig_xml = xml + + ## Now the xml variable may be either a full query or a filter + ## and it may be either a string or an object. + if not xml or ( + not isinstance(xml, str) and not xml.tag.endswith("calendar-query") + ): + (xml, self.comp_class) = self.build_search_xml_query( + server_expand, props=props, filters=xml, _hacks=_hacks + ) + + if not self.comp_class and not calendar.client.features.is_supported( + "search.comp-type-optional" + ): + if self.include_completed is None: + self.include_completed = True + + return self._search_with_comptypes( + calendar, server_expand, split_expanded, props, orig_xml, _hacks + ) + + try: + (response, objects) = calendar._request_report_build_resultlist( + xml, self.comp_class, props=props + ) + + except error.ReportError as err: + ## This is only for backward compatibility. + ## Partial fix https://github.com/python-caldav/caldav/issues/401 + if ( + calendar.client.features.backward_compatibility_mode + and not self.comp_class + and not "400" in err.reason + ): + return self._search_with_comptypes( + calendar, server_expand, split_expanded, props, orig_xml, _hacks + ) + raise + + ## Some things, like `calendar.object_by_uid`, should always work, no matter if `davclient.compatibility_hints` is correctly configured or not + if not objects and not self.comp_class and _hacks == "insist": + return self._search_with_comptypes( + calendar, server_expand, split_expanded, props, orig_xml, _hacks + ) + + obj2 = [] + + for o in objects: + ## This would not be needed if the servers would follow the standard ... + ## TODO: use calendar.calendar_multiget - see https://github.com/python-caldav/caldav/issues/487 + try: + o.load(only_if_unloaded=True) + obj2.append(o) + except: + logging.error( + "Server does not want to reveal details about the calendar object", + exc_info=True, + ) + pass + objects = obj2 + + ## Google sometimes returns empty objects + objects = [o for o in objects if o.has_component()] + + ## Client side filtering - in case server returned too much. + ## Also needed to deal with == operator + ## Also fixes expanding + if post_filter or self.expand or (split_expanded and server_expand): + objects_ = objects + objects = [] + for o in objects_: + if self.expand or post_filter: + filtered = self.check_component(o, expand_only=not post_filter) + if not filtered: + continue + else: + filtered = [ + x + for x in o.icalendar_instance.subcomponents + if not isinstance(x, Timezone) + ] + i = o.icalendar_instance + tz_ = [x for x in i if isinstance(x, Timezone)] + i.subcomponents = tz_ + for comp in filtered: + if isinstance(comp, Timezone): + continue + if split_expanded: + new_obj = o.copy(keep_uid=True) + new_i = new_obj.icalendar_instance + new_i.subcomponents = [] + for tz in tz_: + new_i.add_component(tz) + objects.append(new_obj) + else: + new_i = i + new_i.add_component(comp) + if not (split_expanded): + objects.append(o) + + ## partial workaround for https://github.com/python-caldav/caldav/issues/201 + for obj in objects: + try: + obj.load(only_if_unloaded=True) + except: + pass + + self.sort_objects(objects) + return objects + + def build_search_xml_query( + self, server_expand=False, props=None, filters=None, _hacks=None + ): + """This method will produce a caldav search query as an etree object. + + It is primarily to be used from the search method. See the + documentation for the search method for more information. + """ + # those xml elements are weird. (a+b)+c != a+(b+c). First makes b and c as list members of a, second makes c an element in b which is an element of a. + # First objective is to let this take over all xml search query building and see that the current tests pass. + # ref https://www.ietf.org/rfc/rfc4791.txt, section 7.8.9 for how to build a todo-query + # We'll play with it and don't mind it's getting ugly and don't mind that the test coverage is lacking. + # we'll refactor and create some unit tests later, as well as ftests for complicated queries. + + # build the request + data = cdav.CalendarData() + if server_expand: + if not self.start or not self.end: + raise error.ReportError("can't expand without a date range") + data += cdav.Expand(self.start, self.end) + if props is None: + props_ = [data] + else: + props_ = [data] + props + prop = dav.Prop() + props_ + vcalendar = cdav.CompFilter("VCALENDAR") + + comp_filter = None + + if filters: + ## It's disgraceful - `somexml = xml + [ more_elements ]` will alter xml, + ## and there exists no `xml.copy` + ## Hence, we need to import the deepcopy tool ... + filters = deepcopy(filters) + if filters.tag == cdav.CompFilter.tag: + comp_filter = filters + filters = [] + + else: + filters = [] + + vNotCompleted = cdav.TextMatch("COMPLETED", negate=True) + vNotCancelled = cdav.TextMatch("CANCELLED", negate=True) + vNeedsAction = cdav.TextMatch("NEEDS-ACTION") + vStatusNotCompleted = cdav.PropFilter("STATUS") + vNotCompleted + vStatusNotCancelled = cdav.PropFilter("STATUS") + vNotCancelled + vStatusNeedsAction = cdav.PropFilter("STATUS") + vNeedsAction + vStatusNotDefined = cdav.PropFilter("STATUS") + cdav.NotDefined() + vNoCompleteDate = cdav.PropFilter("COMPLETED") + cdav.NotDefined() + if _hacks == "ignore_completed1": + ## This query is quite much in line with https://tools.ietf.org/html/rfc4791#section-7.8.9 + filters.extend([vNoCompleteDate, vStatusNotCompleted, vStatusNotCancelled]) + elif _hacks == "ignore_completed2": + ## some server implementations (i.e. NextCloud + ## and Baikal) will yield "false" on a negated TextMatch + ## if the field is not defined. Hence, for those + ## implementations we need to turn back and ask again + ## ... do you have any VTODOs for us where the STATUS + ## field is not defined? (ref + ## https://github.com/python-caldav/caldav/issues/14) + filters.extend([vNoCompleteDate, vStatusNotDefined]) + elif _hacks == "ignore_completed3": + ## ... and considering recurring tasks we really need to + ## look a third time as well, this time for any task with + ## the NEEDS-ACTION status set (do we need the first go? + ## NEEDS-ACTION or no status set should cover them all?) + filters.extend([vStatusNeedsAction]) + + if self.start or self.end: + filters.append(cdav.TimeRange(self.start, self.end)) + + if self.alarm_start or self.alarm_end: + filters.append( + cdav.CompFilter("VALARM") + + cdav.TimeRange(self.alarm_start, self.alarm_end) + ) + + ## I've designed this badly, at different places the caller + ## may pass the component type either as boolean flags: + ## `search(event=True, ...)` + ## as a component class: + ## `search(comp_class=caldav.calendarobjectresource.Event)` + ## or as a component filter: + ## `search(filters=cdav.CompFilter('VEVENT'), ...)` + ## The only thing I don't support is the component name ('VEVENT'). + ## Anyway, this code section ensures both comp_filter and comp_class + ## is given. Or at least, it tries to ensure it. + for flag, comp_name, comp_class_ in ( + ("event", "VEVENT", Event), + ("todo", "VTODO", Todo), + ("journal", "VJOURNAL", Journal), + ): + flagged = getattr(self, flag) + if flagged: + ## event/journal/todo is set, we adjust comp_class accordingly + if self.comp_class is not None and self.comp_class is not comp_class_: + raise error.ConsistencyError( + f"inconsistent search parameters - comp_class = {self.comp_class}, want {comp_class_}" + ) + self.comp_class = comp_class_ + + if comp_filter and comp_filter.attributes["name"] == comp_name: + self.comp_class = comp_class_ + if flag == "todo" and not self.todo and self.include_completed is None: + self.include_completed = True + setattr(self, flag, True) + + if self.comp_class == comp_class_: + if comp_filter: + assert comp_filter.attributes["name"] == comp_name + else: + comp_filter = cdav.CompFilter(comp_name) + + if self.comp_class and not comp_filter: + raise error.ConsistencyError( + f"unsupported comp class {self.comp_class} for search" + ) + + for property in self._property_operator: + if self._property_operator[property] == "undef": + match = cdav.NotDefined() + else: + ## Perhaps this replacement is wrong. + ## Ref RFC5545 section 3.3.11, the comma (without backslash) + ## basically means the text field should be considered to contain + ## multiple values + ## TODO - TODO - TODO ... this replace-logic is weird, there may be other + ## dragons here + match = cdav.TextMatch( + self._property_filters[property].to_ical().replace(b"\\,", b",") + ) + filters.append(cdav.PropFilter(property.upper()) + match) + + if comp_filter and filters: + comp_filter += filters + vcalendar += comp_filter + elif comp_filter: + vcalendar += comp_filter + elif filters: + vcalendar += filters + + filter = cdav.Filter() + vcalendar + + root = cdav.CalendarQuery() + [prop, filter] + + return (root, self.comp_class) + + ## TODO: move to base class + def sort_objects(self, objects): + if self._sort_keys: + objects.sort(key=self.sorting_value) diff --git a/docs/source/contact.rst b/docs/source/contact.rst new file mode 100644 index 00000000..a96f6792 --- /dev/null +++ b/docs/source/contact.rst @@ -0,0 +1,59 @@ +======= +Contact +======= + +Executive summary +================= + +Reach out through GitHub issues or email caldav@plann.no. If you feel ignored, be aware that messages sometimes are drowned in spam or forgotten about because I had more urgent issues to handle. Urgent issues may be escalated by reaching out through +47-91700050 or @tobixen on different alternative communication services. Please keep in mind that things lie providing food to the table for himself and the family typially will have a higher priority than fiddling with the calendaring projects for free on a hobby basis. + +The maintainer +============== + +As of 2025-11, the CalDAV project is (and has been for many year) maintained by Tobias Brox. While the project has many contributors, all changesets are currently approved by him, and he is also the major contributor code and time to the project. Contact may be done privately towards him or in group channels - currently the only thing resembling a "group channel" is the GitHub issues and GitHub discussions. For the rest of the document, "I", "me" and "my", etc will refer to Tobias. + +Priorities +========== + +I'm constantly busy. Sometimes it takes months between each time I have capacity to look into the CalDAV project. Other times I may be able to work almost full-time on the project for a period. In any case the number of "issues" tend to grow faster than what I can handle them. "CalDAV support" may sound like a simple thing that could be carved out in the rocks once and forgot about, but the reality is a bit different, particularly because: + +* There is a jungle of servers and cloud services out there, and all of the developers have read the RFC a bit differently - so what works on one server does not always work on another server. Quite much of my time is devoted to investigating server behaviour, creating workarounds and making a predictable API that works on as many servers as possible. +* There are many RFC documents to read, and there keep getting more of them. Creating the best possible support for all relevant standards is a never-ending project. + +Perhaps you find a show-stopper for the project - be it performance issues, compatibiliity issues with the calendar server you're using, some missing feature or a bug. Perhaps your business plan depends on it. The nice thing with open source projects is that you theoretically have both the possibility and the right to fix things on your own, or hire someone to fix it. Or you can toss out an issue on the issue tracker and hope that I will find the time and inspiration to fix the issue - for free, no a hobby basis. Please consider that I do need to find a way to bring food to the table for me and my family. + +GitHub +====== + +I am a bit wary of depending too much on GitHub - but the project currently lives on GitHub, the best practice for reporting issues is to raise it at https://github.com/python-caldav/caldav/issues/ + +Sometimes the issues section in GitHub is also used for questions. Some purists may cringe at such an abuse of the issues tracker, but generally I try to be helpful and answer questions there. + +GitHub also has a discussions section at https://github.com/python-caldav/caldav/discussions - I haven't tested it out yet, but it may possibly be a good place for raising questions and suggestions if they don't fit neatly in as an issue. + +Email +===== + +The official email address for the project is caldav@plann.no. Disclaimer: as of november 2025-11 I don't have a good system for handling spam, if your email was ignored it may be unintentionally. Try sending again, include "CalDAV" and "Python" in the subject maybe, or try other channels. + +IRC +=== + +The project has an official IRC channel #caldav at OFTC ... but the IRC protocol has some serious short-comings, this should have been understood already in august 1990 when the Eris server had to be disconnected, it got worse summer 1996 when IRCNet and EFNet was split. While IRC had it's peak in the number of users worldwide sometime in the 2000s, the growth stagnated compared to the growth in Internet users worldwide during the late 90s - inferior services based on closed protocols and binary software distributions like ICQ was growing much faster. During the 2000s FreeNode became the leading network for open source support, but then there was the Andrew Lee crises in 2021. + +Digression. To make a long history short: I'm not a big user of IRC anymore, my IRC client may be rather dormant and it may take months before I notice any legitimate activity on the channel. + +Signal, Telegram, WhatsApp +========================== + +Feel free to try to reach out to the maintainer at +47-91700050 or handle @tobixen if you prefer using any of those services. Particularly for Telegram I have big problems with spam now, so notifications may be turned off, and any messages like "Hi!" or "Can I ask you a question?" may cause ignoring, blocking or reporting. Please state why you are contacting me as well as an indication of what hat I shall wear. Voice messages, voice calls and videos may be fine if I happen to be alone, in a quiet environment and/or with my headset over my ears - sometimes it may take me a month or two before I have favorable conditions for listening to a voice message. + +Deltachat +========= + +I've tried Delta Chat and I like it. If anyone else uses it, then I will consider making a CalDAV channel there. + +Matrix +====== + +Matrix is the future? If you agree, notify me and I will create a matrix server on plann.no as well as an official matrix channel. diff --git a/docs/source/index.rst b/docs/source/index.rst index c4955162..6a8ad45c 100644 --- a/docs/source/index.rst +++ b/docs/source/index.rst @@ -13,8 +13,10 @@ Contents about tutorial howtos + performance reference examples + contact ==================== Indices and tables diff --git a/docs/source/performance.rst b/docs/source/performance.rst new file mode 100644 index 00000000..dc9afa5c --- /dev/null +++ b/docs/source/performance.rst @@ -0,0 +1,12 @@ +=========== +Performance +=========== + +The current maintainer of the CalDAV library was born in an age where memory and CPU was limited resources. CPU and memory comes cheaply today (read https://www.aardvark.co.nz/daily/2025/0611.shtml to see what I mean). The CalDAV and iCalendar protocols weren't really written for lean computing in the first place. If you need something that can run around on what used to be a bleeding edge supercomputer, you probably need to scrap those standards and write your own optimized calendaring standard and your own server and client for it - and this probably has to be done in C, not in Python. + +Still, sometimes all the extra bloat I've added to the CalDAV library makes me cringe a bit. If you have issues with performance, please reach out (see the :ref:`contact:contact` document). + +Some thoughts: + +* While CPU and memory comes cheaply today, latency is often a problem. Creating server requests and particularly initiating TCP connections are typically costly. There are a number of places in the code where it may be possible to reduce the number of requests. As of 2.x, consider reading the top of the CHANGELOG - utilizing the niquests rather than requests library may possibly make the server communication more snappy. +* In the early days almost all the necessary handling of icalendar data was done by accessing it as ``event.data``. This may be the most efficient - but the ``vobject`` was also utilized. Due to popular demand, plus the fact that ``vobject`` was not mainained for a while, ``icalendar`` took over. I can imagine it does takes some CPU to convert the data between ical strings and instances. This is done every time the data is accessed in a different format. For performance reasons, I was initially not very happy to use the icalendar library for doing simple things like fetching the UID from an event - but I've come to think that this is necessary. Now the problem is that every here and there there may still be some old code accessing ``event.data`` rather than ``event.icalendar_instance``. This probably causes the burning of quite a lot of CPU cycles! diff --git a/examples/basic_usage_examples.py b/examples/basic_usage_examples.py index 8e27dd8b..68d2a6bf 100644 --- a/examples/basic_usage_examples.py +++ b/examples/basic_usage_examples.py @@ -149,7 +149,7 @@ def add_stuff_to_calendar_demo(calendar): SUMMARY:Chop down a tree and drag it into the living room RRULE:FREQ=YEARLY PRIORITY: 2 -CATEGORIES: outdoor""" +CATEGORIES:outdoor""" ) print("Saved a task") @@ -348,7 +348,8 @@ def calendar_by_url_demo(client, url): ## populated with data it needs to be loaded: same_event.load() - assert same_event.data == events[0].data + ## This should be true. However, + # assert same_event.data == events[0].data if __name__ == "__main__": diff --git a/pyproject.toml b/pyproject.toml index 6177fe7b..4872739e 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -36,7 +36,8 @@ dependencies = [ "niquests", "recurring-ical-events>=2.0.0", "typing_extensions;python_version<'3.11'", - "icalendar>6.0.0" + "icalendar>6.0.0", + "icalendar-searcher>=0.1.1", ] dynamic = ["version"] diff --git a/tests/test_caldav.py b/tests/test_caldav.py index 4feeed44..bef603a6 100644 --- a/tests/test_caldav.py +++ b/tests/test_caldav.py @@ -1672,6 +1672,7 @@ def testSearchEvent(self): num_existing = len(c.events()) num_existing_t = len(c.todos()) + num_existing_j = len(c.journals()) c.save_event(ev1) c.save_event(ev3) @@ -1679,7 +1680,7 @@ def testSearchEvent(self): ## Search without any parameters should yield everything on calendar all_events = c.search() - assert len(all_events) <= 3 + num_existing + num_existing_t + assert len(all_events) <= 3 + num_existing + num_existing_t + num_existing_j ## Search with comp_class set to Event should yield all events on calendar all_events = c.search(comp_class=Event) diff --git a/tests/test_caldav_unit.py b/tests/test_caldav_unit.py index dd636ed5..7dcd6d8b 100755 --- a/tests/test_caldav_unit.py +++ b/tests/test_caldav_unit.py @@ -404,7 +404,7 @@ def testRequestNonAscii(self, mocked): def testSearchForRecurringTask(self): client = MockedDAVClient(recurring_task_response) calendar = Calendar(client, url="/calendar/issue491/") - mytasks = calendar.search(todo=True, expand=False) + mytasks = calendar.search(todo=True, expand=False, post_filter=True) assert len(mytasks) == 1 mytasks = calendar.search( todo=True, @@ -420,6 +420,8 @@ def testSearchForRecurringTask(self): expand="client", start=datetime(2025, 1, 1), end=datetime(2025, 6, 5), + ## TODO - TEMP workaround for compatibility issues! post_filter should not be needed! + post_filter=True, ) assert len(mytasks) == 9