Thanks@theOehrly looks like a great amount of work. I've taken a quick look, and this looks like a good proof-of-concept.

Overall I am concerned about feature drift if we have two ways of dealing with dates and times, datetime and timedelta. There is alot of duplicated code, and that usually means a re-architecture could be considered for better maintainability.

Finding ticks for datetime and timedelta is the same, at least for the first month, so did you consider simply making timedeltas be datetime (ie dt = epoch + timedelta) under the hood (i.e. date2num would just take timedeltas as well as datetimes), and then changing the formatters, and perhaps the locator defaults? For instance, it looks to me like most of the new locators and formatters could be just specialized versions of the old ones with theself.intervald, orformats andoffset_formats changed to not deal with months or larger. Sure there are a few times when you calltimedelta2num instead ofdate2num, but again, I don't see any reasondate2num can't take timedeltas.

I had basically tried to keep date and timedelta separate. But yes, it turned out to have a lot of duplicate code which I hadn't quite expected to this extend. In hindsight it's not that much of a surprise really.
Your ideas and recommendations sound very reasonable. I'm going to work on that.

Sure - well keep the old version as it looks like it mostly works! Is there anything api-like we should discuss? I'll try and bring this up at a developer call, probably next week...

As a selling point, I'd suggest turning a few of your tests into graphical examples and at least adding them to the start of the PR. It helps folks know exactly what you have done and see why this is a useful addition.

I don't think I have any api related questions currently but will ask if anything comes up.

Keep the old version? How do you mean that. Should I for now just add addtional classes based on the date locators/formatters? So that there are two implementations for now?

I will create some graphical examples. That's certainly necessary,

... I just meant save a branch or make sure its really clear in the commit tree.

Ok, sure

The proposal for using date2num for both date and timedelta works well. Although it's a bit more convoluted than I'd like it to be,
Also I kept the ability to deal with pandas NaT. The separate function for that is only ever called as a last fallback before finally throwing an error if pandas NaT wasn't the problem before. Data that doesn't contain pandas NaT values will never use it.

I do have some issues though with subclassing the AutoDateLocator for timedeltas. If I simply do this...

class AutoTimedeltaLocator(AutoDateLocator):    def __init__(self, minticks=5, maxticks=None):        super().__init__(tz=None, minticks=minticks,                         interval_multiples=True)        self.maxticks = {YEARLY: 0, MONTHLY: 0, DAILY: 11, HOURLY: 12,                         MINUTELY: 11, SECONDLY: 11, MICROSECONDLY: 8}        if maxticks is not None:            try:                self.maxticks.update(maxticks)            except TypeError:                # Assume we were given an integer. Use this as the maximum                # number of ticks for every frequency and create a                # dictionary for this                self.maxticks = dict.fromkeys(self._freqs, maxticks)        self.intervald = {            YEARLY: [1, ],  # cannot be empty            MONTHLY: [1, ],              DAILY: [1, 2, 5, 10, 20, 25, 50, 100, 200, 500, 1000, 2000,                    5000, 10000, 20000, 50000, 100000, 200000, 500000,                    1000000],            HOURLY: [1, 2, 3, 4, 6, 8, 12],            MINUTELY: [1, 2, 3, 5, 10, 15, 20, 30],            SECONDLY: [1, 2, 3, 5, 10, 15, 20, 30],            MICROSECONDLY: [1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000,                            5000, 10000, 20000, 50000, 100000, 200000, 500000,                            1000000],        }        self._byranges = [None, None, None,                          range(0, 24), range(0, 60), range(0, 60), None]

... it seems to sort of work for short ranges of time. Say a few days. But for longer ones I'm not sure how to persuade rrule to use reasonable tick locations.

For example plotting a range of 70 days (95 to 165), this would be nice:

But the rrule based locator creates this:

I don't know how to tell rrule to consider 100 a nice number. It usually goes by days of a month and so on only. Here it seems to simply start somewhere at the beginning of the view limits for lack of a better alternative.

Also, I haven't yet figured out how to make .get_locator() of the AutoDateLocator completely disregard YEARLY and MONTHLY intervals. The above code doesn't work for that. It does not seem possible to completely skip both these frequencies by simply changing the intervals, maxticks and ranges.

I'm not sure whether this can be accomplished without multiple changes to AutoDateLocator and RRuleLocator and whether this can be done within the same methods. (Without making it very complicated). And then I'm no longer sure if this would be better than having completely separate locators.

One more question I have is about the general aim for the number of ticks. Am I right that the AutoDateLocator aims to for as many ticks as possible within maxticks? And minticks is considered the absolute minimum but not a desirable number?
I had previously thought of this exactly the other way round.

I've tried to clean this up a bit. Now there's no trial and error involved anymore to figure out dtypes.

The special conversion for pandas.NaT is more concise now. I think that np.iinfo('int64').min should be the same always? So comparing against that is a reliable way to do this in my opinion.

I have two questions. First, is there a reason why conversion is always done todatetime64[us] specifically in date2num and not todatetime64[ms] for example? Because in_dt64_to_ordinalf the values are converted again anyways.

Second, about the locators. If a minimum and a maximum number of ticks is specified and multiple intervals can give a number of ticks within that range, which interval should be chosen? The one that gives more ticks or the one that gives less ticks?

I have two questions. First, is there a reason why conversion is always done todatetime64[us] specifically in date2num and not todatetime64[ms] for example? Because in_dt64_to_ordinalf the values are converted again anyways.

[us] is what we guarantee being able to convert to (matches datetime's resolution. If we converted to[ms] we would lose possible precision.

All the fussing in_dt64_to_ordinalf is there to preserve as much floating point as possible. Its possible that could be simplified, but I think I tried and it broke some tests...

Second, about the locators. If a minimum and a maximum number of ticks is specified and multiple intervals can give a number of ticks within that range, which interval should be chosen? The one that gives more ticks or the one that gives less ticks?

I guess more? I think its always better to err on the side of giving he user too many ticks that they can then weed out by adjustment, rather than too few, and they don't know what ticks are there. But I'm not 100% sure. Some of the number of ticks logic is a bit arbitrary.

I do have another question. This time about the RRuleLocator, specifically about this part here

def tick_values(self, vmin, vmax):    delta = relativedelta(vmax, vmin)    # We need to cap at the endpoints of valid datetime    try:        start = vmin - delta    except (ValueError, OverflowError):        # cap        start = datetime.datetime(1, 1, 1, 0, 0, 0,                                  tzinfo=datetime.timezone.utc)    try:        stop = vmax + delta    except (ValueError, OverflowError):        # cap        stop = datetime.datetime(9999, 12, 31, 23, 59, 59,                                 tzinfo=datetime.timezone.utc)

This needs to be done a bit differently for timedeltas to allow ticks on interval multiples. But I want to be sure to fully understand why this is done this way specifically for dates. Why is the delta added/subtracted? Why does the range need to be extended at all? Is it just conevenient to use the delta or is there some very specific reason for that?

I don't particularly know. It looks like it is just to set the clipping? Maybe the rule needs to be set wider than thebetween values below? Certainly vmin and vmax need to be capped.

You might try simplifying this in a different PR (i.e. as a smaller change) and see if it breaks any of the tests.

I hadn't intended to change this for dates, but to have a subclass for timedeltas that overwrites this (partly at least). But I'm not sure about that yet. Maybe a cleaner way can be found. I guess I will have to play around with this some more.
The Daily locator for timedelta is a similar special case to the Yearly date locator. And then the Microsecond locator is another special case for both currently. This is not ideal.

I came across two things that are a bit of an issue potentially. Both are cause by converting timedelta to datetime by adding the epoch.

First, this "conversion" means that with the default epoch the largest valid negative timedelta is much smaller than the largest valid positive timedelta. It's somewhat obvious but I only thought of this now. Basically, the range of timedelta is constrained to the maximum datetime value which is 9999 years. We set the zero point at 1970 with the default epoch. Therefore the range of valid timedeltas is -1970 years to 8029 years. This is not very intuitive for a user I think.

The second problem concerns the locators and conversion to floating point values. The existing locators for microseconds, seconds, minutes and hours work fine with timedeltas. But only as long as the epoch is set to a full day..dates.set_epoch also accepts something like "2020-05-07T12:43" as a value for the epoch though. And if you set this, the locators will fail to determine proper tick locations (for timedelta only). I don't know why anyone would do that, but still it can be done.

I thought about having the epoch that is used as an offset not be the epoch that is set by the user. Instead maybe use a fixed offset of "5000-01-01" to fix both these problems. I haven't spend much time trying this yet though. So far, it didn't work well.

Any thoughts on these two things?

I have tried to work around datetime's issues a couple of times, but I don't think its worth the effort. As@efiring will point out, python dates are not particularly reliable missing leap seconds and other esoterica over long timescales. I don't think timedeltas on the scale of hundreds of years are particularly useful thing to support except as best we can viadatetime (and reallyrrules limitations). A user who is dealing with 10^6 days likely does not care exactly when the start of the month was 2000 years ago, and most such users typically use non-Gregorian calendars anyway.

The situation before changing the epoch to 1970, which is quite recent, was that we only allowed positive years, and folks have managed. If this were to be truly improved, I think someone would need to write a numpy-version of rrule that allowed much bigger date range.

I certainly prefer if I don't need to work around limitations set by python's datetime. I also don't think that supporting ranges of millions of days is very important. But I wanted to know your opinion on this.

As for not setting the epoch with minutes or seconds. Do you think this can be fixed by simply adding a comment in the documentation for set_epoch. Or should the timedelta locators just always work, even if somebody were set the internal epoch to something like "2020-05-18T21:15:05.125"? I see no reason why that is necessary and why anyone would do that.

As for not setting the epoch with minutes or seconds. Do you think this can be fixed by simply adding a comment in the documentation for set_epoch.

Sorry I'm not 100% following here - are strange epochs hard to handle somehow?

It's not about weird epochs but about having no zero point (literally).
The problem occurs because timedelta is treated as date internally.

I'll try to explain it with an example.

The data value is what the user wants to have plotted.
The internal value is what is passed to RRule after adding the epoch for conversion to date.

The values for hours, minutes,... differ between data value and internal value if the epoch is 'weird'.

You can still get ticks every say 15 minutes just fine.
But those ticks will no longer be at 0, 15, 30, 45.
RRule will chose the interval multiples based on the internal value (date).

Locating on days works fine because they need special treatement and cannot use rrules byranges anyways.

Part of the problem is that reusing the date functionalities means that a function is not aware wether it is working with date or timedelta values.

The best idea I had so far is to only allow modification of year, month and day of the epoch.
Or only add a warning to not break backwards compatibility.

Some other ideas I had (none of which I consider good):

• Use a separate, fixed offset epoch for converting to timedelta.
  Problem: num2date can't know whether it receives a date or timedelta as a number.
  Similarly if a num2timedelta function existed, locators wouldn't know which one to call.

• Somehow subtract a fixed epoch from the user set epoch to get the offset;
  use this to correct the values at some appropriate point.
  I couldn't make this work in some quick experiments so far.

• Make all objects aware of whether they are working with dates or timedeltas currently;
  pass that information on separately with the data values.
  This would introduce lots of extra code and make everything complex and difficult to understand.

@theOehrly sorry to let this languish. I was just thinking about time delta today and realized this was still here.

It seems you got hung up on strange epochs. Ihope no one is planning to set the epoch to anything other than Jan 1 XXXX T00:00:00.0000. If they do, then they are welcome to whatever breakage they get. If your point is that maybe we shouldn't allow arbitrary epoch values, that is probably true, but it is a little late for that.

@jklymak Yes, this was just a major unknown here for me. If strange epochs need to be supported, then the whole approach as it currently is, is probably not feasible.
If the requirement can be made that the internal epoch needs to be set to Jan 1 XXXX T00:00:00.0000 then everything should more or less work in its current state. I'd need some more time to look at it again, to tell you more precisely which points still need to be addressed from my point of view (if any).

Everything is still a bit convoluted, though. It's not immediately obvious that timedelta is just treated like a date internally.
There are separate formatters, but locators are shared. Except for the DayLocator which can't be shared between date and timedelta at all. And the AutoTimedeltaLocator needs to be separate, but can be a subclass of the AutoDateLocator to override just some of the logic.
I think this is mostly inherent to the approach of treating timedelta as date. It mostly works without problems, but you get some additional edge cases.

Well certainly you could go back to the approach of having a whole newtimedelta unit. Leaving aside the complication of the epoch (again I have little sympathy for random epochs), the issue here are time deltas longer than a week. Perhaps someone should define what they want to have happen if the timedelta is 50 days.

@jklymak The problem is that I also agree with your original concern that the completely separate approach introduces too much duplicate code. I'm not fully convinced by either version to be honest. But currently I don't really have enough time to do some useful work on this.

I don't really understand what you mean by defining what should happen with longer timedeltas. How they are represented? As days, weeks, years? I think days is the largest possible unit that makes sense. Weeks could be used too maybe, but in my opinion that's less intuitive. Everything longer than that doesn't have a fixed length anymore and would therefore be ambiguous.

Hi@theOehrly just pinging again about this. If you feel swamped, perhaps I would take it over (keeping you as a co-author)? I think weshould add something like this, but I'd have to play with it some as you have. However, its not urgent, so if you plan to take another pass at it, that would be fine as well.

@jklymak Hi, I actually haven't forgotten about this. I'm currently finishing my bachelor's thesis and should have substantially more time again from early February on. I was thinking about taking another look at this, among other things.

We should probably define a bit clearer what we expect this to achieve, what should be supported and what not. And then I can have a fresh look at it. You'd probably prefer this to be split into smaller chunks, too, instead of one large PR. That might be somewhat tricky. But I guess things like support for pandas NaT without importing pandas could be separated out at least.

Great! Just wanted to make sure it was on your radar. I agree that making datetime.timedelta and numpy.timedelta64 work should be the priority. However I don't think the length here is too onerous, it's just the very similar code that seems to say we need a refactor somewhere.

Hi@jklymak, I've just started to look into this again.

You said this some time ago:

[...] the issue here are time deltas longer than a week. Perhaps someone should define what they want to have happen if the timedelta is 50 days.

Thinking a bit about it, I suppose that the largest "unit" that we can support fortimedelta is "weeks". Everything longer just needs to be a multiple of that. A minor problem that I see here is that I'd probably prefer "100 days" over "15 weeks" in most cases. So this should be easily customizable.

If someone wantstimedelta values with ticks at "1 month", "2 months", ... we need to have a reference starting date, so that we can determine how long each specific month actually is. At that point, all values should just bedatetime, in my opinion. Ticks like that could then be achieved with a month locator and a custom tick formatter. I think this is a separate issue, if we wanted to support this natively.

If we can only supporttimedelta units with constant length, then we should really use a subclass of theMultipleLocator, I'd say. So, something closer to what I had originally done.
In my opinion, this reduces the overall complexity and makes the implementation more understandable. Also, in terms of functionality, I can't think of any advantage that the approach with conversion todatetime can offer. There's the disadvantage that it introduces a somewhat arbitrary limit on the largesttimedelta value that we could handle. Granted, with the limit being at a few thousand years, few people would ever run into this problem, but still.

Did you have any more ideas or think of anything that we should consider here?

@jklymak, just pinging you again as a quick reminder that this is still here.

I think we "just" need to determine what the proper way to implement this is. Most importantly for that, we need to agree on what this should be able to achieve. When we have that figured out, I could also try to do a bit of a concept specification where the constraints and expectations are made a bit more clear if you think that'd be helpful (don't know how I'd do that but we could figure that out too).

Sorry this has languished. I recently wanted this facility, and wished that we had it.@theOehrly are you happy to review this version, or should we start over? If I were starting over, I may just suggest we convert to float, where the float is in the units of the timedelta64 and then just plot as normal floats. But I can also see the appeal of having days/hours/minutes....

@jklymak I can review, update or rewrite this PR if you'd like to have this feature implemented in this way. I should be able to get this done sometime in October, if you want.

I guess the question is, whether you want days/hours/minutes/... or just the float version that you mentioned. I suppose there are a few things to consider here:

• code complexity and maintenance: prefer floats
• user preference: difficult to tell, likely differs throughout the user base
• flexibility and ability to create plots for a non-technical audience: prefer the days/hours/minutes/... version
• consistency and feature parity: the extend version as well, to match datetime support

I may just suggest we convert to float, where the float is in the units of the timedelta64

One note about this, from my (limited) experience with people who only use Python + Matplotlib as a tool in the scientific community and industry but are not very software affine, they are often very much unaware that there is a difference betweentimedelta64[ns],timedelta64[s], .... So this might lead to some confusion.

I have already created a module that patches full timedelta support into Matplotlib two years ago (see here:Timple). In case that Matplotlib decides to implement the float-only version, I'll still maintain that module for the foreseeable future. It doesn't have many users on its own (I think), but my main projectFastF1 depends on it and a large part of those users indirectly use Timple (with autolocators and autoformatters) because a large part of the data there is Timedelta values.

I agree with your summary above.

I added this to the weekly Matplotlib meeting.@theOehrly you are definitely invited to attend if you like.https://hackmd.io/l9vkn_T4RSmk147H_ZPPBA

My thinking for floats versus full datetime-like support is that it is quite easy to make timedelta64 a datetime64 by adding a starting datetime64. Then the users get all the flexibility of our datetime64 support with no added code.

td = np.arange(100, dtype='timedelta64[s]')dt = td + np.datetime64('2000-01-01')

Matplotlib can't do that automatically, because we don't know when the user started their experiment. The difficulty with this in the past has also been that some timedeltas are indeed nanosecond scale, and because we drop from datetime64 to datetime, we lose the resolution.

The advantage of co-ercing float by default is that it is the users getsomething rather than an error. If their data is indeed at the nanoscale, they need never mess with our datetime code. Changing units is also straight forward, if a bit annoying, and is the way users could coerce to float anyhow:

td = td / np.timedelta64(1, 'h')

@jklymak I should be able to participate in the meeting. And I'll try to give this some more thought until then.

For now, I have only rebased onto the current main. Nothing else has been changed so far.

Also, here's the answer to one of the questions that came up during the meeting, about how this is handled by Pandas at the moment. It was noted that their implementation for timedelta plotting is very concise. Here's the reason:

• Pandas does not use nice tick intervals for Timedelta

  Click to show

  

• How their implementation works:

  • All timedelta-like types are already converted topandas.Timedelta in the
    DataFrame or Series. Therefore, their plotting code only needs to handle this
    one data type for timedelta-like values.
  • Pandas passes the timedelta values to matplotlib as numpy arrays with
    dtypetimdelta64[ns].
  • Matplotlib'sAutoLocator is used which explains the bad tick locations.
  • Pandas registers a custom Timedelta formatter that creates a tick string as 'D days HH:MM:SS.F'.
    Hours, minutes, seconds are always shown. Days are only shown when necessary.
    Microseconds are only shown when necessary. The formatter automatically determines the number of
    significant digits that needs to be shown. The formatter is not adjustable by
    the user.

  Overall, their implementation is not as capable or adjustable as the implementation we'd be aiming for.

• Important: Adding our new implementation breaks pandas' Timedelta formatting.
  It doesn't throw an error, but the tick labels are incorrect. Here we might need to
  coordinate with Pandas, given the size of their userbase.

@jklymak update on the progress of this PR

The implementation of all features is done. As far as features are comparable, there is basically feature parity with the datetime support. Tests are all done, from my point of view. Coverage is near 100%. This part is ready for review. If you have time, you can or someone else can already have a look at the code and tests.

Documentation is still a bit of a work in progress. I've started to restructure the whole documentation formatplotlib.dates slightly. Some feedback here would be appreciated.
All classes/methods are documented at the moment, but there is still some room for improvement. Some small examples in the docstrings still need to be added. Also, I will go over everything again and maybe improve how some things are written. There are no gallery examples so far, this still is TBD.

I haven't added any type hints so far. Type hints through.pyi files seem to be (fairly) new in Matplotlib? Most modules seem to have a.pyi file, but notmatplotlib.dates. Can you give me some guidance here on what to do/what is expected? Should I add type hints for the whole module?

String formats for theConciseTimedeltaFormatter maybe still need to be discussed. I'm not entirely happy with the intuitive readability of the labels. Given that this is quick to change at any point, I'd wait discussing this until I have created a gallery example that shows off this formatter. Then there will be a visual example based on which we can discuss readability of the labels.

The partially custom formatting codes for timedelta are done. They are a subset with extensions of the standard datetime formatting codes. The supported codes are fully documented, with notes where they differ from the datetime standard. Some feedback here is appreciated as well.

We discussed that locating ticks on weeks is not desired in most cases, and that we will stick with days as the largest unit. Accordingly, I've removed weekly locating from the auto-locators. It is still available to be manually configured through the manualTimedeltaLocator because why not. This way, a user can still easily get weekly ticks when actually desired.

It was suggested, that the timedelta support is marked as preliminary/experimental part of the API for now. How would you like this to be represented in the documentation? Should I add a..note:: or..warning:: for every new function and class? I fear that this might be a bit heavy visually. On the other hand, we wouldn't want this to be easily missed by a user. Is there a better alternative?