This is still fetching 500 entries of history on each cell execution. I just tried this locally and for my current history it is extra 180kB on the websocket wire for each cell; execute a notebook with 100 cells and you got 18MB noise on websocket. This is compared to execution and reply messages being barely 300-600 bytes.

I would suggest we only fetch history once when it is first requested by the user.

I have it fetching history on each reset so that newly run cells are included in the history without manually appending them. Also, fetching it each time allows a user to access the history from different kernels as well, though they would have to scroll through the history of the current kernel first. What if I implemented something where it only requested a few at a time, requesting more as the user scrolls farther back through the history.

I'm also now wondering if accessing different kernel histories could be considered a security risk with multiple users on jupyter_server? Would users be able to access the kernel history for other users as it is now?

It is fine to fetch say 10-20 items at a time and then maybe more if there is no overlap with what was fetched before. All I am arguing is that it should never happen on cell execution, but only when user does invokenotebook:access-previous-history-entry/notebook:access-next-history-entry. We could use cell execution as a heuristic for how many items to fetch; in pseudo-code:

class{onCellExecuted(){this._minToFetch+=1;}previousEntry(){this._ensureHistorySynced();this._setPrevious()}private_ensureHistorySynced(){lethistoryReconcliliated=false;letinitialToFetch=this._minToFetch+5;lethistoryChunk=this._fetch(initialToFetch);consthistoryChunks=[];while(!historyReconcliliated&&historyChunk.length>0){if(historyChunkoverlapsthis._historyCache){historyChunks.push(historyChunk[overlapStart:]);historyReconcliliated=true;}else{// this assumes no execution in the meantimehistoryChunks.push(historyChunk);historyChunk=this._fetch(10);}}this._historyCache.push(...reversed(historyChunks));this._minToFetch=0;}private_minToFetch=0;}

Though this will be tricky to get right. Do we really want to have history from concurrently executing kernels though?

Maybe usingsession,start andstop arguments of,history_request could help? In particular if we restrict ourselves to singlesession, the problem of cross-session history leakage would be eliminated (but then the history would be more limited).

Yeah I was on the fence on whether or not to include multiple kernel histories or just the history of the notebook kernel, I ultimately decided to include them all since that's how the console history works. I did mostly implement the single notebook kernel history though, and that's actually why cell history is collected on execution. I couldn't figure out another way to get the kernel id number without collecting from the first history item immediately after a cell in the notebook was run, though if we go that way it should probably only call history after the first execution. So it seems to me the options are:

1. include whole history, and only collect on access commands or
2. include only history from current kernel, getting kernel number from first execution of the session

and then with either of those options it can either:

a) grab history in intervals or
b) grab n number (what we currently have)

What are your thoughts?

I would ask users who requested this feature.

@rsaim, do you have an opinion on this?