- Notifications
You must be signed in to change notification settings - Fork1.5k
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
One of the things I have problems with is too much info. This is why I am constantly updating my first post, it is an attempt to include a summary of all thing in this discussion. (so you only have to read the first post). The problems with the way pyscript handles VFS now.Problem 1 It is hard to load files into the VFS.Suppose I want to create a library on pyscipt.com. This is what I was I created a directory myLib and in that directory I have 2 python scripts. ls myLib
now suppose I have a main.py file in the top level directory with contents
You have to edit my pyscript.toml config and put these
as you can see this is problematic (as a file grows more and more files are Problem 2 The files are not persistent.If I wanted to make a open source pyscript editor like the one found on pyscript.com, I would some how have to store my files; currently, there no easy way to do this. Even though emscripten have a mount optionfor persistent-data there no documented way to access this api. Problem 3 You can not modify files during run time though the JavaScript API.Again, use the example from problem 2 ; If I am going to create an open source editor that support git the library fromisomorphic-git require the node like fs api. Problem Summary
Solution and work aroundsProblem 1 (It is hard to load files into the VFS) partial solutions(At the time of this writing 2-22-25 there is an export option that allows py-script.com to zip up you project it says this)
you can take export.zip and put it in the VFS so you only have to extract it once. Use the example above put something like this in your pyscript.toml
problem 2 (The files are not persistent )partial solution 2you can use problems with this approach
Current status
Key take away
I been doing a lot of thing that require modifying the files on the vfs some more examples
|
BetaWas this translation helpful?Give feedback.
All reactions
Replies: 9 comments 11 replies
-
mounting and reaching thos files would assume that you know AOT all files paths because the server won't likely give us a list of files to crawl so it would be a poormountFS experience, imho, but the thing is,PyScript gives you allFS primitives to created directories and fetch files as binary into those. True that we don't expose an explicit helper/utility but that's all we do internally on bootstrap via theconfig ... as summary: would a basic API such as About persisting files viaGitHub, I have successfully usedgist-fs to do that but the name of the gist and/or the token must be kept as private as possible, which is the common bummer of theWeb, something I've started working on but not solved yet to date. If you carefully avoid potential 3rd party malicious script or users out there to see and retrieve your authentication token though, you can go pretty far with that and use that asjs_module in yourPyScript project. |
BetaWas this translation helpful?Give feedback.
All reactions
-
forgot to mention we also already have the ability to unzip or un-tar.gz archives out of the box viaconfig and files, so I wonder why that wouldn't work for you 'cause we use it already without issues (buttar.gz is preferred overzip simply because it doesn't need 3rd party libraries). |
BetaWas this translation helpful?Give feedback.
All reactions
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
@WebReflection. The short answer yes,I am getting really confused , I think you have misinterpreted what I was saying. The Config files doesn't work during run time.I might have used mount wrong. I was thinking, It does what files does in your config . I renamed it mount because it sounded better and that's how zfs handles it ; but save would be a better name and adding a third perimeter would allow it specify how to store the files gotten by the server and how to handle it in your pyscipt-vfs(stored in webstorage ,vfs , or other place (a SQL server maybe). I am only (at the moment)only think one way fetching data from a server I wasn't even think about pushing it back to the server
Since all the files are in a zip we only deal with 1 file we are just decide how and where where extracting it at run time. What you suggested for an API probably would work if handles , zip ,bzip, someways it would be nice if it could handle iso but that probably not worth your time.. I thought you could add a third perimeter on how to store it , index.db chrome fsapi etc. Something you should know right now. That you might have missed that Is very important is I am zipping my whole project up and extracting it through your file config option in the lib directory This in theory should get around having to put each python file i am using in my pyscript-config.toml( or whatever is called). I couldn't find anything mentioning on dealing with the file system in your user guide. I know pyodide does have some documention but I am using micropyhton. You may have it somewhere documented but this type of thing should really be in you user guide, and as you said you already have the capability I just want to expanded a little bit and have it documented. |
BetaWas this translation helpful?Give feedback.
All reactions
-
so here it's me being confused ... as I don't fully understand what you are asking but please note that the virtual filesystem is provided byEmscripten, notpyodide normicropython, it's always the sameVFS and you could have it also available in JS viahttps://github.com/WebReflection/fs or other ports, as example. About the API, the exposed VFS and Python normal ways to deal with a file system are the same, it's the unzipping that is maybe missing out of the box (requires a library) buttar.gz is available and usable natively in both Pyodide and MicroPython, yet we don't provide a simplification of it over the network. If I might: are we saying you'd like to just grab a zip and extract it at runtime over your virtual MicroPython or Pyodide virtual FileSystem? If that's the case, I don't see why we shouldn't provide such utility that works also with regular files ... as in move or fetch_file or something to eventually discuss and shape while all internals are already capable of doing that. |
BetaWas this translation helpful?Give feedback.
All reactions
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
so here it's me being confused ... as I don't fully understand what you are asking but please note that the virtual filesystem is provided byEmscripten, not pyodide nor micropython, it's always the same VFS and youcould have it also available in JS viahttps://github.com/WebReflection/fsor other ports, as example.I was not aware ofhttps://github.com/WebReflection/fs (I'm going to haveto look at that) but I am aware that pyodide and micropyhton usesemscripen( Wich is a build tool with an API) but I don't know how toaccess emscripen API. I only said py-script VFS to be clear I am talkingabout the virtual file system py-script uses I could have said emscripenVFS If I might: are we saying you'd like to just grab a zip and extract it at runtime over your virtual MicroPython or Pyodide virtual FileSystem? Ifthat's the case, I don't see why we shouldn't provide such utility thatworks also with regular files ... as in move or fetch_file or something toeventually discuss and shape while all internals are already capable ofdoing that.yes ,you may (it not a problem I am more worried about you getting annoyedby me.)Basically yes that exactly what I want but with more control and moreoptions (if you don't mind). It would be nice to be able to use the API tomake the home directory be stored in web storage or using the chromefilesystem API you could even make it be able to use Dropbox or anotheronline storage. Those are just ideas maybe not doable but mainly I justwant to fetch a file from a url and extra it to the emscripen (whatever)VFS which is what file is doing in py-script config. I should havegiven you an example so I'll do that now (I update my original commit it there now.)I updated this for better readability(I hope). |
BetaWas this translation helpful?Give feedback.
All reactions
-
heh ... let's take one step at the time ... the OPFS is something that needs investigation but we want to provide that ability sooner than later, remote services such as Dropbox are an entirely different topic as just connecting to those is problematic if you need to store credentials in a config file ... we don't want people to store private fields or credentials in their config, we offer aSECRETS like reverse-proxy in PyScript .com but even there, we need to be careful about what's possible and what isn't ... so, back to this request that The persistent part of affair would probably come later but persistent files at runtime might be more problematic to address (the config is a one-off thing, runtime behavior is a different beast, we already saw that with the recentmicropip locked/frozen cache too). Let's see how this moves forward, but I think as long as we are aligned to what's possible, what's not secure, and what's desirable, I think this is a good feature request 👋 |
BetaWas this translation helpful?Give feedback.
All reactions
👍 1
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
Thank you I would be looking forward to whatever you decide. I only mentioned things like Dropbox ,git client just to keep that in mind while design the API. Actually with above in place I, think we probably could make those a plugin but as you said one step at a time. The export function does indeed do something at least as yesterday with out it I would not be able to zip my project on py-script.com. I found it in the config of a new created project the comment above if says something to the like of .... and that it , maybe I shouldn't have said undocumented, it sort of documented,I can't expect much from beta. very interesting anyways, I don't think it meant to work like I am using it but it is very handy. |
BetaWas this translation helpful?Give feedback.
All reactions
-
can you please tell me where that |
BetaWas this translation helpful?Give feedback.
All reactions
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
also, we cannot keepDropbox in mind with current API because we have no way, or mean to, expose users' credentials out there ... that API is going to be different and not necessarily a 1:1 PyScript API thing as it would require different steps before ever working |
BetaWas this translation helpful?Give feedback.
All reactions
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
I Am not concerned about Dropbox. The export option is not in the files section so it is probably not in the py-script stack (I had my config written wrong I fixed it above now ). the export probably comeing from the ide from py-script.com. Py-script allows to get extra arguments from there config (at first I thought it might have been from the stack) . I found the export option on the first line when starting a new project. I tried porting my code to gitlab and it doesn't do anything there but on py-script.com it puts an export.zip file in my project. Could you give a link to the code freeze bug you had talked about I would mind finding out more about what you are talking about.
I want to spend just amount of time understanding you (if not more) than you spent on me. I would think that type of thing would not have been possible with the way you use promises.
I update this for better readability. |
BetaWas this translation helpful?Give feedback.
All reactions
-
btw, from the Pyodide API it looks like they havea persistent FileSystem mechanism that works only in Chrome so far, but might give you both an FS API and persistency. It's beyond the Python API and it's exposed via the JS API nly but that might be our starting point to have persistent FS inPyScript too. |
BetaWas this translation helpful?Give feedback.
All reactions
-
@WebReflection .I did not say a proper Thankyou last time for spend so much time I greatly appreciate your patience and support I know I am hard to deal with and value all the links you gave me. |
BetaWas this translation helpful?Give feedback.
All reactions
-
I want to turn this into an issue but I would like to know what i would need to do this correctly yet . That was the main reason for the redo on the first commit. If anyone does work on this can they give me a link or let me know I would like to try all betas and keep an eye on the progress. |
BetaWas this translation helpful?Give feedback.
All reactions
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
Tested a persistent fs today using opfs and it worked well with micropython on osx(chrome, firefox, safari), android(chrome) and ios(safari). Just aded a proposal for it and saw that post. It will be great to have this as official module in pyscript. |
BetaWas this translation helpful?Give feedback.
All reactions
❤️ 1
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
i am an writer not an developer and needed to dig a little deeper intopyscript to find |
BetaWas this translation helpful?Give feedback.
All reactions
-
@Ridensium nice job! We'd love feedback on how we could make this easier (docs?). |
BetaWas this translation helpful?Give feedback.
All reactions
-
@ntoll thanks, will give feedback sure. Just want to make some tests. While plaing with opfs saw that its file read/open is 100 times faster than the native one in micropython (for 5mb text file 0.035s vs 2.27s on osx/safari for example) which is really worth it to be considered for use apart from the persistence. As well wanted to push both interpreters up to the limits. Check this out: pyodide interpreter made with 10mb chunks 5010.0 Mb file in opfs for 2.7s ;) yep 5Gb indeed |
BetaWas this translation helpful?Give feedback.
All reactions
👍 1
-
I would like to say, I always love doc and and very untechnical so anything you write i would like to read. I am also very excited to see the work you guys are putting into this. There a lot that is confusing and i have some suggestions. To give you some example there's really about 3 type of files in the file systems files on the server, file and cookie storage on the client sandbox that not visible outside of the browsers (if i am not mistaken) and the third client sandbox directory that's visible out of the browser.. The current api documention does really talk about it that well at least for me but then again I have that type of trouble. The thing with the persistent file system is also confusing how It can work on many browsers but the file system API which is part of does not. |
BetaWas this translation helpful?Give feedback.
All reactions
-
I see it was discussed already in here, well, I have same Nicholas questions aroundhow can we improve or rather what is it that is missing in our case ... but maybe I need to just look closer at your demo? |
BetaWas this translation helpful?Give feedback.