Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Be able to unzip archive at run time to virtual filesystem (and some idea for improvement)#2286

frog-o started this conversation inProposals
Discussion options

One of the things I have problems with is too much info. This is why I am constantly updating my first post, it is an attempt to include a summary of all thing in this discussion. (so you only have to read the first post).

The problems with the way pyscript handles VFS now.

Problem 1 It is hard to load files into the VFS.

Suppose I want to create a library on pyscipt.com. This is what I was
doing.

I created a directory myLib and in that directory I have 2 python scripts.
let's call it myFirstGoup.py and mySecGroup.py it looks something like this

ls myLib

myLib
myLib\myFirstGoup.py
myLib\mySecGoup.py

now suppose I have a main.py file in the top level directory with contents

import from mylib myFirstGoup , mySecGoup

You have to edit my pyscript.toml config and put these
two files in the config so it would look something like this

[file]

”./myLib/myFirstGoup.py” = "/lib/myLib/myFirstGoup.py"
"./myLib/mySecGoup.py" = "/lib/myLib/mySecGoup.py"

as you can see this is problematic (as a file grows more and more files are
need to add to the config)

Problem 2 The files are not persistent.

If I wanted to make a open source pyscript editor like the one found on pyscript.com, I would some how have to store my files; currently, there no easy way to do this. Even though emscripten have a mount optionfor persistent-data there no documented way to access this api.

Problem 3 You can not modify files during run time though the JavaScript API.

Again, use the example from problem 2 ; If I am going to create an open source editor that support git the library fromisomorphic-git require the node like fs api.

Problem Summary

  • It is hard to load files into the VFS
  • Files are not Persistent
  • You can not modify files during run time though the JavaScript API.

Solution and work arounds

Problem 1 (It is hard to load files into the VFS) partial solutions

(At the time of this writing 2-22-25 there is an export option that allows py-script.com to zip up you project it says this)

# Bundle your project into a zip file which can be downloaded or used
#as a module in your Anaconda Toolbox Excel Add-in

you can take export.zip and put it in the VFS so you only have to extract it once. Use the example above put something like this in your pyscript.toml

export= true
[file]
”./export.zip" = "/lib/mylib/*"
#This doesn't work on pyscript.com ( at least ) The lib directory
doesn't exist and I think it is only capable of creating One
directory deep.

  • does not work doing runtime
  • the export option is buggy and beta

problem 2 (The files are not persistent )partial solution 2

you can use

problems with this approach

  • zenfs an other library could help with this but often requires node and problematic otherwise
  • libary like gist-fs you have to be careful not to leak you personnel access token and other info
  • Persistent FileSystem only works on chrome.

Current status

  • There are people who are test the original private file system for static file system.
  • they talk about there documentation and money patching micropython.
  • they mentionedFebruary last pyscript realse

Key take away

  • The thing I am asking for is a node compatible fs for use withisomorphic-git even though it does have to be . I would just like a javascript way to put files into the VFS
  • It looks like pyscipt can do a lot already though the files section in the pyscript.toml, it has the basic functionality I need but does not run at run time.
  • I am using micropyhton and think this would be very useful.
  • I really like what zenfs can do and how much it can do with the file system. I was wondering if you could do something similar?

I been doing a lot of thing that require modifying the files on the vfs some more examples

  • I wanted to create a Linux like system in the browser, I would need to have a shell which need more control
  • I wanted to build a box86 for the browser.(There are some closed source ones but I wanted open one). I need to be able to load an x86_amd64 binary at runtime.
  • Please note I been updating my commit trying to make them more readable. I have a disability that make it hard for me to communicate properly. I want to do the best I can to make it easer for people to help me.
You must be logged in to vote

Replies: 9 comments 11 replies

Comment options

mounting and reaching thos files would assume that you know AOT all files paths because the server won't likely give us a list of files to crawl so it would be a poormountFS experience, imho, but the thing is,PyScript gives you allFS primitives to created directories and fetch files as binary into those. True that we don't expose an explicit helper/utility but that's all we do internally on bootstrap via theconfig ... as summary: would a basic API such aspyscript.files.save('/url/source.txt', '/vfs/dest/file.txt') work for you?

About persisting files viaGitHub, I have successfully usedgist-fs to do that but the name of the gist and/or the token must be kept as private as possible, which is the common bummer of theWeb, something I've started working on but not solved yet to date. If you carefully avoid potential 3rd party malicious script or users out there to see and retrieve your authentication token though, you can go pretty far with that and use that asjs_module in yourPyScript project.

You must be logged in to vote
0 replies
Comment options

forgot to mention we also already have the ability to unzip or un-tar.gz archives out of the box viaconfig and files, so I wonder why that wouldn't work for you 'cause we use it already without issues (buttar.gz is preferred overzip simply because it doesn't need 3rd party libraries).

You must be logged in to vote
2 replies
@frog-o
Comment options

@WebReflection. The short answer yes,I am getting really confused , I think you have misinterpreted what I was saying. The Config files doesn't work during run time.I might have used mount wrong.

I was thinking, It does what files does in your config . I renamed it mount because it sounded better and that's how zfs handles it ; but save would be a better name and adding a third perimeter would allow it specify how to store the files gotten by the server and how to handle it in your pyscipt-vfs(stored in webstorage ,vfs , or other place (a SQL server maybe). I am only (at the moment)only think one way fetching data from a server I wasn't even think about pushing it back to the server

mounting and reaching thos files would assume that you know AOT all files paths because the server ..

Since all the files are in a zip we only deal with 1 file we are just decide how and where where extracting it at run time.

What you suggested for an API probably would work if handles , zip ,bzip, someways it would be nice if it could handle iso but that probably not worth your time.. I thought you could add a third perimeter on how to store it , index.db chrome fsapi etc.

Something you should know right now. That you might have missed that Is very important is I am zipping my whole project up and extracting it through your file config option in the lib directory This in theory should get around having to put each python file i am using in my pyscript-config.toml( or whatever is called). I couldn't find anything mentioning on dealing with the file system in your user guide. I know pyodide does have some documention but I am using micropyhton. You may have it somewhere documented but this type of thing should really be in you user guide, and as you said you already have the capability I just want to expanded a little bit and have it documented.

@WebReflection
Comment options

I am zipping my whole project up and extracting it through your file config option in the lib directory

so here it's me being confused ... as I don't fully understand what you are asking but please note that the virtual filesystem is provided byEmscripten, notpyodide normicropython, it's always the sameVFS and you could have it also available in JS viahttps://github.com/WebReflection/fs or other ports, as example.

About the API, the exposed VFS and Python normal ways to deal with a file system are the same, it's the unzipping that is maybe missing out of the box (requires a library) buttar.gz is available and usable natively in both Pyodide and MicroPython, yet we don't provide a simplification of it over the network.

If I might: are we saying you'd like to just grab a zip and extract it at runtime over your virtual MicroPython or Pyodide virtual FileSystem? If that's the case, I don't see why we shouldn't provide such utility that works also with regular files ... as in move or fetch_file or something to eventually discuss and shape while all internals are already capable of doing that.

Comment options

so here it's me being confused ... as I don't fully understand what you
are asking but please note that the virtual filesystem is provided byEmscripten, not pyodide nor micropython, it's always the same VFS and youcould have it also available in JS viahttps://github.com/WebReflection/fsor other ports, as example.I was not aware ofhttps://github.com/WebReflection/fs (I'm going to haveto look at that) but I am aware that pyodide and micropyhton usesemscripen( Wich is a build tool with an API) but I don't know how toaccess emscripen API. I only said py-script VFS to be clear I am talkingabout the virtual file system py-script uses I could have said emscripenVFS
If I might: are we saying you'd like to just grab a zip and extract it at
runtime over your virtual MicroPython or Pyodide virtual FileSystem? Ifthat's the case, I don't see why we shouldn't provide such utility thatworks also with regular files ... as in move or fetch_file or something toeventually discuss and shape while all internals are already capable ofdoing that.yes ,you may (it not a problem I am more worried about you getting annoyedby me.)Basically yes that exactly what I want but with more control and moreoptions (if you don't mind). It would be nice to be able to use the API tomake the home directory be stored in web storage or using the chromefilesystem API you could even make it be able to use Dropbox or anotheronline storage. Those are just ideas maybe not doable but mainly I justwant to fetch a file from a url and extra it to the emscripen (whatever)VFS which is what file is doing in py-script config. I should havegiven you an example so I'll do that now (I update my original commit it there now.)I updated this for better readability(I hope).
You must be logged in to vote
1 reply
@WebReflection
Comment options

make the home directory be stored in web storage or using the chrome
filesystem API you could even make it be able to use Dropbox or another
online storage

heh ... let's take one step at the time ... the OPFS is something that needs investigation but we want to provide that ability sooner than later, remote services such as Dropbox are an entirely different topic as just connecting to those is problematic if you need to store credentials in a config file ... we don't want people to store private fields or credentials in their config, we offer aSECRETS like reverse-proxy in PyScript .com but even there, we need to be careful about what's possible and what isn't ... so, back to this request

thatexport=true you mentioned you found undocumented does nothing (IIRC) it actually messes with thefiles parsing logic ... the.zip to/path/* looks what we provide already, so if that's what you'd like to have, us exporting thatfiles mechanism so you can, at runtime, define and await, any time you like, a config like dictionary to tell it where stuff should go, I think I am 👍on that, but I'd like to hear from@ntoll here, where basically he should agree about us exposing afs API that allows people to do what we do with the config at runtime: technically doable, practically useful to me, but he might have a different opinion.

The persistent part of affair would probably come later but persistent files at runtime might be more problematic to address (the config is a one-off thing, runtime behavior is a different beast, we already saw that with the recentmicropip locked/frozen cache too).

Let's see how this moves forward, but I think as long as we are aligned to what's possible, what's not secure, and what's desirable, I think this is a good feature request 👋

Comment options

Thank you I would be looking forward to whatever you decide. I only mentioned things like Dropbox ,git client just to keep that in mind while design the API. Actually with above in place I, think we probably could make those a plugin but as you said one step at a time. The export function does indeed do something at least as yesterday with out it I would not be able to zip my project on py-script.com. I found it in the config of a new created project the comment above if says something to the like of ....
for use with the beta of Python for Excel toolbox .

and that it , maybe I shouldn't have said undocumented, it sort of documented,I can't expect much from beta.

very interesting anyways, I don't think it meant to work like I am using it but it is very handy.

You must be logged in to vote
2 replies
@WebReflection
Comment options

can you please tell me where thatexport = true is within a[files] definition in our stack/docs? I don't even know why that works, and I've implemented thefiles definition out ofPyScript so I've double checked and nothing comes up out there in the stack we use to parse files ... thanks!

@WebReflection
Comment options

also, we cannot keepDropbox in mind with current API because we have no way, or mean to, expose users' credentials out there ... that API is going to be different and not necessarily a 1:1 PyScript API thing as it would require different steps before ever working

Comment options

I Am not concerned about Dropbox. The export option is not in the files section so it is probably not in the py-script stack (I had my config written wrong I fixed it above now ). the export probably comeing from the ide from py-script.com. Py-script allows to get extra arguments from there config (at first I thought it might have been from the stack) . I found the export option on the first line when starting a new project. I tried porting my code to gitlab and it doesn't do anything there but on py-script.com it puts an export.zip file in my project.
I do a similar thing on gitlab but it has a docker like config where I have it zip the source code.

Could you give a link to the code freeze bug you had talked about I would mind finding out more about what you are talking about.

The persistent part of affair would probably come later but persistent files at runtime might be more problematic to address (the config is a one-off thing, runtime behavior is a different beast, we already saw that with the recent micropip locked/frozen cache too).

I want to spend just amount of time understanding you (if not more) than you spent on me. I would think that type of thing would not have been possible with the way you use promises.

Please note I am not a very good programmer and I am all self taught so there going to be a lot I don't understand and I don't expect you or even want you to spend a lot of time on me I am just committing to try to answer your questions.

I update this for better readability.

You must be logged in to vote
0 replies
Comment options

btw, from the Pyodide API it looks like they havea persistent FileSystem mechanism that works only in Chrome so far, but might give you both an FS API and persistency.

It's beyond the Python API and it's exposed via the JS API nly but that might be our starting point to have persistent FS inPyScript too.

You must be logged in to vote
0 replies
Comment options

@WebReflection .I did not say a proper Thankyou last time for spend so much time I greatly appreciate your patience and support I know I am hard to deal with and value all the links you gave me.

You must be logged in to vote
0 replies
Comment options

I want to turn this into an issue but I would like to know what i would need to do this correctly yet . That was the main reason for the redo on the first commit. If anyone does work on this can they give me a link or let me know I would like to try all betas and keep an eye on the progress.

You must be logged in to vote
0 replies
Comment options

Tested a persistent fs today using opfs and it worked well with micropython on osx(chrome, firefox, safari), android(chrome) and ios(safari). Just aded a proposal for it and saw that post. It will be great to have this as official module in pyscript.

https://rdsm.pyscriptapps.com/persipy/latest/

You must be logged in to vote
6 replies
@Ridensium
Comment options

i am an writer not an developer and needed to dig a little deeper intopyscript to findinterpreter in_pyscript and needed to make it usecreateSyncAccessHandle to support all the browsers insteadcreateWritable which is supported only by Cromium. Ah and give tomountNativeFS handle for the opfs, whichmdndocs promised will work everywhere, instead FileSystemDirectoryHandle sole mentioned in pyodide docs (again only Cromium), hoping that both are "handles" and got lucky. Take me full day this succer :) so for non technical will say medium difficulty, guess for you guys will be a piece of cake. But happy now that it work out especially withmicropython as saving files in opfs need to be done in a worker not in a main thread. So thats the final module:
https://github.com/Ridensium/persipy

@ntoll
Comment options

@Ridensium nice job! We'd love feedback on how we could make this easier (docs?).

@Ridensium
Comment options

@ntoll thanks, will give feedback sure. Just want to make some tests. While plaing with opfs saw that its file read/open is 100 times faster than the native one in micropython (for 5mb text file 0.035s vs 2.27s on osx/safari for example) which is really worth it to be considered for use apart from the persistence. As well wanted to push both interpreters up to the limits. Check this out:
https://rdsm.pyscriptapps.com/fwriteopenperf/latest/

pyodide interpreter made with 10mb chunks 5010.0 Mb file in opfs for 2.7s ;) yep 5Gb indeed

@frog-o
Comment options

I would like to say, I always love doc and and very untechnical so anything you write i would like to read. I am also very excited to see the work you guys are putting into this. There a lot that is confusing and i have some suggestions. To give you some example there's really about 3 type of files in the file systems files on the server, file and cookie storage on the client sandbox that not visible outside of the browsers (if i am not mistaken) and the third client sandbox directory that's visible out of the browser.. The current api documention does really talk about it that well at least for me but then again I have that type of trouble. The thing with the persistent file system is also confusing how It can work on many browsers but the file system API which is part of does not.

@WebReflection
Comment options

I see it was discussed already in here, well, I have same Nicholas questions aroundhow can we improve or rather what is it that is missing in our case ... but maybe I need to just look closer at your demo?

Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Labels
None yet
4 participants
@frog-o@ntoll@WebReflection@Ridensium

[8]ページ先頭

©2009-2025 Movatter.jp