Thank you for submitting this pull request. We really appreciate you spending the time to work on SurrealDB. 🚀 🎉

What is the motivation?

See#1652. Files are a big part of data, and it's something we'd like to offer with a certain capacity in SurrealDB. This PR aims to be a first start in building an ecosystem of tools around files, whilst also building the foundation of features to come in the future.

What does this change do?

This PR adds a couple new features and changes regarding file support:

• Allows you to define a "Global Bucket".
• Allows you to define a database-level bucket, using either the global bucket, or specifying a custom "backend", such as in-memory or a file system backed bucket.
• Introduces File pointers, to interact with files stored inside a bucket:
  • f"bucket:/some/key/to/a/file.txt"
  • f"bucket:/some/key/with\ escaped"
  • f"bucket:/some/key".put(b"00aa")
  • f"bucket:/some/key".get()
  • The file pointer is also introduced in the JS scripting functions as aFile class (which takes a bucket name and key inside the constructor method). It also implements a.key() and.bucket() method on the class, and can be passed to the file functions as an argument.
• Introduces byte strings. The current way of interacting with bytes value, and how they are represented, is confusing and a lot of conversions between bytes and other value types are missing.
  • b"0099aaff" - A "b" prefixed string, similar to UUIDs and Datetimes, allows you to create a<bytes> value using a HEX string.
  • Bytes are also formatted this way
  • Conversions between strings <-> bytes, and between arrays <-> bytes are all implemented now.
• Introduces functions to work with file pointers:

  • file::bucket(file)

    • Returns:string
    • Idiom:<file>.bucket()

  • file::key(file) -> string

    • Returns:string
    • Idiom:<file>.key()

  • file::list(T (string), option<ListOptions>)

    • ListOptions:{ prefix: option<string>, start: option<string>, limit: option<int> }
    • Returns:array<{ size: int, file: file<T>, updated: datetime }>

  • file::put(file, string | bytes)

    • Idiom:<file>.put(string | bytes)

  • file::put_if_not_exists(file, string | bytes)

    • Idiom:<file>.put_if_not_exists(string | bytes)

  • file::get(file)

    • Returns:bytes
    • Idiom:<file>.get()

  • file::head(file<T>)

    • Returns:{ size: int, file: file<T>, updated: datetime }
    • Idiom:<file<T>>.head()

  • file::exists(file)

    • Returns:bool
    • Idiom:<file>.exists()

  • file::delete(file)

    • Idiom:<file>.delete()

  • file::copy(file, string)

    • Idiom:<file>.copy(string)

  • file::copy_if_not_exists(file, string)

    • Idiom:<file>.copy_if_not_exists(string)

  • file::rename(file, string)

    • Idiom:<file>.rename(string)

  • file::rename_if_not_exists(file, string)

    • Idiom:<file>.rename_if_not_exists(string)
• Additional functions:
  • type::file(T (string), string) (bucket, key)
    • Returns:file<T>
  • type::string_lossy(bytes)
    • Returns:string

Security

• For local file backends, every folder which you want to point a file bucket to, needs to be allow-listed in theSURREAL_FILE_ALLOWLIST environment variable. The file backend implementation considers any nested path listed within that environment variable to be allowed.
• TheSURREAL_FILE_ALLOWLIST option now no longer filters out folders which do not yet exist. Instead, it is up to the implementation which uses this list to do that. The function already used to check if a path is allowed already does canonalization, so existing implementation are still secure.

What is your testing strategy?

Added tests for basic file operations. Further tests to be added in a next PR

Is this related to any issues?

• ClosesFeature: Support for storing large binary data / files #1652

Does this change need documentation?

• Needs documentation

https://github.com/surrealdb/docs.surrealdb.com/tree/files

Have you read the Contributing Guidelines?

• I have read theContributing Guidelines