RFC 9404 | JMAP Blob | August 2023 |
Gondwana | Standards Track | [Page] |
The JSON Meta Application Protocol (JMAP) base protocol (RFC 8620) providesthe ability to upload and download arbitrary binary data via HTTP POST and GETon a defined endpoint. This binary data is called a "blob".¶
This extension adds additional ways to create and access blobs by makinginline method calls within a standard JMAP request.¶
This extension also adds a reverse lookup mechanism to discover where blobsare referenced within other data types.¶
This is an Internet Standards Track document.¶
This document is a product of the Internet Engineering Task Force (IETF). It represents the consensus of the IETF community. It has received public review and has been approved for publication by the Internet Engineering Steering Group (IESG). Further information on Internet Standards is available in Section 2 of RFC 7841.¶
Information about the current status of this document, any errata, and how to provide feedback on it may be obtained athttps://www.rfc-editor.org/info/rfc9404.¶
Copyright (c) 2023 IETF Trust and the persons identified as the document authors. All rights reserved.¶
This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.¶
Sometimes JMAP[RFC8620] interactions require creatinga blob and then referencing it. In the same way that IMAP literals wereextended by[RFC7888], embedding small blobs directlyinto the JMAP method calls array can be an option for reducing round trips.¶
Likewise, when fetching an object, it can be useful to also fetch the rawcontent of that object without a separate round trip.¶
Since raw blobs may contain arbitrary binary data, this document definesa use of the base64 coding specified in[RFC4648] for both creating andfetching blob data.¶
When JMAP is proxied through a system that applies additionalaccess restrictions, it can be useful to know which objects referenceany particular blob; this document defines a way to discover thosereferences.¶
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14[RFC2119][RFC8174] when, and only when, they appear in all capitals, as shown here.¶
The definitions of JSON keys and datatypes in the document follow theconventions described in[RFC8620].¶
The capabilities object is returned as part of the JMAP Sessionobject; see[RFC8620],Section 2.¶
This document defines an additional capability URI.¶
The presence of the capabilityurn:ietf:params:jmap:blob
in theaccountCapabilities property of an account represents support for additionalAPI methods on the Blob datatype. Servers that include the capability in oneor more accountCapabilities propertiesMUST also include theproperty in the capabilities property.¶
The value of this property in the JMAP Session capabilitiespropertyMUST be an empty object.¶
The value of this property in an account's accountCapabilitiesproperty is an object thatMUST contain the following informationon server capabilities and permissions for that account:¶
maxSizeBlobSet: "UnsignedInt|null"¶
The maximum size of the blob (in octets) that the server will allow to be created (including blobs created by concatenating multiple data sources together).¶
ClientsMUST NOT attempt to create blobs larger than this size.¶
If this value isnull
, then clients are not required to limit the size of the blob they try to create, though servers can always reject creation of blobs regardless of size, e.g., due to lack of disk space or per-user rate limits.¶
maxDataSources: "UnsignedInt"¶
The maximum number of DataSourceObjects allowed per creation in a Blob/upload.¶
ServersMUST allow at least 64 DataSourceObjects per creation.¶
supportedTypeNames: "String[]"¶
An array of data type names that are supported forBlob/lookup
. If the server does not support lookups, then this will be the empty list.¶
Note that the supportedTypeNames list may include private types that are not in the "JMAP Data Types" registry defined by this document. ClientsMUST ignore type names they do not recognise.¶
supportedDigestAlgorithms: "String[]"¶
An array of supported digest algorithms that are supported forBlob/get
. If the server does not support calculating blob digests, then this will be the empty list. Algorithms in this listMUST be present in the "HTTP Digest Algorithm Values" registry defined by[RFC3230]; however, in JMAP, they must be lowercased, e.g., "md5" rather than "MD5".¶
ClientsSHOULD prefer algorithms listed earlier in this list.¶
{ "capabilities": { ..., "urn:ietf:params:jmap:blob": {} }, "accounts": { "A13842": { ... "accountCapabilities": { "urn:ietf:params:jmap:blob": { "maxSizeBlobSet": 50000000, "maxDataSources": 100, "supportedTypeNames" : [ "Mailbox", "Thread", "Email" ], "supportedDigestAlgorithms" : [ "sha", "sha-256" ] } } } }}¶
A blob is a sequence of zero or more octets.¶
JMAP[RFC8620] defines theBlob/copy
method, which is unchanged by this specification and isselected by theurn:ietf:params:jmap:core
capability.¶
The following JMAP methods are selected by theurn:ietf:params:jmap:blob
capability.¶
This is similar to a Foo/set in[RFC8620] in someways. However, blobs cannot be updated or deleted, so onlycreate
isallowed in the method call. Also, blobs do not have state, so there is nostate
field present in the method response.¶
Parameters¶
accountId: "Id"¶
The id of the account in which the blobs will be created.¶
create: "Id[UploadObject]"¶
A map of creation id to UploadObjects.¶
Result¶
The result is the same as for Foo/set in[RFC8620], withcreated
andnotCreated
objectsmapping from the creation id.¶
Thecreated
objects contain:¶
id: "Id"¶
The blobId that was created.¶
type: "String|null"¶
The media type as given in the creation (if any). If not provided, the serverMAY perform content analysis and return one of the following: the calculated value, "application/octet-string", or null.¶
size: "UnsignedInt"¶
The created objects will also contain any other properties identical to those that would be returned in the JSON response of the upload endpoint described in[RFC8620]. This may be extended in the future; in this document, it is anticipated that implementations will extend both the upload endpoint and the Blob/upload responses in the same way.¶
If there is a problem with a creation, then the server will return anotCreated
response with a map from the failed creation id to aSetError
object.¶
For each successful upload, serversMUST add an entry to thecreatedIds
map ([RFC8620],Section 3.3) for the request; even if the caller did not explicitly pass acreatedIds, the value must be available to later methods defined in the sameRequest Object. This allows the blobId to be used via back-reference insubsequent method calls.¶
The created blob will have the same lifetime and same expiry semantics asany other binary object created via the mechanism specified in[RFC8620],Section 6.¶
Uploads using this mechanism will be restricted by the maxUploadSize limitfor JMAP requests specified by the server, and clientsSHOULDconsider using the upload mechanism defined by[RFC8620] for blobs larger than a megabyte.¶
UploadObject¶
data: "DataSourceObject[]"¶
An array of zero or more octet sources in order (zero to create an empty blob). The result of each of these sources is concatenated in order to create the blob.¶
type: "String|null" (default: null)¶
A hint for media type of the data.¶
DataSourceObject¶
Exactly one of:¶
data:asText: "String|null" (raw octets, must be UTF-8)¶
data:asBase64: "String|null" (base64 representation of octets)¶
or a blobId source:¶
Ifnull
, then offset is assumed to be zero.¶
Ifnull
, then length is the remaining octets in the blob.¶
If the range cannot be fully satisfied (i.e., it begins or extends pastthe end of the data in the blob), then the DataSourceObject is invalidand results in a notCreated response for this creation id.¶
If the data properties have any invalid references or invalid datacontained in them, the serverMUST NOT guess the user's intentandMUST reject the creation and return a notCreated response for thatcreation id.¶
Likewise, invalid characters in the base64 of data:asBase64 or invalidUTF-8 in data:asTextMUST result in a notCreated response.¶
It is envisaged that the definition for DataSourceObject might beextended in the future, for example, to fetch external content.¶
A serverMUST accept at least 64 DataSourceObjects per create, asdescribed inSection 3.1 of this document.¶
The data:asBase64 field is set over multiple lines for ease of publicationhere; however, the entire data:asBase64 field would be sent as a continuousstring with no wrapping on the wire.¶
Method Call:¶
[ "Blob/upload", { "accountId": "account1", "create": { "1": { "data" : [ { "data:asBase64": "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABAQMAAAAl21bKA AAAA1BMVEX/AAAZ4gk3AAAAAXRSTlN/gFy0ywAAAApJRE FUeJxjYgAAAAYAAzY3fKgAAAAASUVORK5CYII=" } ], "type": "image/png" } } }, "R1"]¶
Response:¶
[ "Blob/upload", { "accountId" : "account1", "created" : { "1": { "id" : "G4c6751edf9dd6903ff54b792e432fba781271beb", "type" : "image/png", "size" : 95 } } }, "R1"]¶
Method Calls:¶
[ [ "Blob/upload", { "create": { "b4": { "data": [ { "data:asText": "The quick brown fox jumped over the lazy dog." } ] } } }, "S4"],[ "Blob/upload", { "create": { "cat": { "data": [ { "data:asText": "How" }, { "blobId": "#b4", "length": 7, "offset": 3 }, { "data:asText": "was t" }, { "blobId": "#b4", "length": 1, "offset": 1 }, { "data:asBase64": "YXQ/" } ] } } }, "CAT"],[ "Blob/get", { "properties": [ "data:asText", "size" ], "ids": [ "#cat" ] }, "G4" ]]¶
Responses:¶
[ [ "Blob/upload", { "oldState": null, "created": { "b4": { "id": "Gc0854fb9fb03c41cce3802cb0d220529e6eef94e", "size": 45, "type": "application/octet-stream" } }, "notCreated": null, "accountId": "account1" }, "S4" ], [ "Blob/upload", { "oldState": null, "created": { "cat": { "id": "Gcc60576f036321ae6e8037ffc56bdee589bd3e23", "size": 19, "type": "application/octet-stream" } }, "notCreated": null, "accountId": "account1" }, "CAT" ], [ "Blob/get", { "list": [ { "id": "Gcc60576f036321ae6e8037ffc56bdee589bd3e23", "data:asText": "How quick was that?", "size": 19 } ], "notFound": [], "accountId": "account1" }, "G4" ]]¶
A standard JMAP get, with two additional optional parameters:¶
offset: "UnsignedInt|null"¶
Start this many octets into the blob data. If null or unspecified, this defaults to zero.¶
length: "UnsignedInt|null"¶
Return at most this many octets of the blob data. If null or unspecified, then all remaining octets in the blob are returned. This can be considered equivalent to an infinitely large length value, except that the isTruncated warning is not given unless the start offset is past the end of the blob.¶
Request Properties:¶
Any of:¶
supportedDigestAlgorithms
capability)¶If not given, the properties default todata
andsize
.¶
Result Properties:¶
data:asText: "String|null"¶
The raw octets of the selected range if they are valid UTF-8; otherwise, null.¶
data:asBase64: "String"¶
The base64 encoding of the octets in the selected range.¶
digest:<algorithm>: "String"¶
The base64 encoding of the digest of the octets in the selected range, calculated using the named algorithm.¶
isEncodingProblem: "Boolean" (default: false)¶
isTruncated: "Boolean" (default: false)¶
size: "UnsignedInt"¶
The number of octets in the entire blob.¶
The size valueMUST always be the number of octets in the underlying blob,regardless of offset and length.¶
The data fields contain a representation of the octets within the selectedrange that are present in the blob. If the octets selected are not validUTF-8 (including truncating in the middle of a multi-octet sequence)anddata
ordata:asText
was requested, then the keyisEncodingProblem
MUST be set totrue
, and thedata:asText
response valueMUST benull
.In the case wheredata
was requested and the data is not valid UTF-8,thendata:asBase64
MUST be returned.¶
If the selected range requests data outside the blob (i.e., theoffset+length is larger than the blob), then the result is either just theoctets from the offset to the end of the blob or an empty string if theoffset is past the end of the blob. Either way, the isTruncatedproperty in the resultMUST be set totrue
to tell theclient that the requested range could not be fully satisfied. If digest wasrequested, anydigest
is calculated on the octets that would bereturned for adata
field.¶
ServersSHOULD store the size for blobs in a format that isefficient to read, and clientsSHOULD limit their request tojust the size parameter if that is all they need, as fetching blob contentcould be significantly more expensive and slower for the server.¶
In this example, a blob containing the string "The quick brown fox jumped overthe lazy dog." has blobIdGc0854fb9fb03c41cce3802cb0d220529e6eef94e
.¶
The first method call requests just the size for multiple blobs, andthe second requests both the size and a short range of the data for oneof the blobs.¶
Method Calls:¶
[ [ "Blob/get", { "accountId" : "account1", "ids" : [ "Gc0854fb9fb03c41cce3802cb0d220529e6eef94e", "not-a-blob" ], "properties" : [ "data:asText", "digest:sha", "size" ] }, "R1" ], [ "Blob/get", { "accountId" : "account1", "ids" : [ "Gc0854fb9fb03c41cce3802cb0d220529e6eef94e" ], "properties" : [ "data:asText", "digest:sha", "digest:sha-256", "size" ], "offset" : 4, "length" : 9 }, "R2" ]]¶
Responses:¶
[ [ "Blob/get", { "accountId": "account1", "list": [ { "id": "Gc0854fb9fb03c41cce3802cb0d220529e6eef94e", "data:asText": "The quick brown fox jumped over the lazy dog.", "digest:sha": "wIVPufsDxBzOOALLDSIFKebu+U4=", "size": 45 } ], "notFound": [ "not-a-blob" ] }, "R1"],[ "Blob/get", { "accountId": "account1", "list": [ { "id": "Gc0854fb9fb03c41cce3802cb0d220529e6eef94e", "data:asText": "quick bro", "digest:sha": "QiRAPtfyX8K6tm1iOAtZ87Xj3Ww=", "digest:sha-256": "gdg9INW7lwHK6OQ9u0dwDz2ZY/gubi0En0xlFpKt0OA=", "size": 45 } ] }, "R2" ]]¶
Theb1
value is the text "The quick brown fox jumped over the \x81\x81 dog.", which contains an invalid UTF-8 sequence.¶
The results have the following properties:¶
G1: Defaults todata
andsize
, so b1 returnsisEncodingProblem
and a base64 value.¶
G2: Sincedata:asText
was explicitly selected, does notattempt to return a value for the data, justisEncodingProblem
forb1.¶
G3: Since onlydata:asBase64
was requested, there is noencoding problem, and both values are returned.¶
G4: Since the requested range could be satisfied as text, both blobsare returned asdata:asText
, and there is no encoding problem.¶
G5: Both blobs cannot satisfy the requested range, so isTruncated istrue for both.¶
Note: Some values have been wrapped for line length. There would beno wrapping in thedata:asBase64
values on the wire.¶
Method Calls:¶
[ [ "Blob/upload", { "create": { "b1": { "data": [ { "data:asBase64": "VGhlIHF1aWNrIGJyb3duIGZveCBqdW1wZW Qgb3ZlciB0aGUggYEgZG9nLg==" } ] }, "b2": { "data": [ { "data:asText": "hello world" } ], "type" : "text/plain" } } }, "S1" ], [ "Blob/get", { "ids": [ "#b1", "#b2" ] }, "G1" ], [ "Blob/get", { "ids": [ "#b1", "#b2" ], "properties": [ "data:asText", "size" ] }, "G2" ], [ "Blob/get", { "ids": [ "#b1", "#b2" ], "properties": [ "data:asBase64", "size" ] }, "G3" ], [ "Blob/get", { "offset": 0, "length": 5, "ids": [ "#b1", "#b2" ] }, "G4" ], [ "Blob/get", { "offset": 20, "length": 100, "ids": [ "#b1", "#b2" ] }, "G5" ]]¶
Responses:¶
[ [ "Blob/upload", { "oldState": null, "created": { "b2": { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "size": 11, "type": "application/octet-stream" }, "b1": { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "size": 43, "type": "text/plain" } }, "updated": null, "destroyed": null, "notCreated": null, "notUpdated": null, "notDestroyed": null, "accountId": "account1" }, "S1" ], [ "Blob/get", { "list": [ { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "isEncodingProblem": true, "data:asBase64": "VGhlIHF1aWNrIGJyb3duIGZveCBqdW1wZW Qgb3ZlciB0aGUggYEgZG9nLg==", "size": 43 }, { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "data:asText": "hello world", "size": 11 } ], "notFound": [], "accountId": "account1" }, "G1" ], [ "Blob/get", { "list": [ { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "isEncodingProblem": true, "size": 43 }, { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "data:asText": "hello world", "size": 11 } ], "notFound": [], "accountId": "account1" }, "G2" ], [ "Blob/get", { "list": [ { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "data:asBase64": "VGhlIHF1aWNrIGJyb3duIGZveCBqdW1wZW Qgb3ZlciB0aGUggYEgZG9nLg==", "size": 43 }, { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "data:asBase64": "aGVsbG8gd29ybGQ=", "size": 11 } ], "notFound": [], "accountId": "account1" }, "G3" ], [ "Blob/get", { "list": [ { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "data:asText": "The q", "size": 43 }, { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "data:asText": "hello", "size": 11 } ], "notFound": [], "accountId": "account1" }, "G4" ], [ "Blob/get", { "list": [ { "id": "G72cfa4804194563685d9a4b695f7ba20e7739576", "isTruncated": true, "isEncodingProblem": true, "data:asBase64": "anVtcGVkIG92ZXIgdGhlIIGBIGRvZy4=", "size": 43 }, { "id": "G2aae6c35c94fcfb415dbe95f408b9ce91ee846ed", "isTruncated": true, "data:asText": "", "size": 11 } ], "notFound": [], "accountId": "account1" }, "G5" ]]¶
Given a list of blobIds, this method does a reverse lookup in each ofthe provided type names to find the list of Ids within that data typethat reference the provided blob.¶
Since different datatypes will have different semantics of "contains",the definition of "reference" is somewhat loose but roughlymeans "you could discover this blobId by looking at this object orat other objects recursively contained within this object".¶
For example, with a server that supports[RFC8621], if a Mailbox references a blob and if any Emailswithin that Mailbox reference the blobId, then the Mailbox references thatblobId. For any Thread that references an Email that references a blobId, itcan be said that the Thread references the blobId.¶
However, this does not mean that if an Email references a Mailbox in itsmailboxIds property, then any blobId referenced by other Emails inthat Mailbox are also referenced by the initial Email.¶
Parameters¶
accountId: "Id"¶
The id of the account used for the call.¶
typeNames: "String[]"¶
A list of names from the "JMAP Data Types" registry or defined byprivate extensions that the client has requested. Only namesfor which "Can reference blobs" is true may be specified, and thecapability that defines each type must also be used by the overallJMAP request in which this method is called.¶
If a type name is not known by the server, or the associated capabilityhas not been requested, then the server returns an "unknownDataType"error.¶
ids: "Id[]"¶
A list of blobId values to be looked for.¶
Response¶
BlobInfo Object¶
id: "Id"¶
The blobId.¶
matchedIds: "String[Id[]]"¶
A map from type name to a list of Ids of that data type (e.g., the name"Email" maps to a list of emailIds).¶
If a blob is not visible to a user or does not exist on the server at all,then the serverMUST still return an empty array for each typeas this doesn't leak any information about whether the blob is on the serverbut not visible to the requesting user.¶
Method Call:¶
[ "Blob/lookup", { "typeNames": [ "Mailbox", "Thread", "Email" ], "ids": [ "Gd2f81008cf07d2425418f7f02a3ca63a8bc82003", "not-a-blob" ] }, "R1"]¶
Response:¶
[ "Blob/lookup", { "list": [ { "id": "Gd2f81008cf07d2425418f7f02a3ca63a8bc82003", "matchedIds": { "Mailbox": [ "M54e97373", "Mcbe6b662" ], "Thread": [ "T1530616e" ], "Email": [ "E16e70a73eb4", "E84b0930cf16" ] } } ], "notFound": [ "not-a-blob" ] }, "R1"]¶
All security considerations for JMAP[RFC8620] apply to this specification.Additional considerations specific to the data types and functionalityintroduced by this document are described here.¶
JSON parsers are not all consistent in handling non-UTF-8 data.JMAP requires that all JSON data be UTF-8 encoded, so serversMUST only return a null value ifdata:asText
isrequested for a range of octets that is not valid UTF-8 and setisEncodingProblem: true
.¶
ServersMUST apply any access controls, such that if the authenticated user wouldbe unable to discover the blobId by making queries, then this fact cannot bediscovered via a Blob/lookup. For example, if an Email exists in a Mailbox thatthe authenticated user does not have access to see, then that emailIdMUST NOT bereturned in a lookup for a blob that is referenced by that email.¶
The serverMUST NOT trust that the data given to aBlob/upload is a well-formed instance of the specified media type. Also, ifthe server attempts to parse the given blob, only hardened parsers designed todeal with arbitrary untrusted data should be used. The serverSHOULD NOT reject data on the grounds that it is not a valid specimen of thestated type.¶
With carefully chosen data sources, Blob/upload can be used to recreatedangerous content on the far side of security scanners (anti-virus orexfiltration scanners, for example) that may be watching the upload endpoint.Server implementationsSHOULD provide a hook to allow securityscanners to check the resulting blob after concatenating the data sources inthe same way that they do for the upload endpoint.¶
Digest algorithms can be expensive for servers to calculate. Servers thatshare resources between multiple users should track resource usage by clientsand rate-limit expensive operations to avoid resource starvation.¶
IANA has registered the "blob" JMAP capability as follows:¶
IANA has registered the "unknownDataType" JMAP error code as follows:¶
IANA has created a new registry called "JMAP Data Types".Table 1 shows the initial contents of thisnew registry.¶
Type Name | Can Ref Blobs | Can Use for State Change | Capability | Reference |
---|---|---|---|---|
Core | No | No | urn:ietf:params:jmap:core | [RFC8620] |
PushSubscription | No | No | urn:ietf:params:jmap:core | [RFC8620] |
Mailbox | Yes | Yes | urn:ietf:params:jmap:mail | [RFC8621] |
Thread | Yes | Yes | urn:ietf:params:jmap:mail | [RFC8621] |
Yes | Yes | urn:ietf:params:jmap:mail | [RFC8621] | |
EmailDelivery | No | Yes | urn:ietf:params:jmap:mail | [RFC8621] |
SearchSnippet | No | No | urn:ietf:params:jmap:mail | [RFC8621] |
Identity | No | Yes | urn:ietf:params:jmap:submission | [RFC8621] |
EmailSubmission | No | Yes | urn:ietf:params:jmap:submission | [RFC8621] |
VacationResponse | No | Yes | urn:ietf:params:jmap:vacationresponse | [RFC8621] |
MDN | No | No | urn:ietf:params:jmap:mdn | [RFC9007] |
The registration policy for this registry is "Specification Required"[RFC8126]. Either an RFC or a similarly stable reference document defines a JMAP Data Typeand associated capability.¶
IANA will appoint designated experts to review requests for additions to thisregistry, with guidance to allow any registration that provides a stable document describingthe capability and control over the URI namespace to which the capability URI points.¶
Joris Baum,Jim Fenton,Neil Jenkins,Alexey Melnikov,Ken Murchison,Robert Stepanek, andthe JMAP Working Group in the IETF.¶