README

README.adoc

capnp-ocaml===========This is an http://ocaml.org[OCaml] code generator plugin for thehttp://kentonv.github.io/capnproto[Cap\'n Proto] serialization framework.This plugin is roughly feature-complete for the Cap\'n Proto basic serialization.For RPC support, see <https://github.com/mirage/capnp-rpc>.Design Notes------------The generated code follows the general Cap\'n Proto convention of using theserialized message format as the in-memory data structure representation. Thisenables the "infinitely faster" performance characteristic, but it also meansthat the generated code is not directly compatible with OCaml records, arrays,etc.The generated code is functorized over the underlying message format. Thisenables client code to operate equally well on messages stored as--forexample--OCaml `bytes` buffers or OCaml `Bigarray` (perhaps most interestingfor applications involving `mmap()`).Generated Code--------------The Cap\'n Proto types are mapped to OCaml types as follows:.Type mapping[width="50%",cols="2",options="header"]|================================================| Cap\'n Proto Type | OCaml Type| `Void`            | `unit`| `Bool`            | `bool`| `Int8`            | `int`| `Int16`           | `int`| `Int32`           | `int32`| `Int64`           | `int64`| `UInt8`           | `int`| `UInt16`          | `int`| `UInt32`          | `Uint32.t` (from https://github.com/andrenth/ocaml-stdint[`stdint`] library)| `UInt64`          | `Uint64.t` (from https://github.com/andrenth/ocaml-stdint[`stdint`] library)| `Float32`         | `float`| `Float64`         | `float`| `Text`            | `string`| `Data`            | `string`| `List<T>`         | `('a, T, 'b) Capnp.Array.t`|================================================The `Capnp.Array` module is roughly API-compatible with OCaml arrays andprovides similar performance characteristics.A Cap\'n Proto struct maps to an OCaml module.  The generated module hierarchyreflects the hierarchy of structs specified in the schema file.  For example,the Cap\'n Proto schema--------------------------------------------------------------------------------@0xbdb65202adbdc4a0;struct Outer1 {  struct Inner {  }}struct Outer2 {}--------------------------------------------------------------------------------would yield a generated module with the signature[source,ocaml]--------------------------------------------------------------------------------module type S = sig  module Reader : sig    module Outer1 : sig      type t      module Inner : sig        type t        (* Read-only accessors for Inner.t *)        ...      end      (* Read-only accessors for Outer1.t *)      ...    end    module Outer2 : sig      type t      (* Read-only accessors for Outer2.t *)      ...    end    ...  end  module Builder : sig    module Outer1 : sig      type t      module Inner : sig        type t        (* Read/write accessors for Inner.t *)        ...      end      (* Read/write accessors for Outer1.t *)      ...    end    module Outer2 : sig      type t      (* Read/write accessors for Outer2.t *)      ...    end    ...  endend--------------------------------------------------------------------------------The `Reader` and `Builder` submodules provide read-only and read/write messageaccessors, respectively.  The accessors for struct fields are constructed asfollows:Primitive Fields~~~~~~~~~~~~~~~~A struct field `foo` of primitive type will result in generation of thefollowing accessors:[source,ocaml]--------------------------------------------------------------------------------(* for field 'foo' of type Void *)val foo_get : t -> unitval foo_set : t -> unit -> unit(* for field 'foo' of type Bool *)val foo_get : t -> boolval foo_set : t -> bool -> unit(* for field 'foo' of type Float32 or Float64 *)val foo_get : t -> floatval foo_set : t -> float -> unit(* for field 'foo' of type Text or Data *)val has_foo : t -> boolval foo_get : t -> stringval foo_set : t -> string -> unit(* for field 'foo' of type Int8 *)val foo_get : t -> int(* Raise [Invalid_argument] if out of Int8 range *)val foo_set_exn : t -> int -> unit(* for field 'foo' of type Int16 *)val foo_get : t -> int(* Raise [Invalid_argument] if out of Int16 range *)val foo_set_exn : t -> int -> unit(* for field 'foo' of type Int32 *)val foo_get : t -> int32(* Raise [Message.Out_of_int_range] if not representable as int *)val foo_get_int_exn : t -> intval foo_set : t -> int32 -> unit(* Raise [Invalid_argument] if out of Int32 range *)val foo_set_int_exn : t -> int -> unit(* for field 'foo' of type Int64 *)val foo_get : t -> int64(* Raise [Message.Out_of_int_range] if not representable as int *)val foo_get_int_exn : t -> intval foo_set : t -> int64 -> unitval foo_set_int : t -> int(* for field 'foo' of type UInt8 *)val foo_get : t -> int(* Raise [Invalid_argument] if out of UInt8 range *)val foo_set_exn : t -> int -> unit(* for field 'foo' of type UInt16 *)val foo_get : t -> int(* Raise [Invalid_argument] if out of UInt16 range *)val foo_set_exn : t -> int -> unit(* for field 'foo' of type UInt32 *)val foo_get : t -> Uint32.t(* Raise [Message.Out_of_int_range] if not representable as int *)val foo_get_int_exn : t -> intval foo_set : t -> Uint32.t -> unit(* Raise [Invalid_argument] if out of UInt32 range *)val foo_set_int_exn : t -> int -> unit(* for field 'foo' of type UInt64 *)val foo_get : t -> Uint64.t(* Raise [Message.Out_of_int_range] if not representable as int *)val foo_get_int_exn : t -> intval foo_set : t -> Uint64.t -> unit(* Raise [Invalid_argument] if out of UInt64 range *)val foo_set_int_exn : t -> int -> unit--------------------------------------------------------------------------------`_get` accessors will be available in both the `Reader` and the `Builder`modules; `_set` accessors will be available only for `Builder` types.Embedded Struct Fields~~~~~~~~~~~~~~~~~~~~~~A struct field `foo` which is of struct type will result in generation ofthe following accessors:[source,ocaml]--------------------------------------------------------------------------------(* Assuming that field foo has generated type Foo.t... *)(** [has_foo s] returns [true] if field [foo] was set in structure [s]. *)val has_foo : t -> bool(** [foo_init s] initializes the value of field [foo] to the default value    for its type.    @return a reference to the content of field [foo] *)val foo_init : t -> Foo.t(** [foo_get s] gets a reference to the content of field [foo].  (For the    Builder implementation, if the field was not previously initialized    then as a side-effect this function will default-initialize the    structure and cause [has_foo s] to return [true].)    @raise Message.Invalid_message if the message is ill-formatted *)val foo_get : t -> Foo.t(** [foo_get_pipelined s] is a reference to the field [foo] in the     (possibly not yet received) struct [s]. Only available in the Reader     section. *)val foo_get_pipelined : struct_t StructRef.t -> Foo.struct_t StructRef.t(** [foo_set_reader s v] sets the content of field [foo] by making a deep    copy of the Reader-typed structure.    @return reference to the content of field [foo]    @raise Message.Invalid_message if the message is ill-formatted *)val foo_set_reader : t -> Reader.Foo.t -> Builder.Foo.t(** [foo_set_builder s v] sets the content of field [foo] by making a deep    copy of the Builder-typed structure.    @return reference to the content of field [foo]    @raise Message.Invalid_message if the message is ill-formatted *)val foo_set_builder : t -> Builder.Foo.t -> Builder.Foo.t--------------------------------------------------------------------------------List Fields~~~~~~~~~~~A struct field `foo` which is of list type will result in generation ofthe following accessors:[source,ocaml]--------------------------------------------------------------------------------(* Assuming that field foo contains values of type Inner... *)(** [has_foo s] returns [true] if field [foo] was set in structure [s]. *)val has_foo : t -> bool(** [foo_init s n] initializes field [foo] to a zero-initialized list of    length [n] (i.e. primitive types are initialized as zero, struct types    are initialized as the default value for the struct type).    @return a reference to the content of field [foo] *)val foo_init : t -> int -> (rw, Inner.t, 'a) Capnp.Array.t(** [foo_get s] gets a reference to the content of field [foo].  (For the    Builder implementation, if the field was not previously initialized    then as a side-effect this function will default-initialize the    list and cause [has_foo s] to return [true].)    @raise Message.Invalid_message if the message is ill-formatted *)val foo_get : t -> ('cap, Inner.t, 'arr) Capnp.Array.t(** [foo_get_list s] creates an OCaml list containing the content of    field [foo].    @raise Message.Invalid_message if the message is ill-formatted *)val foo_get_list : t -> Inner.t list(** [foo_get_array s] creates an OCaml array containing the content of    field [foo].    @raise Message.Invalid_message if the message is ill-formatted *)val foo_get_array : t -> Inner.t array(** [foo_set s v] sets the content of field [foo] by creating a deep copy    of list [v].  (This may result in reallocation of [foo], which may    lead to poor performance.)    @return a reference to the content of field [foo]    @raise Message.Invalid_message if the message is ill-formatted *)val foo_set : t -> ('cap, Inner.t, 'a) Capnp.Array.t ->                (rw, Inner.t, 'b) Capnp.Array.t(** [foo_set_list s v] sets the content of field [foo] from OCaml list [v].    (This may result in reallocation of [foo], which may lead to poor    performance.)    @return a reference to the content of field [foo]    @raise Message.Invalid_message if the message is ill-formatted *)val foo_set_list : t -> Inner.t list -> (rw, Inner.t, 'b) Capnp.Array.t(** [foo_set_array s v] sets the content of field [foo] from OCaml array [v].    (This may result in reallocation of [foo], which may lead to poor    performance.)    @return a reference to the content of field [foo]    @raise Message.Invalid_message if the message is ill-formatted *)val foo_set_array : t -> Inner.t array -> (rw, Inner.t, 'b) Capnp.Array.t--------------------------------------------------------------------------------Union Fields~~~~~~~~~~~~Cap\'n Proto has first-class support for union (sum) types.  These are mappedto OCaml variants in a straightforward way.  To retrieve a union value,use the generated `get` function which will return a variant specifying whichof the possible fields is present.  To set a union value, use the generated`set_foo` (or `init_foo`) functions which simultaneously set (or init) the fieldvalue and set the union discriminant.Variant constructors are generated simply by capitalizing the first letters ofthe associated union fields.  In addition, to allow forward compatibilitythe constructor `Undefined of int` is added to the variant type definition.This constructor value is returned whenever an unknown union discriminant isdecoded.Enum Fields~~~~~~~~~~~Enums map to OCaml variants in the way one would expect.  Enum fields withinstructs will lead to generation of `foo_get` and `foo_set` accessors whichwork just like the accessors for other primitive types.Additional Operations on Structs~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~In addition to field accessors, modules associated with structs alsocontain the following functions:[source,ocaml]--------------------------------------------------------------------------------(* Assuming that the struct is called Bar... *)(** [of_message m] parses message [m] to retrieve the root struct.    @return a reference to the content of the root struct    @raise Message.Invalid_message if the message is ill-formatted *)val of_message : 'cap message_t -> t(** [of_builder b] converts a read/write reference to the struct into    a read-only interface.  (Found only in the Reader module.) *)val of_builder : Builder.Bar.t -> Reader.Bar.t(** [to_reader b] converts a read/write reference to the struct into    a read-only interface.  (Found only in the Builder module.) *)val to_reader : Builder.Bar.t -> Reader.Bar.t(** [init_root ?message_size ()] constructs a new message and    initializes an instance of this struct type as the root struct    of the message.  The optional [message_size] can be used to set    the initial message size.    @return a reference to the content of the root struct *)val init_root : ?message_size:int -> unit -> Bar.t(** [init_pointer p] initializes an instance of this struct type inside    [p]'s message and updates [p] to point to the new struct. This is    useful when dealing with AnyPointer fields.    @return a reference to the content of the struct *)val init_pointer : pointer_t -> t(** [to_message s] retrieves the underlying message which is used as    the backing store for struct [s]. *)val to_message : t -> rw message_t--------------------------------------------------------------------------------Interfaces~~~~~~~~~~A struct field `foo` which is of interface type will result in generation ofthe following accessors:[source,ocaml]--------------------------------------------------------------------------------(* Assuming that field foo contains interfaces of type Foo... *)(** The caller is responsible for freeing the result. *)val foo_get : t -> Foo.t Capability.t option(** [foo_get_pipelined t] is a capability that can be used to invoke methods    on the object in the [foo] field of [t], which might not have arrived yet.    The caller is responsible for freeing the result. *)val foo_get_pipelined : struct_t StructRef.t -> Foo.t Capability.t(** [foo_set t c] sets the field to capability [c], increasing [c]'s ref-count    (i.e. the caller is still responsible for freeing [c]). *)val foo_set : t -> Foo.t Capability.t option -> unit--------------------------------------------------------------------------------Each interface `Foo` generates a module `Foo`. There will be one submodule foreach method, with `Params` and `Results` submodules for any implicit structsneeded for the method arguments and results:[source,ocaml]--------------------------------------------------------------------------------module Reader : sig  module Foo : sig    type t = [`Foo_c36d76740ee15e68]    module MyMethod : sig      module Params : sig        type struct_t = [`MyMethod_b104a8c98610c556]        type t = struct_t reader_t        val arg1_get : t -> string        [...]      end      module Results : sig        type struct_t = [`MyMethod_c6367b042fce8e87]        type t = struct_t reader_t        val result_get : t -> string        [...]      end    end  endend--------------------------------------------------------------------------------[source,ocaml]--------------------------------------------------------------------------------module Builder : sig  module Foo : sig    type t = [`Foo_c36d76740ee15e68]    module MyMethod : sig      module Params : sigtype struct_t = [`MyMethod_b104a8c98610c556]type t = struct_t builder_tval arg1_set : t -> string -> unit        [...]      end      module Results : sigtype struct_t = [`MyMethod_c6367b042fce8e87]type t = struct_t builder_tval result_set : t -> string -> unit        [...]      end    end  endend--------------------------------------------------------------------------------The generated file will also contain `Client` and `Service` top-level modules:[source,ocaml]--------------------------------------------------------------------------------  module Client : sig    module Foo : sig      type t = [`Foo_c36d76740ee15e68]      val interface_id : Uint64.t      module MyMethod : sig        module Params = Builder.Foo.MyMethod.Params        module Results = Reader.Foo.MyMethod.Results        val method_id : (t, Params.t, Results.t) Capnp.RPC.MethodID.t      end    end  end  module Service : sig    module Foo : sig      type t = [`Foo_c36d76740ee15e68]      val interface_id : Uint64.t      module MyMethod : sig        module Params = Reader.Foo.MyMethod.Params        module Results = Builder.Foo.MyMethod.Results      end      class virtual service : object        inherit MessageWrapper.Untyped.generic_service        method virtual my_method_impl :          (MyMethod.Params.t, MyMethod.Results.t) MessageWrapper.Service.method_t      end      val local : #service -> t MessageWrapper.Capability.t    end  end--------------------------------------------------------------------------------The `Client` module is for use by clients. Each method links to a *builder* forthe parameters and a *reader* for the results (in the `Service` section they arethe other way around). The client section also includes the method's globally-uniqueID. This is just a `(Uint64.t * int)` pair, but its type gives the type of theinterface and of the request and response structs. Consult your RPC library'sdocumentation for information about how to call the method.To implement a service, inherit from the generated virtual service class andimplement the virtual methods. Use the `local` function to export your serviceas a capability.Inheritance is not currently supported.Generating Code---------------You will need tohttp://kentonv.github.io/capnproto/install.html[install the Cap\'n Proto compiler].Once the Cap\'n Proto compiler and capnp-ocaml are both installed, you should beable to use `capnp compile -o ocaml yourSchemaFile.capnp` in order to generate`yourSchemaFile.mli` and `yourSchemaFile.ml`.  These modules will link againstthe `capnp` library.Instantiating the Modules-------------------------The modules generated by capnp-ocaml are functors which take the underlyingmessage type as input.In principle, messages can be stored using any underlying data structure thatsatisfies the `Capnp.MessageStorage.S` signature.  At present, capnp-ocamlcontains one implementation: `Capnp.BytesStorage` provides message storage inthe form of native OCaml `bytes` buffers, and `Capnp.BytesMessage` providesa `Message` implementation based on `BytesStorage`.  This module makes it easy toretrieve messages in a format suitable for use with file I/O, socket I/O,etc.To instantiate your code using BytesMessage, you could use the followingpattern:[source,ocaml]--------------------------------------------------------------------------------module YSF = YourSchemaFile.Make(Capnp.BytesMessage)let root_struct = YSF.Builder.Foo.init_root () in(* ... *)--------------------------------------------------------------------------------Performance-----------For certain applications, the overhead associated with OCaml functors maybe problematic. The functors may be eliminated by compiling with an flambdabuild of the OCaml compiler (e.g. `opam switch 4.04.1+flambda`) and using the`@inlined` annotation, like this:[source,ocaml]--------------------------------------------------------------------------------module YSF = YourSchemaFile.Make[@inlined](Capnp.BytesMessage)--------------------------------------------------------------------------------You can use the `ocamlopt -inlining-report` option to check that the code hasbeen inlined. It may also be a good idea to compile with `-O3` if you careabout speed.I Need to See an Example------------------------There are some simple examples in the https://github.com/capnproto/capnp-ocaml/tree/master/src/examples[examples] directory.The https://github.com/capnproto/capnp-ocaml/tree/master/src/tests[tests]and https://github.com/capnproto/capnp-ocaml/tree/master/src/benchmark[benchmark]subdirectories may also be helpful to look at.The https://github.com/mirage/capnp-rpc/blob/master/README.md[RPC tutorial] also contains many examples.Installation------------capnp-ocaml requires OCaml >= 4.02.You should be able to install capnp-ocaml withhttp://opam.ocaml.org[OPAM] using using `opam install capnp`.If you prefer to compile manually, you will need jbuilder, Findlib, and OCamlpackages `core_kernel`, `extunix`, `uint`, `ocplib-endian`, and `res`.Run `jbuilder build` to build both the compiler and the runtime library,and then use `jbuilder install` to copy them into appropriate places within yourfilesystem.Contact-------pelzlpj at gmail dot com

1. stdint>= "0.5.1"
2. res
3. ocplib-endian>= "0.7"
4. stdio
5. base>= "v0.11"
6. result
7. dune>= "2.3"
8. ocaml>= "4.08.0"

1. conf-capnprotowith-test
2. ounit2with-test
3. base_quickcheckwith-test

1. capnp-rpc< "0.2" | >= "2.0"
2. capnp-rpc-lwt>= "0.4.0" & < "2.0"
3. capnp-rpc-mirage
4. capnp-rpc-net
5. current_rpc>= "0.4"
6. DkSDKFFIOCaml_Std

None