Did you consider using attributes instead of a custom language lexer/parser for TEST blocks?

Alain Frisch (2016/07/11 07:59 -0700):

Did you consider using attributes instead of a custom language
lexer/parser for TEST blocks?

Well, not really so far, to be honest.

I can see two questions:

1. Are attribute trustworthy enough compared to lex/yacc? I mean,
   lex/yacc have been used to compile the compiler, so they are know to
   work, or at least it is likely that they work correctly. Can the same be
   said about attributes?

Second, I have to say I am not familiar with attributes. Thus, I don't
know whether they would provide the same level of freedom in the form of
the TEST block. At the moment the block can contain tst trees, with the
possibility for each test to define its own environment variables. I
don't know whtehr that would be doable with attributes, at which cost
for the developer and for users. And I don't know either what the mpact
would be on ocamltest's overall architecture.

After reading the readme, I am not sure which tests can be converted to the ocamltest format.
Am I wrong to think that none of the tool-related (e.g. testsuite/tests/tool-ocamldoc-open ) tests, nor the tests based on expect_test (e.g. testsuite/tests/typing-gadts) , nor any test based on Makefile.okbad (e.g. testsuite/tests/typing-recmod), nor any specific Makefiles can be converted as for now?

If I understand correctly all specific makefiles would need to be rewritten within ocamltest (i.e. a new compiler-specific tool), and all tests would need to be given a specification in TSL, a new compiler-specific DSL.

This seems to add quite a bit of overhead compared to the existing makefile-based situation. Could you comment in more detail what would be the advantages of such transition?

Florian Angeletti (2016/07/11 14:31 -0700):

If I understand correctly all specific makefiles would need to be
rewritten within ocamltest (i.e. a new compiler-specific tool),

You are correct, except for the "compiler-specific" aspect. The tool has
indeed been written with the idea that it could be re-used by other
projects with similar needs, especially in the OCaml ecosystem.

Not sure this reusability aspect has been reached yet, but it is at
least a target.

and all tests would need to be given a specification in TSL, a new
compiler-specific DSL.

That's correct, too.

This seems to add quite a bit of overhead compared to the existing
makefile-based situation.

I think there are at least a few developers who may have a different
opinion on this. It indeed seems that the current makefile-based
approach is not too easy to maintain or to extend.

Could you comment in more detail what would be the advantages of such
transition?

I'd say having all the logic in one place, having it written in OCaml
rather than Makefile, having an easier to debug system. For instance
with the current system it is rather hard to figure out what goes wrong
when a test fails: the commands used to produce it are not logged, the
final binary, "program", will in general be overwritten by the next test
(except for the last test in a directory). And even getting to such
a result requires some hacking when adding tests, especially when
the tests are in a new directory.

What's the plan for Windows? The AppVeyor pass is a fake: see lines 5366-5441 of the log.

Why is this written in C when we have a perfectly fine functional language with yacc and lex to use?

Goswin von Brederlow (2016/07/12 01:24 -0700):

Why is this written in C when we have a perfectly fine functional
language with yacc and lex to use?

There is only the part that runs programs that has been written in C. It
supports redirections, timeouts and things like that. As
ocamltest/README explains, it is not possible to use the Unix library at
this stage because it has not been tested yet.

I understand the desire to reduce the "trust base", but well, Unix has been tested for years and if some bad regression is introduced, it is likely to break tests, not make them succeed silently, so I don't see it as a real issue to rely on Unix in the testsuite. In a sense, the new code in the testsuite (C or not) will not be more tested than Unix, so there is no reason to expect it to be more trustworthy than Unix.

Making the testsuite more portable and easier to build seems more important at this stage.

Alain Frisch (2016/07/12 01:37 -0700):

I understand the desire to reduce the "trust base", but well, Unix has
been tested for years and if some bad regression is introduced, it is
likely to break tests, not make them succeed silently, so I don't see
it as a real issue to rely on Unix in the testsuite. In a sense, the
new code in the testsuite (C or not) will not be more tested than
Unix, so there is no reason to expect it to be more trustworthy than
Unix.

I definitely agree with this. And while I was writing this code it felt
odd, I couldn't convince myself that this way was the right one.

But would it really be possible / simpler to achieve what run is
doing in pure OCaml?

But would it really be possible / simpler to achieve what run is doing in pure OCaml?

I did not check, but I do not see upfront why a "testsuite driver" would depend on features not available in Unix (and even in the subset available under Windows). Basically, you need to launch processes and monitor (with Unix.select) their stderr/stdout and get their exit code; anything else?

I disagree that the Unix interface is well tested, esp. its Win32 implementation. (While working on PR#650 I noticed a number of places where the behavior is not obvious to me and I don't know if anyone exercised them ever.) Plus, the "launch external process with a timeout" primitive that ocamltest needs would be nearly impossible to implement under Win32 using only what is available in the OCaml Unix interface. So, I think this primitive must be implemented in C anyway.

Plus, the "launch external process with a timeout" primitive that ocamltest needs would be nearly impossible to implement under Win32 using only what is available in the OCaml Unix interface.

Can you elaborate?Unix.kill works under Win32 for killing a process. The timeout would be managed by the driver (that would rely onselect I guess).

@alainfrisch: What is needed here is roughlywaitpid with a timeout.select won't help as it selects on file descriptors, not process ID. The Win32 API accessed directly from C provides the desired functionality with a single system call (WaitForSingleObject). Restricting ourselves to the Unix API of OCaml, the only solutions I see involve signals (SIGALRM interruptingwaitpid or SIGCHLD interruptingsleep) orfork, none of which is implemented under Win32. The best I could do is "semi-active polling":waitpid withWNOHANG, then sleep for a short while, and repeat. Not very satisfying.

At any rate, I think this question "in which language is the process launcher implemented" is inessential. I'm more concerned about what needs to be done to support the full variety of tests that we have today or will need shortly.

I think this kind of primitive should still be used from OCaml, possibly after being added toUnix. Currently I cannot really use some high-level subprocess-related functions fromUnix because I need the PID, a timeout and the return code along with stdout/err (no way to get all this).

expect_test wasn't capturingFormat.{std,err}_formatter, so the output of#-directives was going to the terminal. I changed this in703d786

Under unix a simple fix would be to keep a pipe to the process open and select on it. When the process dies the pipe gets closed and select returns. But windows has no pipes, right?

Thanks alot@diml for your prompt and helpful response!

@mvrn: Windows very much has pipes! But passing the handles to the child processes is much clunkier than on Unix. It would probably be easier to co-opt selecting on one of the standard handles instead, if that can be made to work?

@shindere: it's the AppVeyor CI log from the "All checks have passed" section. The initial fix is probably simple - it's just thatMakefile.nt doesn't build ocamltest. I'd love to be offering more pragmatic support at this stage, but there're only so many hours in the day - but I would think you will have a much better time getting Windows working at the "60 tests are ported stage" and going from there than waiting until 500 tests are ported and then trying to do it!
For an easier start, just using one of the mingw ports keeps the build-system gcc-like but forces you not to rely on Posix/libc features. Getting the msvc ports to work is usually then just a matter of adapting command/filename invocations.

Currently, the AppVeyor CI machine seems broken (see for examplethe fma build), and I have the impression that this is ocamltest-related:

make[3]: Entering directory '/cygdrive/c/projects/ocaml/ocamltest'cl -nologo -O2 -Gy- -MD -WX -D_CRT_SECURE_NO_DEPRECATE -DCAML_NAME_SPACE -DUNICODE -D_UNICODE -DWINDOWS_UNICODE=1 -I../byterun -DCAML_INTERNALS  -c run_win32.crun_win32.crun_win32.c(61): error C2220: warning treated as error - no 'object' file generatedrun_win32.c(61): warning C4133: 'function': incompatible types - from 'const char *' to 'LPCWSTR'run_win32.c(64): warning C4133: 'function': incompatible types - from 'char *' to 'LPWSTR'run_win32.c(66): warning C4133: 'function': incompatible types - from 'char **' to 'LPWSTR *'run_win32.c(88): warning C4133: 'function': incompatible types - from 'const char *' to 'LPCWSTR'run_win32.c(91): warning C4133: 'function': incompatible types - from 'char *' to 'LPWSTR'run_win32.c(93): warning C4133: 'function': incompatible types - from 'char **' to 'LPWSTR *'run_win32.c(53): warning C4133: 'initializing': incompatible types - from 'char [5]' to 'LPCTSTR'run_win32.c(140): warning C4133: 'function': incompatible types - from 'const char *' to 'LPCWSTR'run_win32.c(157): warning C4133: 'function': incompatible types - from 'const char *' to 'LPCWSTR'run_win32.c(250): warning C4133: 'function': incompatible types - from 'char *' to 'LPCWSTR'run_win32.c(251): warning C4133: 'function': incompatible types - from 'char *' to 'LPWSTR'make[3]: *** [Makefile:151: run_win32.obj] Error 2make[3]: Leaving directory '/cygdrive/c/projects/ocaml/ocamltest'

Is this a known issue?

ocamltest needs to be adapted to the Unicode changes introduced by#1200; I have a PR almost ready with the changes. I am also investigating some kind of quoting issue (not sure what exactly yet) withocamltest andflexlink under Windows that cause the tests to fail.

Will submit a PR as soon as it is ready.

@shindere seemsmake one is not working properly on ocamltests based test now . I am ontrunk branch andmake one DIR=tests/typing-modules does not run anything.

This is reproducible on both my RedHat Linux machine with GNU Make 4.2.1 and my 10.12 OS X machine with GNU Make 3.8.1.

Seems the fix should be installedhere

My attempted fix only works on my OS X machine and does not work on Linux. I don't know why and my Makefile writing skill is bad. I would appreciate if you can look into it.

make promote is not working either. We should also updateHACKING.adoc undertestsuite

Yep, neither is working with ocamltest. If you look at#1530,make one is not supposed to work with the new tests for some reason.#1519 also has some discussions about howmake promote doesn't work, although it's actually not just expect-test, even tests-with-reference-files are broken in this way.