Repository files navigation

When judging student performance, it is useful to make both small,composable, quantitative judgements, and to collect qualitativeremarks. This makes both spreadsheets and mere text-files ill-suitedfor marking student work. Althoughorg-mode cansolve this problem to a great extent, it becomes a heavy tool in thelight of having to mark hundreds of students in a distributed,decentralized fashion. With org-mode, everything is in one file, whileglobal, intra-student statistics are not needed until all the studentshave been fully marked.

1. One, or several, human-readable/editable file(s)per student.
   • This design goal has had the unfortunate consequence thatteaching assistantsskip installingremarks, and merelyedit the source files. This works, but isn't the intended methodof use.
   • However, having it be human-readable, has also had the fortunateconsequence that reading remarks files has always been simple.Even withoutremarks installed, it is feasible to runcross-student statistics using standard Unix-like command-lineutilities.
2. git-friendly file format.
3. Export options to spreadsheet-formats.
4. Synchronization options with Dropbox and/or Google Drive.

Goal 4 is not necessarily related toremarks, but is related to markingstudent work with external examiners, who are not always willing to use moreexplicit version-control systems, such as git.

User features:

• remarks valid [<file>] checks that the file system structure iswell-formed. For instance, checks that all total point sumsare correct.
• remarks check [<file>] likevalid but also ensures that all points have been given and sums correctly.
• remarks show [<file>] checks the file system structure as above, and showsthe overall judgement for each given argument.
• remarks feedback [<file>] prints the accumulated feedback.
• remarks summary [--depth <depth>] [<file>] checks and summarizes the points.Depth 0 (default) lists just the top-level judgements.
• remarks pending [--depth <depth>] [<file>] shown the corrections that has notbeen completed. Can be cut at a given depth; depth 0 lists just the top-level judgements.
• Exporting remarks files:
  • remarks export [--format "<format>"] [<file>] exports corrections to a semicolon separated list. The format is a semicolon separated string of properties.
  • remarks exportHTML [<file>] exports all corrections to a dynamic html-table.
  • remarks exportResults [<file>] export results to a table with result of each sub question.
  • remarks exportMD [<file>] exports remarks to a MarkDown file
  • remarks exportPdfMark [<file>] export remarks to format that can be inserted as comments in PDf with GhostScript.

Developer features:

• remarks parse [<file>] parses the given files and shows their ASTs.

remarks is written in Haskell. If you are new to Haskell, we can recommendinstalling eitherHaskell Platform orTheHaskell Tool Stack. Followthe instructions below to then installremarks.

remarks is onHackage, so youcan useCabal:

$ cabal install remarks

If you clone the repository, ordownload thesources, you get two furtheroptions:

• If you are using the purely functional package managerNix, you can do this:

  $ nix-build

  This will create a symlinkresult, pointing to the directory in your Nixstore, containing the binary.

• Or, if usingStack, youcan do this:

  $ stack build$ stack install

A.mrk file is a list of judgements.

A judgement starts with a header mark (a sequence of#), a title(followed by a:), given points (followed by/), and maximumpoints (followed by a line break). The number of# determines thedepth of the header, and every filemust start at depth 1, but mayhave multiple depth 1 judgements. Headings may be arbitrarily nested,but must sum up correctly.

For instance, here is a file containing only quantitative remarks:

# Theory: 27/50## Question 1: 10/10## Question 2: 10/20## Question 3: 7/20# Practice: 35/50## Task 1: 20/25## Task 2: 15/25

Sums of given points can be omitted in higher-level judgements, andremarks will calculate these from points given to lower-leveljudgements. For instance, we could omit the given points fromPractice above:

# Practice: /50## Task 1: 20/25## Task 2: 15/25

It is possible to indicate that a student did not submit an answerwith writing- instead of a number points. This will be treated like0 points. For instance:

# Theory: 17/50## Question 1: 10/10## Question 2: -/20## Question 3: 7/20

The header of a judgement may be followed by qualitative remarks. Remarks beginwith an indent (two spaces), and a mood mark:

• * for neutral/structural remarks;
• ^ for very positive remarks;
• + for positive remarks;
• - for negative remarks;
• v for very negative remarks;
• ~ for mixed remarks;
• ? for impartial remarks;
• ! for warning remarks.

Impartial remarks are good for judgements where the mood is left to be judgedby a higher authority. For instance, when a teaching assistant is uncertain,and would like to let the teacher decide, or the solution cannot be used tomake a judgement about this point.

Structural remarks are good for listing things to look for. For instance, a(template) judgement for an operating systems exercise might look somethinglike this:

## T1: /15### Formal requirements: /5  * Has code  * Has XML in a reloadable/testable form  * Has an explanation### Quality assessment: /10  * Understands semaphores  * Understands deadlocks  * Understands starvation  * Avoids deadlocks  * Avoids starvation  * Avoids race conditions  * Has a good degree of multiprogramming  * Solves the problem### Bonus: +0  * Has an accompanying implementation / simulation

Once filled in by a teaching assistant, this may look something like this:

## T1: 9/15### Formal requirements: 5/5  * Has code    + Yes.  * Has XML in a reloadable/testable form    + Yes.  * Has an explanation    + Yes.### Quality assessment: 2/10  * Understands semaphores    - Seems to treat them as counters. Report doesn't talk about procuring or      vacating at all.    - Checks semaphore value directly.    - Seems to have mixed up procure and vacate.  * Understands deadlocks    - If more than 5 cars arrive, then only 5 cars will be allowed to drive      through, and everything will then deadlock.  * Understands starvation    ? Seems to, but this is hard to judge.  * Avoids deadlocks    ? See above.  * Avoids starvation    ? Can't judge this.  * Avoids race conditions    - Uses a busy loop in attempt to synchronize.    - There is a race condition after the busy loop.  * Has a good degree of multiprogramming    + It's a ticket system, so it could be okay.  * Solves the problem    + In a complicated way, but yes.### Bonus: +2  * Has an accompanying implementation / simulation    + Yes!

Bonus-judgementsmust have the titleBonus, but they don't have to be thereif you don't like them. The presence of bonus judgments also allows points tosum up to more than the given maximum. This is regarded as a feature.

You do not need to manually enter sums. Points are only required for thebottom-most judgements. Given the template above you can do aremarks checkto check if all necessary points have been given. This makes a "pointless"template a rather useful starting point for grading.

Points can be integers or half-points, that is, the number may end in.5.Please note,.5 is exactly representable in an IEEE-754 (the internalrepresentation of points), so no numerical imprecision occurs when dealing withhalf-points. Maximum points may only be integers (the internal representationof maximum points is still an IEEE-754 double to avoid numerical conversions).

The file-format is kept "git-friendly" by keeping it comprehensible inplain-text, and allowing for independent marking by splitting the remarks for astudent into multiple files.

The simplest setup is to have one.mrk file per student (e.g.basic.mrk).

To support more exotic setups,remarks can also work with directories:

• If supplied with a directory path,remarks looks for files ending in.mrk inside that directory, and comprehends the files as above, inlexicographic filename order (e.g.,directory-with-mrk-files).

• If there exists a directory<basename> for any<basename>.mrk,<basename> is recursively searched for further.mrk files. Theircontents is appended, in lexicographic filename order, to the lasttop-level judgement of<basename>.mrkmixed-directory).

See theorganization samples for some examples of howjudgements may be structured using files and directories.

├── basic.mrk├── directory-with-mrk-files│   ├── 01-theory.mrk│   └── 02-practice.mrk└── mixed-directory    ├── 01-theory.mrk    ├── 02-practice    │   ├── Task1.mrk    │   └── Task2.mrk    └── 02-practice.mrk

basic.mrk,directory-with-mrk-files, andmixed-directory all parse to thesame judgements. In particular, the output from the followingremarkscommands is identical:

$ cd samples/organization/$ remarks parse basic.mrk$ remarks parse directory-with-mrk-files$ remarks parse mixed-directory