Repository files navigation

Objective: test the Qucs (GUI) and Qucsator (simulator).

The test projects under thetestsuite directory are all run by default.

To skip failing projects use the--exclude option with a list of projects to be ignored.

usage: run.py [-h] [--prefix PREFIX] [--qucs] [--qucsator]              [-p [{sch,dpl,all}]] [--add-test FILE] [--exclude FILE]              [--include FILE] [--project PROJECT] [--compare PATH [PATH ...]]              [-v [LEVEL]] [--reset] [--timeout TIMEOUT] [--rtol RTOL]              [--atol ATOL] [--plot-interactive] [-mp [NUM]]Qucs testing script.optional arguments:  -h, --help            show this help message and exit  --prefix PREFIX       prefix of installed Qucs (default: /usr/local/bin/)  --qucs                run qucs tests  --qucsator            run qucsator tests  -p [{sch,dpl,all}], --print [{sch,dpl,all}]                        run qucs and prints schematics and/or data displays to                        file  --add-test FILE       add schematic file to the testsuite  --exclude FILE        file listing projects excluded from test  --include FILE        file listing projects selected for test  --project PROJECT     path to a test project  --compare PATH [PATH ...]                        two full paths to directories containing qucs or                        qucsator binaries for comparison test  -v [LEVEL], --verbose [LEVEL]                        increase verbosity: 0 = only warnings, 1 = info, 2 =                        debug. No number means info. Default is no verbosity.  --reset               Reset (overwrite) data and log files of test                        projects.Run qucsator given with --prefix.  --timeout TIMEOUT     Abort test if longer that timeout (default: 90 s).  --rtol RTOL           Set the element-wise relative tolerace (default 1e-1).                        See: Numpy allclose function.  --atol ATOL           Set the element-wise absolute tolerace (default 1e-5).                        See: Numpy allclose function.  --plot-interactive    Plot and show error graphs interactively. Hardcopy PNG                        saved by default.  -mp [NUM], --processes [NUM]                        Use NUM processes to run the simulations (default:                        number of CPU cores).                        This option is disabled on Windows and macOS.

• The script is rather verbose and the progress is printed to the terminal (with colors, yeah!).

• A table with the run project, Qucs schematic version and simulation runtim is saved automatically and time-stamped.

• A report table with all components available to Qucs and the coverage with respect to the types of simulations.

Example of running the test-suite for Qucsator while skipping a few test projects:

$python3 run.py --prefix /home/user/local/qucs-master/bin/ --exclude skip.txt --qucsator

The test can be run using several processes in parallel, which will shorthen the overall simulation time if the machine has a multi-core processor, using the-mp option; this option not supported on Windows and macOS.

$python3 run.py --prefix /home/user/local/qucs-master/bin/ --exclude skip.txt --qucsator -mp

Two differentqucsator versions can be compared with the command:

$ python3 run.py --compare /home/user/bin1/ /home/user/bin2/ --qucsator -mp

where/home/user/bin1/ and/home/user/bin2/ are the two directories containing thequcsator binaries to be tested.

See below examples of printing schematics to file (pdf):

• printing all
• printing all data display (.dpl) files only
• skipping a list of projects
• a single project from the testsuite

$python3 run.py --prefix /home/user/local/qucs-master/bin/ --print$python3 run.py --prefix /home/user/local/qucs-master/bin/ --print dpl$python3 run.py --prefix /home/user/local/qucs-master/bin/ --exclude skip_print.txt -print$python3 run.py --prefix /home/user/local/qucs-master/bin/ --print --project AC_SW_resonance_prj

As for thequcsator test, the-mp option can be added to run multiple (printing) processes in parallel (this option not supported on Windows and macOS).

The script will generate a table showing, for every file, the time needed to print the schematics to file or thequcs error code in case the file generation was not successful.

Two differentqucs versions can be used to generate the print files and the results for every binary will be shown in the report table, to allow for a quick comparison. Use a command like:

$ python3 run.py --compare /home/user/bin1/ /home/user/bin2/ --print -mp

where/home/user/bin1/ and/home/user/bin2/ are the two directories containing thequcs binaries to be tested.

Qucs is featured with an equation system available in the schematic area.

The script below tests several of the built-in functions.

In short, it works as follows:

• For each entry in src/application.h.
• Generate random (or constrained) arguments (double, complex, matrix...)
• Create a test equation and computes the expected result (with Python and Numpy).
• Save test equation into a netlist
• Run the netlist and read the simulator output.
• Compare simulator output to the expected result.

usage: run_equations.py [-h] [--prefix PREFIX] [--operation OPERATION]Qucs testing script.optional arguments:  -h, --help            show this help message and exit  --prefix PREFIX       prefix of installed Qucs (default: /usr/local/bin/)  --operation OPERATION                        test one particular operation, ex. '+')

To run all the tests, use:

$ python run_equations.py --prefix /home/user/git/qucs-clone/

Centos 3.10.0-1062.4.1.el7.x86_64, Python 3.6.8, numpy 1.18.1

Mac OSX not tested yet

Windows not tested yet

You might need to installargparse which is not included with Python 2.6. Something like:

$sudo apt-get update$sudo apt-get install python-setuptools$sudo easy_install pip$sudo pip install setuptools --no-use-wheel --upgrade$sudo pip install argparse

• List of Qucs components

• Testsuite simulation report

• Testsuite component coverage report

• Currently failing projects

• Output without errors

• Output with errors

under construction

It considers every directory undertestsuite as a project to be run. Then,

• It run tests to check the schematic to netlist conversion, using the Qucs-GUI code.

• It compares (diff) a reference netlist with the generated netlist on the step above.

• A simulation is run for each project

• Results are for the dependent variables are compared with reference results

• It collects : failed netlist diffs, simulation runtime, failed variable comparison, output and warnings (not used currently).

Running Qucs and Qucsator from the command line makes it easy to automated testing.

It is possible to use Qucs GUI from the command line to read a schematic and create the netlist.

$qucs -n -i file.sch -o netlist.txt

The Qucs GUI can be used from the command line to read a schematic and crete a printout (feature in development).

$qucs -p -i file.sch -o netlist.pdf

Having a netlist file it is possible to run the Qucsator backed to read the netlist and create the results (and log messages).

$qucsator -i netlist.txt -o result.dat

• Tests should not take too much time to run.
• Tests should focus into a particular feature which can be tested in isolation
• Tests could be categorised, ex. DC, AC analysis, a component...
• Tests should exercise corners that are crucial for stable releases.
• Before releasing, the test set should be run with no regressions.
• Git can be used to version test result (timing history?)
• Usegit diff -G '^([^#])' to ignore the changes on the header line of the netlist files.

A test case is stored on a directory, similar to a Qucs project

Suggested contents:

• test.txt
  • test config file
• Schematic (.sch)
  • Test schematic (name convention)
  • Subcircuits dependencies
• Netlist (.txt)
  • reference netlist
• Result (.dat)
  • reference results
• Optional
  • reference log file (.txt)
  • PDF dump of schematic/data display

What about asco, verilog, vhdl?

The test-case is run depending on the available inputs. The main steps are listed below

• convert schematic into test netlist
• compare reference netlist with test netlist
• run netlist (or test netlist) store test result
• compare reference result with test result

Schematic ==> Netlistinput:  [name].schoutput: netlist_test.txtNetlist ==> Resultinput:  netlist.txt | netlist_test.txtoutput: result.dat | result_test.txt        log.txt        [asco.log]

compare:  - netlist X netlist_test  - result  X result_test

Simple diff, skip first line. Return different lines.

Qucsator has the option to check a netlist. ?? what for ??

It uses the numpyallclose function to check how close together are all the dependent variables (arrays) on the reference and test data files.

Tests should be similar to projects. So making it easy to open on Qucs an use as examples.

qucs-test/  run.py  testsuite    TR_collpits_prj/      collpits.sch    AC_bandpass_prj/      bandpass.sch      opa.sch      bandpass.dat      netlist.txt      log.txt    ASCO_AC_fobar_prj/    VERILOG_    VHDL_

Notes:

• Need to identify the test schematic among the subcircuits
  • directory name, and the name of the schematic to be netlisted/simulated.

Whish-list

• what about verilog, vhdl, asco?
• pack/unpack tarballs/examples?

• Create/update list of test/prj:

  • check consistency of the test-suite.
  • it looks what tests are available.
  • it adds new tests as they are found.
  • it preserves the ones that are commented out #, give warning

• Test behavior depends on the provided input/reference files

  • if [sch] provided -> netlist, check
  • if [sch, dat] provided -> nelist, [check], simulate, check
  • if [sch, dat, log] provided -> do netlist, [check], simulate, check, statistics

• Give option to create reference files when missing.

• Table Report pass, fail, missing resource

• Catch errors, time out, kill process if taking more than expected

• Report checks, time improvement/regression

• Html report?