dorie.parscraper

What is dorie.parscraper?

dorie.parscraper is a python package with the purpose of automatic creation of DORiE parameter files in various formats. The developers only need to maintain one single XML file that contains all the relevant information for each parameter.

It provides the following components:

  • Readers for parsing both the XML file containing the meta-information for each parameter and the DORiE C++ source files,

  • a matching function that complements the information from the XML file with the location where each parameter is actually used in the code,

  • Writers for various output formats (e.g. to dynamically create the .ini files used in runs of the main program, or to create a user cheat sheet with further explanations),

  • a wrapper script controlling the scraping process.

Another purpose is to identify mismatches between XML meta-information and source code and notify the developers. Therefore, a warning is issued if either a parameter is only found in the XML file, or only found in the DORiE source code.

How to use dorie.parscraper

The dorie.parscraper scripts are called via CMake while configuring DORiE. This is to make sure that the generated ini files are accessible as soon as the system tests are created.

The dorie.parscraper scripts

scrape_folder.py

This script basically just parses the command line arguments, implements special handling of warnings and exceptions to keep the output as clean as possible, and calls dorie.parscraper.wrapper.scrape_folder.

Command line options:

--xml

Path to the XML file holding parameter meta-information

--source

The C++ source folder to scrape

--out

Output file(s)

--css

Path to a CSS file to include into HTML output (optional)

--debug

Show warnings (optional, default off)

The dorie.parscraper core modules

Parameter class

class dorie.parscraper.parameter.Parameter(model, category, key)[source]

Implements a class for storing DORiE parameters.

Parameters
  • category (str) – The category of the parameter, indicated in the DUNE infiles by square brackets.

  • key (str) – The name of the parameter, at the left hand side of the equal sign in DUNE inifiles.

__write_version_dict(data, heading)
_code_found

Whether the parameter was found in the code at all

_sources

List of all the positions in the code where the parameter was found, encoded in a tuple: (filename, lineno, C++ type)

category

Parameter category

comment

Extra comment showing up in the generated ini files

definition

Long explanation of the parameter, shown in the documentation

deprecated

Versions and descriptions on feature deprecated

hidden

Do not issue an error if parameter is not found

key

Parameter name

model

Model category

suggestion

Suggested (default) value

values

Possible values

versionadded

Versions and descriptions on feature added

versionchanged

Versions and descriptions on feature changed

Warnings

exception dorie.parscraper.warnings.OutputWarning[source]
exception dorie.parscraper.warnings.ScraperWarning[source]
exception dorie.parscraper.warnings.XMLWarning[source]

Matching function

dorie.parscraper.match_parameters.match(xml_parameters, source_parameters, base_path)[source]

Expects the output of dorie.parscraper.readers.xml and dorie.parscraper.readers.source() as input, and tries to match the parameters with each other. Records, where the parameters specified in the XML file are found in the code (file name, line number, and which type they are cast to), and supplements the Parameter instances with this information.

Returns an OrderedDict with categories as keys and lists of Parameter instances as values. Parameters that are only found in the source files, but are not specified in XML, are not included in the output. Parameters that are only specified in the XML file but not in the code are returned, but a warning is issued.

Parameters

Wrapper

dorie.parscraper.wrapper.scrape_folder.scrape(xml, source, out, overwrite=False, model=None, debug=False)[source]

Readers

XML reader

dorie.parscraper.readers.xml.parse(filename, model=None)[source]

Reads and parses a DORiE parameter XML file, using the xml.etree.ElementTree package.

The XML file is expected to have the following structure:

<dorie> ‣ <category name=”…”> ‣ <parameter name=”…”> ‣ (attributes)

Available attributes are all attributes of the dorie.parscraper.parameter.Parameter class, except the ones starting with an underscore.

Parameters

filename (str) – Path to the input XML file

Return type

OrderedDict

Returns

all parsed category names as keys, and a list of all parameters in the corresponding category as values (instances of Parameter)

Raises

XMLWarning – if unknown, redundant, or malformatted XML elements are encountered

C++ source reader

dorie.parscraper.readers.source.parse(filename, model=None)[source]

Reads a source file and looks for patterns like

inifile.get<type>("category.test.parameter",default)

When found, the relevant information gets scraped and assembled in a dict.

Note

You can prevent a parameter to be matched by adding // %HIDDEN% to the end of the corresponding line.

Parameters

filename (str) – Full path to the input file

Returns

list of dicts (one dict for each matching pattern in the source file), each containing all found information

Writers

.ini file writer

dorie.parscraper.writers.ini.write(parameters, out, *args, **kwargs)[source]

Writes the contents of the parameters input dict to a .ini file. The output is structured as

[key]
# value.comment
value.key = value.suggestion

to match the format of the DUNE input files. A missing comment attribute leads to omitting of the comment line; a missing suggestion to the value UNDEFINED.

Parameters
  • parameters (dict) – dict with categories as keys and iterables of Parameter instances as values.

  • out (str) – Path to the output file. Must be writable.

.rst file writer

dorie.parscraper.writers.rst.write(parameters, out, path_base, *args, **kwargs)[source]

Writes the contents of the parameters input dict to a .rst file. The output is structured as

Category: <category>

<parameter>

<description>

Possible values | <values> Default value | <suggestion> Queried at | <file path (if available)>

The description obeys paragraph breaks in the description and properly formats list indentations.

Parameters
  • parameters (dict) – Keys: category, Values: iterables of dorie.parscraper.parameter.Parameter instances.

  • out (str) – Path to the output file. Must be writable.

  • path_base (str) – Base path to the source files. Required for working links to the files.

.html file writer

dorie.parscraper.writers.html.write(parameters, out, path_base, *args, **kwargs)[source]

Writes the contents of the parameters input dict to a .html file. The output is structured as

category

Parameter

Definition

Possible values

Default

Queried at

p.key

p.definition

p.values

p.suggestion

all sources of p

Parameters
  • parameters (dict) – dict with categories as keys and iterables of dorie.parscraper.parameter.Parameter instances as values.

  • out (str) – Path to the output file. Must be writable.

  • path_base (str) – Base path to the source files. Needed for creation of working links to the files.

  • css (str) – Path to a CSS file containing the styling of the HTML output. Is included into the HTML output (inline). Optional.