MatrixTest.MatrixTestRunner

class MatrixTest.MatrixTestRunner.MatrixTestRunner(cmd: str, matrix: Dict[str, List[str]], parser: Optional[Callable[[str], Any]] = None, enable_echo: bool = False, logfile: Optional[str] = None, timing: bool = False, enable_email_notification: bool = False, email_provider: Optional[MatrixTest.EmailService.EmailProvider.EmailProvider] = None, email_recipient: str = '')

Bases: object

MatrixTestRunner is the all-in-one class to create a test suite with multi-dimensional test matrix. Please do not reuse one instance of this class for multiple tests.

__init__(cmd: str, matrix: Dict[str, List[str]], parser: Optional[Callable[[str], Any]] = None, enable_echo: bool = False, logfile: Optional[str] = None, timing: bool = False, enable_email_notification: bool = False, email_provider: Optional[MatrixTest.EmailService.EmailProvider.EmailProvider] = None, email_recipient: str = '')

Instantiate MatrixTestRunner, checking the user input and initializing options.

For the tutorial please move to Github

Parameters

• cmd – The command line template string, which follows the string.Formatter style.

• matrix – The possible arguments.

• parser – The parser function.

• enable_echo – If True, the stdout and stderr of the the command will be piped and print in real time.

• logfile – If given a file path, all the output (stdout) of this function will also be written to that file (in append mode). The content of this file will also be controlled by whether the echo feature is enabled or not (see also __init__(), enable_echo(), and disable_echo()).

• timing –

  If True, the execution time of every attempt will be recorded, just like you run your command with time but here the time is measured as the life time of the child process.

  We recommend you to measure the fine-grained execution time inside the benchmark if an accurate execution time is desired or the execution time is very short.

  If the result returned from parser function includes a key of “time” (this is how the execution time is recorded, this feature will be disabled and a warning will be displayed.

• enable_email_notification –

  If True, an email will be sent at the end of run(), and the email service provider and the recipient must be given either by the following two arguments or by calling enable_email_notification().

  Please note: if you also set send_by_email when calling to_excel(), you will receive two emails with same content but one of them has attachment.

• email_provider – Instance of email service provider. Refer to EmailService.

• email_recipient – The email address of recipient. Only one recipient is allowed.

average(column: Optional[Union[str, List[str]]] = None) → None

This function compute the arithmetic mean of selected columns. The results will store in the same dataframe with names of “avg_*”.

If you would like to include execution time when timing is enabled, just add "time" to the column list.

Parameters

column – str or List of str. Name(s) of selected column(s). If any column does not exist in the result, the calculation will be skipped. Generally, this should be a subset of the keys returned by the parser function.

Returns

None

change_logfile(filepath: str) → None

Change the log file path. This may enable or disable the feature, if provided a empty string or did not provide a logfile at constructor, respectively.

Parameters

filepath – The path to the log file.

Returns

None

See also disable_log()

deregister_email_service() → None

Deregister email service provider and recipient, and disable email notification.

Returns

None

disable_echo() → None

Disable echo feature.

Returns

None

See also: enable_echo()

disable_email_notification() → None

Disable email notification but keep the email service provider and recipient registered, which can be used to send the Excel file by email.

Returns

None

disable_log() → None

Disable log to file feature.

Returns

None

disable_timing() → None

Disable timing.

Returns

None

enable_echo() → None

Enable echo feature, output the cmd’s stdout in real time.

Returns

None

enable_email_notification() → None

Enable email notification. An email will be sent at the end of run(), and the email service provider and the recipient must be given either by the following two arguments or by calling enable_email_notification(), otherwise this function has no effect.

Please note: if you also set send_by_email when calling to_excel(), you will receive two emails with same content but one of them has attachment.

Returns

None

enable_log(logfile: str) → None

Enable log to file feature. Alias of change_logfile().

Parameters

logfile – path to the log file

Returns

None

enable_timing() → None

Enable timing.

Returns

None

get_last_result() → pandas.core.frame.DataFrame

Returns

Return the internal dataframe storing the final results.

get_partial_result() → pandas.core.frame.DataFrame

Returns

Return the internal dataframe storing the final results. The last row may not be complete.

register_email_service(provider: MatrixTest.EmailService.EmailProvider.EmailProvider, recipient: str) → None

Register email service provider and recipient. This function will NOT enable email notification. Please call enable_email_notification() to explicitly enable it.

Parameters

• provider – The instance of an email provider class. Refer to EmailService.

• recipient – The email address of recipient. Only one recipient is allowed.

Returns

None

run(repeat: int = 1) → None

Run the test suite and record the results.

Parameters

repeat – The number of times the test should be repeated for.

Returns

None. The results will be stored internally. You can use get_last_result() to get the last result as a pandas.DataFrame

You can press Ctrl-C to interrupt the process and get the partial results that have already collected. Because Python runtime will forward signal to the child process so the running command will be interrupted also and you may see an error from its stdout.

to_excel(path: str, include_agg: bool = True, include_raw: bool = True, send_by_email: bool = False) → None

Export to Excel spreadsheet.

Parameters

• path – str. The file path to output.

• include_agg – bool. Indicates whether include the aggregated columns or not, such as those generated by average().

• include_raw – bool. Indicates whether include the original results. If it is set to False but have no aggregated results, this argument will be ignored.

• send_by_email –

  If True, the generated Excel file will be sent via email. The email provider and recipient must be registered at initialization or by calling enable_email_notification().

  Please note: if you also set enable_email_notification at initialization or by calling enable_email_notification(), you will receive two emails with same content but one of them has attachment.

Returns

None

MatrixTest.MatrixTestRunner.sigint_handler_wrapper(obj: MatrixTest.MatrixTestRunner.MatrixTestRunner)

SIGINT handler wrapper. Upon SIGINT (Ctrl-C is pressed), output the results already collected to stdout then abort.