It's possible for test files to declare a package with the "_test"
suffix. If these packages contain build errors, they were not correctly
matched to the package without the "_test" suffix.
Refs #145
In some cases we may have captured some output for a package that didn't
have any tests. We should include it in the generated report as it may
contain useful information.
When we encounter a failure but there was not failing test, we create a
failing dummy test in the report. This is to prevent the failure from
being overlooked.
The case where the output contained a failing benchmark was not handled
correctly, resulting in a dummy test failure being added to the report
unnecessarily.
Benchmark durations were originally formatted with higher precision
because they contained just the NsPerOp field. Now that we're
approximating the total duration they are more accurate and don't need
as many decimals.
The NsPerOp was previously casted to a time.Duration (int64) before
multiplying it with the number of iterations. This has been fixed by
doing the casting after the multiplication. The resulting duration
should be a bit more accurate.
The reportBuilder has been updated to use the ordered output collector
to keep track of go test output. This makes it possible to include
benchmark output in the generated report and makes sure that output is
preserved when deleting subtest parents from the report.
The consequence of this is that any output emitted after the test or
benchmark result, will no longer belong to the last test. Instead it
will be appended to the global log. This is necessary to correctly
attribute output to the correct test or benchmark, for example in cases
where both tests and benchmarks appear.
In the past, Go test output printed test logs after the test result.
However, from Go 1.14 this behaviour was changed and test logs are now
printed before the test result.
With this commit, we effectively no longer support the old behaviour in
Go1.13 and earlier. The output will still appear in the generated
report, however it will no longer belong to the correct test.
Many of the testdata files were generated a long time ago with very old
versions of Go. The Go test output has changed over time, and these test
inputs no longer reflect the current state. The intention is to fully
support the test output of the most recent Go versions.
This commit also includes the source used to generate the test output,
so the output can be more easily updated in the future.
Whitespace prefix of output lines was carried over from the old v1
version, but in some cases it removes too much whitespace. Ideally it
would detect the common prefix of all output lines for a given text and
only trim that. Until that's fixed, let's just stop removing the
whitespace.
Fixes#131
There was some code duplication between the go-junit-report binary, its
tests and the testdata/generate-golden script. This has been moved into
an internal package. The go-junit-report binary can now just focus on
flag parsing and validation, and it should be less likely that the
binary, tests and golden report generator behave differently.
In cases where a test result is not found, for example when a panic
happened, a dummy failing test is added to the report. Since we don't
know exactly what failed, its error message was simply "Run error". This
has been renamed to "Runtime error".
The testdata directory contains various inputs and the resulting
report.xml that is expected to be generated. Making changes that result
in differences in the generated XML reports requires the report.xml
files to be manually updated. This is time consuming and error prone.
The generate-golden.go script (re)generates all report.xml files in the
testdata directory from the corresponding input. This can be done by
simply running `go generate` from within the testdata directory. Make
sure the generated reports contain the changes you expected before
committing them.
As part of this change, all reports have been regenerated. The diffs in
the report.xml files are caused by the following recent changes:
- XML attributes now appear in the order as defined in the structs in
the junit package.
- Failure and error messages are now wrapped in <![CDATA[]]>.
- Package names are no longer truncated to the last path component.
When running `go test` without the `-v` flag, the output may not contain
everything that we expect. For example, no output is generated for
passing tests. Even for a failing test the output does not contain a
`=== RUN` line.
Currently, this resulted in us ignoring the test result since we
couldn't find an existing test to assign this result to. We should
however handle this situation gracefully, and just assume a test exists
when only encountering a test result line.
References #109
Sometimes, the text after "# " shows the name of the test binary
("<package>.test") in addition to the package
e.g.: "# package/name [package/name.test]".
This change ensures that the packages' build outputs are associated
correctly in this case.
Mainly:
* Moving the averaging/merging of benchmarks from the parser to the formatter package
* Tightening up the regex so it only captures the numeric values (no more of trimming spaces and the ns/op)
* Deleting the writing up in xml file the benchmark memory sections of B/op and Allocs/op
Also added a test case for parseNanoseconds().
