ICSE22 - victoryang00’ blog

May 31, 2022June 10, 2022

Growing A Test Corpus with Bonsai Fuzzing @ICSE'21

For generating synthesization automatically based on the ChocoPy dialect which I'm in great need of, the author of ChocoPy published their tricky counterpart to C-smith/ Fuzzy Grammer Generator called Bonsal Fuzzing.

Problems and Pair Review

Instead of Fuzz-then-reduce method, the corpus bottom up generation is already concise. enough and can touch much of the corner test cases.

Bounded Exhaustive Testing: input of bounded size are generated systematically but not enumerated exhaustively
So enumerate the k-path with the grammar.
JPF-SE explores the space of program paths, for bounding the size of a comprehensive test suite that covers a diverse set of program paths
different kind of strategies of fuzzing: Coverage-Guided Fuzzing, Specialized Compiler Fuzing, Grammar-based, Semantic Fuzzing(Zest)
Test-Case Reduction by Hieachical delta debugging

Implementation

Bounded Grammar Fuzzers: Bound iteration by idens, items, depths number.
Coverage-Guided Bounded Grammar Fuzzing

The lattice of coverage-guided size-bounded grammar-based fuzzers $F_{m,n,d}$, ordered by three size bounds on the syntax of the test cases they produce: number of unique identifiers m, maximum sequence length n, and maximum nesting depth d.

Test cases flow along directed edges: the inputs generated by each fuzzer are used as the seed inputs to its successors. The result of bonsai fuzzing is the corpus produced by the top-most element.

Bonsai fuzzing with extended lattice

April 21, 2022April 28, 2022

Preempting Flaky Tests via Non-Idempotent-Outcome Tests @ICSE '22

Nonidiomatic test are one of Flaky tests category. In the previous TACAS work, the relation between polluters(modify the shared state) and victims(read from the shared state) are found but there exists a lot false positive for it to have Order dependent variables inside, from our previous tests on these testcases. Also for the latent polluters/victims that potentially read/write the variables. e.g.

//shared variables x,y,z are initialized into 0
void t1(){ assert x==0; } // victim
void t2(){ x = 0; } // polluter
void t3(){ assert y==0; } // latent-victim
void t4(){ z=1; } // latent-polluter
void t5(){ assert w=0;w=1;}

The NIO tests means passes in the first run but fails in the second when run twice consecutively.

The NIO will self pollutes the state that its own assertions depend on