WIP task 3

project_task_sheets/phase_05/project_phase05_tasks/Makefile

@@ -4,7 +4,7 @@ name = MichaelChen
 
 solutionname = Solution_Phase$(phase)_$(name)
 target = $(solutionname)_V$(version).zip
-package = $(solutionname).pdf Duration.java DurationTest.java Mutation_Testing_Sample_Model.xml
+package = $(solutionname).pdf Duration.java DurationTest.java mutants.txt Mutation_Testing_Sample_Model.xml
 latexmkflags =
 
 .PHONY : all dev

project_task_sheets/phase_05/project_phase05_tasks/Solution_Phase05_MichaelChen.tex

@@ -291,22 +291,55 @@
 
     \item As a group, decide on \textbf{one test suite} (original + own tests) that you want to use for this task
     \begin{answer}
-    [TODO: Add answer here]
+    The current head of the \texttt{develop} branch will be used by our group to test our project.
     \end{answer}
 
     \item In consultation with your group, pick a unique mutation tool for Java code (i.e., no two members of one group should use the same tool).
     \begin{answer}
-    [TODO: Add answer here]
+    I picked the Major mutation tool for testing our project. Note that, because the mutation testing tool does not integrate well with Junit we have to create an additional test driver class that manually runs the Junit tests. Because of this manual effort I restrict the test to the \texttt{JSONScanner} class and mutation testing to its \texttt{checkTime} function that I have been developing tests for in the previous exercise.
     \end{answer}
 
     \item Depending on the capabilities of the selected mutation testing tool, determine the amount of \textit{killed mutants} for at least \textbf{3 different mutation operators} (handle the results individually as long as your tool returns the results for each individual operator, or supports the application of single selected operators).
     \begin{answer}
-    [TODO: Add answer here]
+    For the method under test the major tool generated a mutated class with 6239 mutations, 112 of those in the \texttt{checkTime} function. For this method the tool managed to create 28 COR mutations, 11 LVR mutations, 63 ROR mutations and 10 STD mutations. I did not manage to analyze the mutations that were killed. This is probably due to incompatible build tools (Major uses Java 1.7 while fastjson uses Java 1.5). For this reason I took the mutations from the generated log (attached as file \texttt{mutants.txt}) and for all 4 applicable mutation operators in the method under test manually edited the Java file and ran the test suite. The result can be seen in the below table:
+
+
+    \begin{center}
+    \begin{tabular}{c|c|c|c|c}
+    Operator & Line & Original & Mutation & Killed \\ \hline \hline
+    ROR & 729 & \verb+h0 == '0'+ & \verb+h0 <= '0'+ & live \\ % ID: 2275
+    ROR & 729 & \verb+h0 == '0'+ & \verb+h0 >= '0'+ & killed \\ % ID: 2276
+    ROR & 729 & \verb+h0 == '0'+ & \verb+false+ & killed \\ % ID: 2277
+    ROR & 730 & \verb+h1 < '0'+ & \verb+h1 != '0'+ & killed \\ % ID: 2278
+    ROR & 730 & \verb+h1 < '0'+ & \verb+h1 <= '0'+ & live \\ % ID: 2279
+    COR & 730 & \verb+h1 < '0' || h1 > '9'+ & \verb+h1 < '0' != h1 > '9'+ & live \\ % ID: 2284
+    COR & 730 & \verb+h1 < '0' || h1 > '9'+ & \verb+h1 < '0'+ & killed \\ % ID: 2285
+    COR & 730 & \verb+h1 < '0' || h1 > '9'+ & \verb+h1 > '9'+ & killed \\ % ID: 2286
+    COR & 730 & \verb+h1 < '0' || h1 > '9'+ & \verb+true+ & killed \\ % ID: 2287
+    STD & 731 & \verb+return false;+ & \verb+<NO-OP>+ & killed \\ % ID: 2289
+    STD & 735 & \verb+return false;+ & \verb+<NO-OP>+ & live \\ % ID: 2304
+    STD & 739 & \verb+return false;+ & \verb+<NO-OP>+ & killed \\ % ID: 2319
+    LVR & 763 & \verb+false+ & \verb+true+ & killed \\ % ID: 2382
+    LVR & 766 & \verb+false+ & \verb+true+ & killed \\ % ID: 2384
+    LVR & 769 & \verb+true+ & \verb+false+ & killed % ID: 2386
+    \end{tabular}
+    \end{center}
     \end{answer}
 
     \item For each of the applied mutation operators, pick a distinct method of your project that was affected by that operator, and highlight / describe the instructions that are changed. Explain how the test suite does or does not satisfy the criteria of \textit{Reachability}, \textit{Infection}, and \textit{Propagation} for that part.
     \begin{answer}
-    [TODO: Add answer here]
+    \begin{center}
+    \begin{tabular}{l|c|c|c|c|c}
+    Function & Operator & Line & Original & Mutation & Killed \\ \hline \hline
+    \texttt{scanISO8601DateIfMatch} & EVR & 442 & \verb#c0# & \verb#0# & live \\ % ID: 1075
+    \texttt{checkTime} & COR & 730 & \verb+h1 < '0' || h1 > '9'+ & \verb+h1 < '0'+ & killed \\ % ID: 2285
+    \texttt{subString} & AOR & 161 & \verb#offset + count# & \verb#offset % count# & killed \\ % ID: 195
+    \end{tabular}
+    \end{center}
+
+    For the EVR the test suite fails does not cover the line. If \verb+c0+ was non-zero, infection would be guaranteed, but since the mutation is not reachable by the test suite it is a live mutation.
+    The COR has been killed strongly, because the line of code was reached by my extended test suite from the last exercise, and infection has been observed, because I had added a test case that has \verb+h1 > '9'+ which is not caught by the mutated code. The infection propagates directly because the condition directly branches to the return statement.
+    The AOR in the \texttt{subString} function is also killed by three of my \texttt{inputTokenTest} methods from a previous exercise and also two of Mark's tests. This method is at the core of the scanner as it is used to stringify the buffer at the current location to pass it on to specific parser methods. It is thus easily reached by the test suite and will most certainly infect any valid inputs. The propagation is also rather trivially visible as the infected method will return an incorrect string length and thus wrong buffer contents.
     \end{answer}
 
     \item Pick one suitable method (restricted to int values) or class from your project, and derive an automaton expressing its behaviour. For the creation of your automaton, use the \textit{Uppaal} model checker environment.