173 lines
9.1 KiB
TeX
173 lines
9.1 KiB
TeX
\documentclass[a4paper]{scrreprt}
|
||
\usepackage[left=4cm,bottom=3cm,top=3cm,right=4cm,nohead,nofoot]{geometry}
|
||
\usepackage{graphicx}
|
||
\usepackage{tabularx}
|
||
\usepackage{listings}
|
||
\usepackage{enumitem}
|
||
\usepackage{subcaption}
|
||
|
||
\usepackage{pgf}
|
||
\usepackage{tikz}
|
||
\usetikzlibrary{arrows,automata}
|
||
|
||
\usepackage{xparse}
|
||
\usepackage{multirow}
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\setlength{\textfloatsep}{16pt}
|
||
|
||
\renewcommand{\labelenumi}{\alph{enumi})}
|
||
\renewcommand{\labelenumii}{\arabic{enumii}) }
|
||
|
||
\newcommand{\baseinfo}[5]{
|
||
\begin{center}
|
||
\begin{tabular}{p{15cm}r}
|
||
\vspace{-4.5pt}{ \Large \bfseries #1} & \multirow{2}{*}{} \\[0.4cm]
|
||
#2 & \\[0.5cm]
|
||
\end{tabular}
|
||
\end{center}
|
||
\vspace{-18pt}\hrule\vspace{6pt}
|
||
\begin{tabular}{ll}
|
||
\textbf{Name:} & #4\\
|
||
\textbf{Group:} & #5\\
|
||
\end{tabular}
|
||
\vspace{4pt}\hrule\vspace{2pt}
|
||
\footnotesize \textbf{Software Testing} \hfil - \hfil Summer 2022 \hfil - \hfil #3 \hfil - \hfil Sibylle Schupp / Sascha Lehmann \hfil \\
|
||
}
|
||
|
||
\newcounter{question}
|
||
\NewDocumentEnvironment{question}{m o}{%
|
||
\addtocounter{question}{1}%
|
||
\paragraph{\textcolor{red}{Task~\arabic{question}} - #1\hfill\IfNoValueTF{#2}{}{[#2 P]}}
|
||
\leavevmode\\%
|
||
}{%
|
||
\vskip 1em%
|
||
}
|
||
|
||
\NewDocumentEnvironment{answer}{}{%
|
||
\vspace{6pt}
|
||
\leavevmode\\
|
||
\textit{Answer:}\\[-0.25cm]
|
||
{\color{red}\rule{\textwidth}{0.4mm}}
|
||
}{%
|
||
\leavevmode\\
|
||
{\color{red}\rule{\textwidth}{0.4mm}}
|
||
}
|
||
|
||
\newcommand{\projectinfo}[5]{
|
||
\baseinfo{Project Phase #1 - Submission Sheet}{#2}{#3}{#4}{#5}
|
||
}
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
|
||
\def\name{Michael Chen}
|
||
\def\group{Group 01 (fastjson)}
|
||
|
||
\begin{document}
|
||
\projectinfo{1}{Software Testing - Random Testing\small}{\today}{\name}{\group}
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
%%% Task 1 %%%
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\begin{question}{Select your open source project}[3]
|
||
\begin{enumerate}[topsep=0pt, leftmargin=*]
|
||
\item Motivate your selection of the project in 3-5 sentences:
|
||
\begin{answer}
|
||
We set out to look for a local-only library, so we could have a simplified setup without needing any external infrastructure. FastJSON is a straightforward JSON parsing library with around 30k lines of code, and a fairly low code coverage of around 40-50\%. It’s primarily for performance-intensive uses such as on the server-side. We think it is the ideal scope for a project of our consideration.
|
||
\end{answer}
|
||
|
||
\item Document the \textbf{build}, \textbf{install}, and \textbf{run} steps for your project:
|
||
\begin{answer}
|
||
Installation was fairly easy with cloning the remote repository. Since the JSON library is self contained and has no further libraries it depends on the installation is rather trivial.
|
||
|
||
Building the project was also simple because the application is delivered with a well configured Apache Maven project file. Using IntelliJ IDEA the Maven project could be opened and built relatively quickly.
|
||
|
||
Running the project was similarly easy, because we intentionally chose a software library instead of a complex web application. Since the library is not shipped with a main function we can just execute the test suite.
|
||
\end{answer}
|
||
\end{enumerate}
|
||
\end{question}
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
%%% Task 2 %%%
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\begin{question}{Answer basic questions on Random Testing / Randoop}[4]
|
||
\begin{enumerate}[topsep=0pt, leftmargin=*]
|
||
\item Answer the following questions (each in 2-3 brief sentences):
|
||
\begin{enumerate}
|
||
\item How do you define "Random Testing" in your own words?
|
||
\begin{answer}
|
||
Random Testing is a testing procedure where the system under test (SUT) is tested with randomly generated inputs without knowledge of the system's code itself (black-box testing). The test results must then be evaluated according to the system's specifications.
|
||
\end{answer}
|
||
|
||
\item In which testing situations is it used / probably preferred?
|
||
\begin{answer}
|
||
It is preferred in situations where manual checking is prone to fail due to easily overlooked lines of code. Random testing is used mostly in situations where the SUT is well specified, because the output of a system must be analyzed using the specifications to check whether a failure is present. Also for small project teams random testing might be advantageous because the developers of the software are likely biased on how they write tests and what they expect the program to do, because they wrote the software, but it might not behave as it should per specification.
|
||
\end{answer}
|
||
|
||
\item What are its most significant downsides?
|
||
\begin{answer}
|
||
\begin{itemize}
|
||
\item Failures are only detected when specific inputs trigger edge cases of the software.
|
||
\item Bugs may only be found if the test results are properly analyzed and compared to the system specifications.
|
||
\item As the name already says it is random and thus we depend on luck that we find a bug.
|
||
\end{itemize}
|
||
\end{answer}
|
||
|
||
\item How does \textit{Randoop} approach the concept of random testing?
|
||
\begin{answer}
|
||
Randoop employs contracts over the objects that the test subjects use. Specifically it checks for well-formedness on common patterns in object oriented programming, i.e. in case of object equality contracts it checks for reflective and transitive properties.
|
||
Randoop instanciates those objects using the SUT and applies the contracts to them, thereby generating test cases on the objects.
|
||
|
||
Randoop categorizes these tests as regression tests, for tests that capture the current and well-formed behaviour of the SUT and can be used to detect changes in the behaviour of the program during later development, and error-revealing tests, that are currently failing when executed.
|
||
\end{answer}
|
||
\end{enumerate}
|
||
\end{enumerate}
|
||
\end{question}
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
%%% Task 3 %%%
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\begin{question}{Get started with Randoop}[3]
|
||
\begin{enumerate}[topsep=0pt, leftmargin=*]
|
||
\item Briefly explain the concepts of \textit{error-revealing tests} and \textit{regression tests}:
|
||
\begin{answer}
|
||
|
||
\end{answer}
|
||
\end{enumerate}
|
||
\end{question}
|
||
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
%%% Task 4 %%%
|
||
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
|
||
\begin{question}{Use Randoop for your selected project}[6]
|
||
\begin{enumerate}[topsep=0pt, leftmargin=*]
|
||
\item Provide a screenshot of your terminal that indicates your execution of the necessary step(s):
|
||
\begin{answer}
|
||
\lstinputlisting[belowskip=-0.8\baselineskip,breaklines=true,postbreak=\mbox{\textcolor{red}{$\hookrightarrow$}\space}]{randoop_invoke.sh}
|
||
\end{answer}
|
||
|
||
\item For each of both classes, pick a test from one of the generated files, and briefly describe its sequence of constructor / method calls (1-2 sentences per test):
|
||
\begin{answer}
|
||
This following file from the JSONPatch Operation subclass shows a test case where an operation is instanciated with some (arbitrary) values. The test captures that the operation is instanciated without a type (null) and it also captures that checking the operation value property repeatedly does not change its value (is without side effects).
|
||
|
||
\lstinputlisting[firstline=87,lastline=106,belowskip=-0.8\baselineskip,breaklines=true,postbreak=\mbox{\textcolor{red}{$\hookrightarrow$}\space}]{randoop/JSONPatch/RegressionTest4.java}
|
||
|
||
The following test is and error revealing test in the DefaultJSONParser class that revealed that the call to the getOwnerContext method of this specific instanciation of the parser raises a NullPointerException.
|
||
|
||
\lstinputlisting[firstline=56,lastline=64,belowskip=-0.8\baselineskip,breaklines=true,postbreak=\mbox{\textcolor{red}{$\hookrightarrow$}\space}]{randoop/JSONPath/ErrorTest0.java}
|
||
\end{answer}
|
||
|
||
\item Do the tests represent meaningful scenarios?
|
||
\begin{answer}
|
||
The first of the above tests is not a meaningful scenario because usually one does not try to instanciate an actor object without setting the field values and with such a standard data struct-like object that does not contain any functionality itself one does not expect any side-effects.
|
||
|
||
The second test however, if this is behaviour as specified or not, might be a meaningful scenario. Usually a simple member function call should not raise a NullPointerException when called on a non-null object and internal exceptions should be caught and handled accordingly.
|
||
\end{answer}
|
||
\end{enumerate}
|
||
\end{question}
|
||
|
||
\end{document}
|