CpSc 873

Verification and Validation

 

Name ____________________________________

 

There are 5 questions. Answer any 4 of them. Place a large x of the page containing the question you are not answering. Do NOT answer all 5. If you do I will only count the 4 worst answers.

 

This is an individual effort. You may use the textbook, class handouts, and your notes. Nothing else and no one else. No laptops, no sharing notes or handouts. You have 75 minutes, from 8 am until 9:15 am.

 

When you hand in your exam show a pictured ID.

 

 

 

  1. Testing is the most widely used form of V&V activity and is used in V&V processes that use other techniques as well.
    1. List two coverage measures that can be used to describe how thoroughly a Guided Inspection session has covered a model. Explain what each level of coverage guarantees.
    2. How are testing and model checking complementary activities? In other words, why if I use model checking do I still need to do some forms of testing?  Which ones do I need to do?
    3. What elements of testing appear in the PSK’s performance analysis?

 

 

All use cases – this guarantees the system will address all requirements

All states – this guarantees that all configurations of data covered by the state machine are valid

 

Model checking checks flows through the program but it does not compare those to the requirements. System testing is still required.

 

The scenarios which are flows

 

  1. Model checking
    1. Explain the difference in the actions taken by a certifying model checker that are not taken by one that does not certify.
    2. How does a model checker handle the state space explosion?
    3. What is it about Java PathFinder that makes it more useful in a project than other model checkers? Why is it likely to result in higher quality code than other model checking techniques?
    4. Model checkers work on a state space. Where does the state space that Java PathFinder uses come from?

 

The certifying model checker covers all branches in the tree to be certain but the non-certifying one may prune early

 

It handles the state explosion by in some cases by decomposing into multiple trees and working in parallel

 

The models used for Java Pathfinder are the actual source code. No time is wasted translating. This also reduces the possibilities of errors.

 

JavaPathFinder uses Choice points to identifying values that would be in different states

  1. You are the V&V manager for a large project that is developing a software intensive product to run on an embedded processor in a medical device. For each of the development steps select at least one V&V technique. Describe it and why it is appropriate at this point in the development of this type of system. What roles would be involved in each technique and what would they do?

Requirements

Guided Inspection – it is appropriate because we have a non-executable model – requirements people who developed the model answer questions about it, inspectors compare the model to the opinion of domain experts; and domain experts are involved to give their opinion

 

Architecture

Model checking – the modules defined in the architecture interact with other modules due to their independent state machines. Modeling checking determines that the state machines for different modules interact properly – the architects build the model, the V&V person translates into the model checking language (if necessary)

 

 

Detailed Design

Model checking – same as above but on the more detailed behavior of threads/processes

 

 

Implementation

Unit testing – appropriate because at this level individual units are being defined the unit implementer is also the unit tester

 

 

Integration

Integration/system testing – units are being joined together to form larger units and the integrator is also the tester of the new unit

 

 

Deployment

System testing – some system tests are rerun to ensure that correct answers are still produced in the new environment.

 

  1. Assurance case
    1. What types of evidence can an assurance case provide related to the requirements for a product?
    2. How might a claim of “secure operation” be supported by evidence?
    3. How does the assurance case make use of the V&V process definition from question #3?
    4. What V&V techniques did we discuss that investigate the face validity of a model?

 

The assurance case could provide a comparison between a recognized domain model and the requirements to show completeness and correctness.

 

A scenario based analysis in which the most often encountered security threats are addressed one by one and traced through the architecture to see how they would be detected

 

The assurance case describes the chain of evidence by starting with activities that ensure correct, complete and consistent requirements and show that at each step V&V activities ensure that the correctness continues.

 

Guided Inspection provides an opportunity to establish face validity

  1. PSK

 

    1. Name a type of evidence that the PSK provides that would be useful in an assurance case. Explain how this evidence would be used in the assurance case. Into which section would it go? What would it assure?
    2. We simplified a number of elements of the vending machine program to be able to conduct some analyses. Is this still a valid validation of the original program? In what ways is it and in what ways is it not?
    3. VC-Gen creates VC, the verifying condition. What is the VC based on and why is the code certifiable if the VC is true?

 

Information about concurrency, namely livelock and deadlock. This assures that the computation completes. It is evidence.

 

The validation is valid for the level of analysis we did. It does not address detail level interactions but is still valid for the architecture.

 

The VC is based on the annotations. The verifying condition is the claim that the certification certifies.