Coverage Ws Overview 383bx
This document was ed by and they confirmed that they have the permission to share it. If you are author or own the copyright of this book, please report to us by using this report form. Report 3i3n4
Overview 26281t
& View Coverage Ws Overview as PDF for free.
More details 6y5l6z
- Words: 6,669
- Pages: 101
Incisive Coverage Introduction and RAK Overview This also includes Incisive 12.2 workshop/labs overview
© 2013 Cadence Design Systems, Inc.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary • Workshop – Click here -> Coverage Workshop 2
© 2013 Cadence Design Systems, Inc. All rights reserved.
Labs and DUT Introduction • This workshop includes hands-on labs – This is a verification environment for a simple UART DUT component that is part of a larger SoC system. We follow the steps of instrumenting, collecting, and analyzing the coverage. – The labs allow further understanding of the lectures. – To complete workshop with labs – click here - UART block: Converts parallel data from the APB bus into serial U data and vice versa
DMA
Memory
AHB Parallel APB bus
Bridge APB
APB
UART UART Serial UART protocol 3
UART Serial
© 2013 Cadence Design Systems, Inc. All rights reserved.
Core
USB
Ethernet
UART DUT and Verification Environment Module Verification Component
UVM Testbench
Scoreboard
Imitates external device by generating frames
Interface & white-box coverage
Programs the UART and transfers traffic
UART DUT (Verilog RTL) APB Verification Component
APB
Tx FIFO
txd
Transmitter
APB Interface
Mode Switch Rx FIFO
Receiver
Control/Status s
UART Serial
rxd
UART Verification Component
Control/Interrupt Logic
Multi-channel controller
Controls APB and UART components Test 1 4
© 2013 Cadence Design Systems, Inc. All rights reserved.
Test 2
Test 3
Tests indicate which stimulus sequences to execute
Functional Verification Methodologies Directed-Test Driven Results Test 1: Test 2: Test 3: Test 4:
DUT
x
Least effective in finding the hidden bugs
Test targets
Coverage Driven Results G
Coverage CG1 CG2 CG3 CG4
DUT
x
Adds quality & productivity, but difficult to estimate completion
Coverage targets
Metric Driven
Chip Features Results
G
Verif. Plan • Feature A • Feature B • Feature C • Feature D
5
DUT
A
B C D Feature-based Verification targets (Metrics for cov+checks)
© 2013 Cadence Design Systems, Inc. All rights reserved.
Feature-based plan with Feature A 50% extended Subset 1 …. 20%metrics Subset 2 …. 80% enables Feature B efficient Feature C 10% and accurate Feature D 70% 20% project closure
Metric-Driven Verification (MDV) Overview Planning with Unified Verification Metrics Metric-Based Executable Verification Plan
PDF
Yes
No
Signoff?
Done
Target Metrics Milestones Actual Metrics Achieved
VE Start
Module Set One
Module Set Two
Plan
Measure / Analyze
Successful Milestone Missed Milestone
Chip Integration
Prototype
Incisive VIP Portfolio
Construct
Production
Failure and Metric Analysis
Execute IEM
IFV IEV
ISX IES
SN
Testbench Simulation, Formal, HW/SW Co-Sim, LPV, MSV, Sim-Acceleration, Emulation 6
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Assertions Checks
Metric-Driven Verification Productivity Know Where You are Going and Get There Faster Focus and manage complex projects using Verification Planning
Functional and Design Specs
Create Verification Plan from Spec; Capturing Metrics
7
Find the most bugs early, and use analysis to identify and work on critical bugs first
vPlan
Scale Verification Throughput: Automated Parallel Verification Runs vs. Manual Engineering Work
© 2013 Cadence Design Systems, Inc. All rights reserved.
Analyze Results Debug, Fix, Resubmit
Coverage Is the Key for MDV • Coverage implemented to measure goals – Plan specifies Metrics required for DUT features. – After plan is developed, we need to instrument coverage into the verification environment.
• Coverage implemented to track progress – Plan provides feature-based tracking of progress – Regression results annotated back to Plan Features
• Let us dive deeper into coverage
8
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 9
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Metrics Code Coverage: method of assessing how well the test cases have exercised the design. Code Coverage : Block & Branch, Expression,Toggle, FSM Functional Coverage: focuses on functional aspects of a design and provides a very good insight on how the verification goals set by a test plan are being met. Functional Coverage : Assertion and CoverGroup
10
© 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage vs Functional Coverage •
Code coverage • Reflect how thoroughly the DUT code has been exercised • Enable detection of • Untested areas of DUT • Dead code or unused code • Most of the work are done by coverage tools automatically
APB UVC
•
Functional coverage • Measure what features of DUT were exercised • Lot of manual work: • Planning • What are the features? • Where/how to measure? • Coding of assertions and covergroups
11
© 2013 Cadence Design Systems, Inc. All rights reserved.
Scoreboard
DUT
UART UVC
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 12
© 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Block Coverage Identifies the lines of code that are executed during a simulation run. It helps you determine what areas of the DUT design are not exercised by the testbenches. A block is a statement or sequence of statements in Verilog/VHDL that executes with no branches or delays. Either none or all of the statements in a block are executed.
block coverage is an essential first step in the overall verification process.
13
© 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Block Coverage always @ (in ) begin $display("TRUE"); if ( in[1] == 1’b1 ) $display("IF1 TRUE"); else begin $display("IF1 FALSE"); $display("IF1 FALSE"); end; $display("TRUE"); if ( in[1] == 1’b1 ) $display("IF2 TRUE"); $display("TRUE"); end
BLK1
BLK2 BLK3
BLK4 BLK5 BLK6
Any construct that breaks execution flow creates a new block, for example: begin, if, else, case, wait, #, @, for, forever, repeat, while, disable 14
© 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Branch Coverage • Branch Coverage: Yields more precise coverage details than block coverage by obtaining coverage results for various branches individually. • With branch coverage, a piece of code is considered 100% covered when each branch of a conditional statement has been executed at least once.
assign out2 = (cond1) ? 1’b1 : 1’b0; The above statement has the following branches: - If cond1 evaluates to true, out2 is assigned 1’b1 - If cond1 evaluates to false, out2 is assigned 1’b0 Considered as a single block for block coverage. With branch coverage this statement is considered 100% covered when each branch of the statement has executed. 15
© 2013 Cadence Design Systems, Inc. All rights reserved.
Block Coverage Analysis Report
Notice that conditional assign statement was not scored with block coverage analysis
Score is 0 for the true part of if statement line 33 Score is 0 for the true part of if statement line 38
This block of statements was never executed This block of statements was never executed either
16
© 2013 Cadence Design Systems, Inc. All rights reserved.
Branch Coverage Analysis Report
With “Branch Coverage” enable, conditional assign statement is now scored.
17
© 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Expression Coverage • Expression coverage measures how thoroughly the testbench exercises expressions in assignments and procedural control constructs (if/case conditions). It identifies each input condition that makes the expression true or false and whether that condition happened in simulation.
Before scoring expression coverage, make sure you have high block coverage in your regression.
18
© 2013 Cadence Design Systems, Inc. All rights reserved.
3 types of Expression Coverage • SOP – Sum of Products Scoring (Default Mode) • Does each term take a 0 and non-0 value? • Vector inputs are scored as logical (single bit) • Control Scoring • Does each term take a 0 and non-0 value *AND* • Does each term control the output result of the expression during simulation? • Vector Scoring • Does each term take a 0 and non-0 value *AND* • Does each bit of each term control the result of the expression? You can set different scoring modes for selected modules in your design. 19
© 2013 Cadence Design Systems, Inc. All rights reserved.
SOP Scoring Reduces expressions to a minimum set of expression inputs that make the expression both true and false, inherently first-level b
&
<1> Simulation Scoring Count
Coverage Hole
(c
|
d)
<2>
hit
rval
1
1
1
<1>
<3> <2>
<3>
1
-
1
1
1
1
-
0
0
-
0
0
1
0
0
-
-
Resulting value of the expression (either zero or non-zero)
20
© 2013 Cadence Design Systems, Inc. All rights reserved.
Expression Inputs <1>: b <2>: c <3>: d
Control Scoring Checks if each input has controlled the output value of the expression at some time during the simulation − Known as “sensitized condition coverage” or “focused condition coverage” − Breaks an expression into a hierarchy of sub-expressions, more accurate hit b <1>
&
(c
|
d)
<--
2
-->
<3>
<4>
&
21
Level 2: <3>: c <4>: d
© 2013 Cadence Design Systems, Inc. All rights reserved.
<2>
--------------
0
0
1
0
1
0
1
1
1
hit
<3>
<4>
| Level 1: <1>: b <2>:(c | d)
<1>
--------------
0
0
1
0
1
0
1
0
0
Vector Scoring Vector scoring mode is an extension of control scoring mode. Each bit of a multi-bit signal is scored and reported separately and lots of data to analyze. hit
reg [3:0] a, b;
&&
Wire [3:0] val = (a && b);
(a
&& <1>
Level 1: <1>: a <2>: b
Bit table for each vector operand
22
© 2013 Cadence Design Systems, Inc. All rights reserved.
b) <2>
Hit
<1>
<2>
--------------
0
0
1
0
1
0
1
1
1
T1[3] T2[3] [2] [1] hit [2][0] [1] [0]
0
0
0
0 0 0 0 1 0
1
0
0
0
0 0 1 0 0 1
0
0
0
0
1 0 0 1 0 0
0
1
1
1
0 1 0 0 0 0
0
0
0
0
0 0 0 0 0 0
0
Expression Coverage Analysis Report Source code window & selected expression
Detailed expression reports
(v1 == 1) was never false (v2 == 1) was never false Need additional test cases?
23
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Metrics – Toggle Coverage • What does toggle coverage do? – Collect and report design signal toggle activity
• What is a design signal toggle? – Normally a binary transition (and return after a finite delay) of a DUT signal. – Signals may transition through (but not terminate at) an unknown state.
• Why bother with toggle coverage? – It’s the only “code” coverage available for a gate-level netlist – that design interconnect is connected and “wiggles”
1
Z 0 partial 24
© 2013 Cadence Design Systems, Inc. All rights reserved.
full
set_toggle_includez
Code Coverage : FSM Coverage • FSMs are critical control logic of DUT • It is important that FSMs logic are exercised thoroughly by the testbench to ensure that there are no bugs – FSM coverage measures how well this logic is exercised
FSM Coverage Types - State Coverage - Transition Coverage - Arc Coverage
25
: reports what states were visited : reports what transitions occurred : reports why each transition occured
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Code Coverage What coverage metrics should I enable? • Add in stages code coverage to identify holes in test environment – First-order: block or branch – Second-order: expression
• Add in stages the simplest form of functional coverage in the form of FSM coverage – First-order: states and transitions – Second-order: arcs
Note: There is little value in scoring second-order coverages if simpler ones like block or branch still show low coverage
26
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 27
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data Two-Step Simulation Design
+
Testbench Compile, Elaborate and instrument for coverage
coverage options OR coverage configuration file
irun -c
irun -r
28
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Database(s)
Simulate, score and store coverage
Generating Coverage Data IRUN coverage options -coverage
Enables coverage data generation •B
Enable Block coverage
•T
Enable Toggle coverage
•E
Enable Expression coverage
•U
Enable Functional coverage
•F
Enable FSM coverage
•A
Enable ALL coverage types
-covdut
Specify Module for instrumentation
-cov_file
Specify coverage Configuration File Instrument Block/Expression/FSM coverage for Module chipA & chipB % irun –f rtl_debug.f –coverage B:E:F –covdut chipA –covdut chipB
Instrument All coverage types for Module chipA % irun –f rtl_debug.f –coverage A –covdut chipA
29
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data irun runtime commands -covworkdir <workdir>
Basename for the work directory
Default work directory is cov_work -covscope <scope>
Specifies an alternate directory for storing coverage model files. Default is scope
-covtest
Test directory name. Default is test
-covoverwrite
Enable overwriting of coverage files
Invoke irun (during run time) with coverage instrumented during elaboration /home> irun -r <snapshot> Coverage model generated in /home/cov_work/scope (.ucm) Coverage data in /home/cov_work/design/test (.ucd) 30
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data Directory & file structure created for coverage Module Hierarchy
./
top
cov_work/ scope/
chipA
.ucm
mod2
testA/
mod1
.ucd testB/
.ucd % irun –c *.v –coverage B:E:T –covdut chipA
regr_10/
% irun –r top –covtest testA ...
scope/
.ucm
% irun -r top –covtest testB ...
testC/ .ucd 31
% irun –r top –covworkdir regr_10 –covtest testC
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data IRUN Coverage Summary of Commands
32
Switch name
Compile/elab/ runtime
Description
-coverage <string>
elaboration
Enable coverage instrumentation
-covfile
elaboration
Coverage instrumentation control file
-covdut
elaboration
Coverage DUT name
-covworkdir <workdir>
runtime
Coverage work directory
-covscope <scope>
runtime
Coverage scope name
-covtest
runtime
Coverage test name
-covoverwrite
runtime
Overwrite coverage output files
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data Defining a Coverage configuration file top chipA mod2 mod1 mod3
Enable B:E:T in chipA and all its descendents Matches all instances of mod and all of their descendents
covfile.ccf select_coverage –bet –module chipA…
Disable Toggle in module mod2… Enable Toggle in module mod3
deselect_coverage –t –module mod2… select_coverage –t –module mod3 # This is a comment
Score branches (ternary assignments)
set_branch_scoring
select_coverage -fsm –module mod1 select_functional
FSM Extraction in mod1 Score assertions / Covergroups
Run simulation with coverage according to configuration file covfile.ccf % irun *.v –cov_file covfile.ccf 33
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File • Coverage Configuration File Command
34
Command select_coverage
Applies to B:E:T:F
deselect_coverage
B:E:T:F
set_glitch_strobe set_hit_count_limit set_subprogram_scoring set_com set_com_interface set_code_fine_grained_m erging set_libcell_scoring set_merge_with_libname set_parameterized_modul e_coverage
B:E B:E B:E B:E:T B:E:T B:E ALL ALL ALL
© 2013 Cadence Design Systems, Inc. All rights reserved.
Description Select the design units or instances to score for coverage type Subtract from what is instrumented -remove_empty_instances remove empty instances from the coverage hierarchy Specify the glitch rejection time interval Sets the upper limit of hit counts Disable scoring of unused subprograms Identify items never happen and marks them ignored All ports of the modules will be marked as variables Provide merge resilience at concurrent block level instead of the default module/instance level Enable scoring of verilog library cells Match library name of module for merging Different modules dumped to different parameter values combinations used to instantiate the module
Coverage Configuration File • CCF Command apply only to Block Coverage Command
Description
set_implicit_block_scoring
set_branch_scoring
-off: disable scoring of implicit else and/or default case blocks -off: disable scoring of explicit else and/or default case blocks Enable scoring of verilog continous assignments Enable scoring of individual branches
set_statement_scoring
Enable statement coverage scoring
set_explicit_block_scoring set_assign_scoring
35
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File • CCF Command apply only to Expression Coverage Command
Description
set_expr_scoring
-all: enables scoring of all operators and assignments -sop/control/vector: specifies a scoring mode -event: enables scoring of verilog events -no_vhdl_shortcircuit: disables short-circuits evaluation of VHDL AND/NAND/OR/NOR operators -vlog_remove_redundancy: disables splitting of subexpressions -vhdl_not_as_operator: enables scoring of VHDL not operator set_expr_coverable_operators By default, coverage is scored only for Verilog logical operators (|| and &&) and VHDL logical operators (OR, AND, NOR, and NAND), and is scored only in condition expressions. This comand enables scoring coverage for all operators in condition expressions.
36
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File • CCF Command apply only to Toggle Coverage Command
Description
set_toggle_strobe
set_toggle_includex
Define a strobe interval for filtering glitches within the time window on nets Set maximum count for rise and fall transitions after which ncsim stop counting Record transitions x -> 0 and x -> 1.
set_toggle_includez
Record transitions z -> 0 and z -> 1.
set_toggle_noports
Disable scoring of module ports
set_toggle_portsonly
Score only module ports
set_toggle_excludefile
Exclude specific signals from the design during toggle simulation
set_toggle_limit
37
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File • CCF Command apply only to FSM Coverage Command
Description
set_fsm_attribute
Tag an FSM state vector
set_fsm_reset_scoring Enable scoring of reset states and transitions for identified FSMs. set_fsm_arc_scoring Enable scoring of arcs for identified FSMs. set_fsm_arc_termlimit set_fsm_scoring
38
Set the limit on the number of input created for an arc expression -hold_transition: enable scoring of FSM hold transitions.
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File • CCF Command apply only to Functional Coverage Command
Description
set_optimize
-ial_ovm_inst_asrt: treat instances of IAL and OVL components as individual assertions Enables scoring of all compiled assertions and covergroups -ams_control: enables scoring of assertions coverage in AMS modules Multiple switches that allow s to “fine tune” the collection and generation of coverage data; see ICC Guide for more details.
select_functional
set_covergroup
39
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 40
© 2013 Cadence Design Systems, Inc. All rights reserved.
Incisive Metrics Center (IMC) Key Features • Unified metrics center for all languages and all coverage metrics types – Code (block / expression / toggle) – FSM – Functional (Assertions / covergroup) • State of the art GUI capabilities to browse and analyze coverage – Instance and Type based views – Robust filtering and sorting capability – Intuitive page navigation • Unified batch mode Command Line Interface (CLI) • Good merge, rank, refinement and report(text, html) operations. • Exclusion
41
© 2013 Cadence Design Systems, Inc. All rights reserved.
2 modes to launch IMC GUI Mode: This invokes Incisive Metric Center, which is a graphical interface to observe coverage results, compare simulations, and refinement file % imc % imc -gui
Batch Mode: In this mode, imc command prompt is launched allowing to issue IMC commands interactively. Alternately, all commands can be placed in a
which is ed to imc for execution. % imc –batch [command_file]
42
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Help Utilities
“How To”s display topics available for search
Favorites page displays list of pages added as favorites
Search “How To”s
43
© 2013 Cadence Design Systems, Inc. All rights reserved.
Analysis History page displays list of runs launched in this or previous IMC sessions
IMC Summary View
Info Tab Sub-elements
“Activity Center” Page Navigation
Unified Instance/Type Verification Metrics Tree
Sorting / Filtering
Tools & Options 44
© 2013 Cadence Design Systems, Inc. All rights reserved.
Details Tab
IMC Block Analysis View Block Coverage Analysis
45
© 2013 Cadence Design Systems, Inc. All rights reserved.
Green Indicates covered ; Red indicates uncovered
IMC Expression Analysis View Various Expressions
Selected Expression
46
© 2013 Cadence Design Systems, Inc. All rights reserved.
Source Code
Selected Expression Table
IMC Toggle Analysis View
Individual Vector bits
47
© 2013 Cadence Design Systems, Inc. All rights reserved.
Source Code
IMC FSM Analysis View
State Transitions / Arcs
48
© 2013 Cadence Design Systems, Inc. All rights reserved.
FSM bubble diagram
Source viewer with syntax highlighting
Lab 1: Code Coverage
49
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 50
© 2013 Cadence Design Systems, Inc. All rights reserved.
Functional Coverage – What are we measuring? • Stimulus completeness – – – –
All address blocks Packet sizes: short, jumbo, max All instructions Important sequences – Read-after-write – Store* - Branch – Read*
Scoreboard
APB UVC
DUT
• Typically, implemented with SV covergroups • Where should we implement these covergroups? – In data items (uvm_sequence_item, uvm_sequence) directly? – In monitors? – In scoreboards?
51
© 2013 Cadence Design Systems, Inc. All rights reserved.
UART UVC
Implementing Coverage for Stimulus – Unique Data Item • If “data item definition” is unique for each DUT port: – Covergroup(s) can be defined in the data item itself – Type-coverage measures the completeness of the stimulus
Scoreboard
APB UVC
apb_transferDUT
UART uart_frame UVC
• In the example above, APB and UART ports have unique data items – Covergroups can be defined directly in the apb_transfer, uart_frame – Type-coverage of apb_transfer measures the apb stimulus coverage – Type-coverage of uart_frame measures the uart stimulus overage
52
© 2013 Cadence Design Systems, Inc. All rights reserved.
Scoreboard
Covergroup Example – in data item APB UVC
apb_transfer
DUT
Covergroup in apb_transfer data item
53
© 2013 Cadence Design Systems, Inc. All rights reserved.
UART UVC
uart_frame
Implementing Coverage for Stimulus – “Shared” Data Item • If “data item definition” is not unique for each DUT port: – Covergroup should be defined in the monitor or scoreboard – Instance-coverage measures the completeness of the stimulus
Scoreboard Agent 1
APB Port1 apb_transfer
Mon
APB UVC
DUT
Agent 2
UART
uart_frame UVC
APB Port2 apb_transfer
Mon
• In the example above, DUT has two APB ports – Do not define covergroup in the apb_transfer – Covergroup of APB stimulus should be defined in the monitor – Instance-coverage of monitor in Agent1 = stimulus coverage of APB Port1 – Instance-coverage of monitor in Agent2 = stimulus coverage of APB Port2
54
© 2013 Cadence Design Systems, Inc. All rights reserved.
Covergroup Example – in monitor Agent1
Scoreboard
apb_transfer
APB Port1
APB UVC Agent2
DUT
uart_frame
apb_transfer APB Port2
Enable instance coverage
Use set_inst_name() to specify meaningful instance name
55
© 2013 Cadence Design Systems, Inc. All rights reserved.
UART UVC
Explicit naming of covergroup instances • Do not rely on the default (tool-generated) name – Covergroup instance could have multiple paths (object handles) – ==> default name may not always be logical enough for s to follow
– Default name could be different between tests => merge issue
• Two techniques are available for naming the covergroup instances explicitly – Use of set_inst_name() built-in method – Use of option.name
• We recommend use of set_inst_name() for naming the covergroup instances explicitly
56
© 2013 Cadence Design Systems, Inc. All rights reserved.
Functional Coverage – What else are we measuring? • DUT features – Operational modes
Scoreboard
– 1GB, 10GB – 720p, 1080p
– Corner cases
UVC1
– FIFO empty/full – Wrong branch prediction -> flushes of prefetch queues
DUT
• Typically, implemented with covergroups and assertions • Where should we implement covergroups & assertions? – model (UVM_RGM or UVM_REG) – Monitors, scoreboards – vunit
57
© 2013 Cadence Design Systems, Inc. All rights reserved.
UVC2
Covergroup Example – in Model
Generated automatically by package
58
© 2013 Cadence Design Systems, Inc. All rights reserved.
Covergroup Example – FIFO Corner Cases
Covergroups to check for corner cases of FIFOs
59
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 60
© 2013 Cadence Design Systems, Inc. All rights reserved.
Building a Functional Coverage Model • Coverage sampling is expensive ‒ ‒
Determine minimum number of variables required to a design feature Don’t sample every clock; use sample event or call .sample()
• For each feature of the DUT: – Determine what variables we need to sample to this feature – Determine where to implement covergroups for these variables – In data items definition? – In monitors? – In scoreboards?
– Group the values of each sample variable into meaningful “bins” – Name them explicitly => easier for report analysis
61
© 2013 Cadence Design Systems, Inc. All rights reserved.
Building a Functional Coverage Model • For each feature of the DUT (continue): – To determine if certain condition occurred, you might need to cross the value of two or more variables. – These variables need to be in same covergroup – Cross coverage can generate a lot of data; use “ignore_bins” and “illegal_bins” – If you don’t do it now, you will have to “exclude” those during analysis phase
– Check the “Covergroup type options” (e.g., weight, goal) – Name the covergroup using set_inst_name() to help with analysis and merging • Review your coverage model with the design team to ensure accuracy
62
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 63
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Assertion Analysis View Assertion Finished / Failed Counts
Source Code
List of Assertions 64
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Covergroup Analysis View List of Covergroups
Coverpoints 65
© 2013 Cadence Design Systems, Inc. All rights reserved.
Bins
Incisive Metrics Center Search Capability Table search bar
Search on a tree
66
© 2013 Cadence Design Systems, Inc. All rights reserved.
Incisive Metrics Center Refinement Capability But can be hidden as well
Exclude items are gray out by default Exclusion Ability to save/read current state to/from a file
Exclude (recursively) Exclude Local (non-recursive)
Batch Mode also s ability to save/load refinement files Refinement can be applied on all metric kinds including block, Expression (sub-expressions), toggle signal (individual bits), Covergroup, Coverpoint, bins, cross-bins, assertions ,etc. 67
© 2013 Cadence Design Systems, Inc. All rights reserved.
Lab 2: Functional Coverage
68
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 69
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merging Coverage Data
Merging Coverage Data • Coverage analysis generally is performed on aggregated data from multiple tests run on one or more design configurations. • Coverage data must be merged to provide a comprehensive view of what is covered.
71
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merging Coverage Data with IMC “merge” command in IMC merge only ed in IMC batch mode ./ % imc -batch
cov_work/ scope/ test1/
imc> merge test1 test2 test3 –out all
OR imc> merge * –out all
test2/ test3/ all/
% imc -batch
./ regr_jan7_12 / system1/
72
72
IMC assumes tests are under cov_work/scope/
imc> merge regr_jan7_12/system1/* –out regr_jan7_12/system1/all
OR
test1/
imc> preferences –set workDir regr_jan7_12
test2/
imc> preferences –set scopeDir system1
test3/ all/
imc> merge * –out all
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Command merge -out
[-overwrite] {
| -runfile
} [-metrics <metrics_type>] [–run_order {as_specified | by_timestamp}] [–initial_model {empty |
| union_all | primary_run}] [-message [<message_level>]] [-covmodeldir <modeldir>]
73
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Command Initial Model • Initial Model affects how the Merged coverage model looks like • -initial_model –
:
coverage model used as the starting target model – primary_run: Primary test coverage model used as the starting target model. – Primary test is the first test used to merge (based on run_order)
– union_all – Target model is the union of all test coverage models
– empty – creates a target model – Based on: -target, -targettype, -source, -sourcetype
74
© 2013 Cadence Design Systems, Inc. All rights reserved.
Initial Model:
or primary_run
imc> merge Run1 Run2 –initial_model primary_run –out all
75
© 2013 Cadence Design Systems, Inc. All rights reserved.
Initial Model: union_all
imc> merge Run1 Run2 –initial_model union_all –out all 76
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – System to Block level merge
77
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – Block to System level merge
merge_config –sourcetype A –targettype A merge Run1 Run2 –initial_model Run1 –out merged_run merge_config –source System.A1 –targettype A merge_config –source System.A2 –targettype A merge Run1 Run2 –initial_model Run1 –out merged_run 78
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – Create New Hierarchy
79
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – Create New Hierarchy (one more)
80
© 2013 Cadence Design Systems, Inc. All rights reserved.
Ranking Coverage Data
Ranking runs in IMC • Ranking runs helps identify a subset of runs that have the lowest cost and still provide the same coverage grade as the full test suite. • By ranking the runs one can identify the runs that: – Make the most efficient test suite for regression testing eliminating the redundant runs. – Provide the best possible coverage considering the U time or the simulation time. – Help in achieving the target coverage goal.
• Ranking the runs involves: – Setting the Ranking Configuration using the rank_config command (optional) – Ranking the Runs using the rank command
• NOTE: Ranking runs does not require loading of runs.
82
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank_config Command • metric_type_weight_option – increases or decreases the weight of a given metric type. The default weight if not specified is 1 for all metrics. If weight is 0, the coverage metric type is not considered while ranking – – – – – –
-w_block <weight> -w_expression <weight> -w_toggle <weight> -w_fsm <weight> -w_assertion <weight> -w_covergroup <weight>
• -max
– specifies the number of runs after which ranking runs must stop – helps identify the best N runs.
• -target <cumulative_coverage_goal> – Specify target goal to be achieved while ranking – ranking stops when goal hit
83
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank_config command (continued) • -initial_runfile
– Specifies a runfile that contains a list of runs that must be included always in the ranking results. – The runs specified in the
appear above the runs specified for ranking.
• -u | -sim – Specifies if U time or Simulation time should be considered for ranking. By default, neither U time nor simulation time are considered when ranking
84
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command •
| -runfile
– lists runs to rank or place in
• -output
– Specifies the name of the output file where the ranking results are saved. By default ranking results are written to the standard out, unless redirected to a file
• -run_order {as_specified | by_timestamp} – specifies the order in which the runs are processed during model union stage – as_specified : process runs in order specified in command line or runfile. This is the default. – by_timestamp : process runs in reverse chronical order, the latest database is processed first
85
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command (continued) • -use_prev_merge – helps rank runs that were merged earlier
• rank_element_option – specifies the elements for which coverage grading is calculated during ranking. By default ranking is done for all the elements in the design. – -type
: specify the module/entity for which coverage grading are calculated during ranking – -inst
: specify instances for which coverage grading are calculated during ranking – -covergroup
: specify the covergroups for which coverage grading are calculated during ranking – -coverpoint
: specify the coverpoints for which coverage grading are calculated during ranking – -assertion
: specify the assertions for which coverage grading are calculated during ranking
86
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command examples Rank runs considering only the coverage grade of module/type ALU
• imc> rank test1 test2 test3 test4 type ALU
Rank runs to achieve • imc> rank_config -target 85 a cumulative coverage goal of • imc> rank test1 test2 test3 test4 85%
• imc> rank_config -w_block 0 Ignore block and w_expression 0 expression coverage • imc> rank test1 test2 test3 test4
87
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command examples (continued) imc>rank test1 test2 test3 test4 –out rank_result Doing model union of IUS runs ... Ranking With primary run (cov_work/scope/test1): cov_work/scope/icc_59616df2_5bf8faed.ucm configuration IUS target model generated successfully. Coverage ranking options: ============================================================================ Rank elements: Model Weight: Block = 1, Expression = 1, Toggle = 1, Fsm = 1, Assertion = 1, Covergroup = 1 Ranking Cost: N.A Target Cumulative Grade(%): 100 Number of Best Runs: 4 Ranking of coverage runs:
Optimized Runs
Run Cum. Self Cum. Wt Self Wt Contrib. Run Id Grade(%) Grade(%) Grade(%) Grade(%) # items Cost Name ============================================================================ Optimized Runs: Redundant Runs 1 58.18 58.18 58.18 58.18 6634 0 cov_work/scope/test2 2 73.29 64.32 73.29 64.32 500 0 cov_work/scope/test1 Redundant Runs: 3 0.00 58.18 0.00 58.18 0 0 cov_work/scope/test3 4 0.00 58.18 0.00 58.18 0 0 cov_work/scope/test4
88
© 2013 Cadence Design Systems, Inc. All rights reserved.
HTML reporting
Analyzing Coverage Data Generating Text Reports in IMC % imc -batch
Load run “all” containing merged data from all tests imc> load -run all
Optional: Load refinement file (if any) imc> load –refinement my_allexclude.vRefine
Generate Summary Report for all instances imc> report -summary –inst [-html] –out myreportsum.html *…
Generate Detailed Report for all instances imc> report -detailed –inst [–html] -out myreport.det *…
90
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC: HTML reports
Example of a report – Top Level Summary Key data
Concise useful HELP is one click away
High-level summary Total (average)
Navigate hierarchy
Coverage Numbers Link to flat CoverGroup coverage – all CoverGroups in one table
91
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC: HTML reports Example of a report - Summary
Click to open detailed report for instance
Click column header to sort table
Click link to see details 92
© 2013 Cadence Design Systems, Inc. All rights reserved.
Click instance to descend
IMC: HTML reports Example of Global CoverGroup Summary Report
Actual coverage
Covergroup options
93
© 2013 Cadence Design Systems, Inc. All rights reserved.
Bin details
Covergroup instance name
IMC: HTML reports Example of Per Covergroup Coverage
Click on item to view details
94
© 2013 Cadence Design Systems, Inc. All rights reserved.
Lab 3 – Coverage Analysis
95
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 96
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Guidelines Summary – Page 1 • Start with verification planning • Functional coverage design – For each design feature: – Determine minimum number of variables required – Select sampling event carefully – Check the “Covergroup type options” (e.g., weight, goal) to ensure correct grading – Name the covergroup using set_inst_name()
• Running tests and collect functional coverage • Once functional coverage reaches target & design is stable, start code coverage
97
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Guidelines Summary – Page 2 • Code coverage – Start with Block coverage + FSM coverage – Add branch coverage – Expression coverage for critical blocks; work with designers – Is SOP good enough? – If not, work with designers to determine if Control or Vector is required – Toggle coverage for interfaces between sub-blocks
• What about SOC level? – Functional coverage determined by the SOC verification plan – What about code coverage? – Since the concerns here are: interconnect between blocks and “glue” logic – Toggle coverage for interconnect between blocks – Block/branch/expression? coverage for “glue” logic
98
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Guidelines Summary – Page 3
• Be selective about what you measure & collect – Capture only what you really need otherwise…. – …you might have to spend lot of time analyzing results and writing exclusions!
99
© 2013 Cadence Design Systems, Inc. All rights reserved.
Additional Documentation / Tutorial IMC Guide:
/doc/imc/imc.pdf IMC Quick Reference Guide:
/doc/imcug/imcqrg.pdf ICC Guide:
/doc/iccug/iccug.pdf SOC and IP level Functional Verification - Rapid Adoption Kits (RAK) Landing Page To this workshop – Click here -> Coverage Workshop 100
© 2013 Cadence Design Systems, Inc. All rights reserved.
http://.cadence.com http://www.cadence.com
© 2013 Cadence Design Systems, Inc.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary • Workshop – Click here -> Coverage Workshop 2
© 2013 Cadence Design Systems, Inc. All rights reserved.
Labs and DUT Introduction • This workshop includes hands-on labs – This is a verification environment for a simple UART DUT component that is part of a larger SoC system. We follow the steps of instrumenting, collecting, and analyzing the coverage. – The labs allow further understanding of the lectures. – To complete workshop with labs – click here - UART block: Converts parallel data from the APB bus into serial U data and vice versa
DMA
Memory
AHB Parallel APB bus
Bridge APB
APB
UART UART Serial UART protocol 3
UART Serial
© 2013 Cadence Design Systems, Inc. All rights reserved.
Core
USB
Ethernet
UART DUT and Verification Environment Module Verification Component
UVM Testbench
Scoreboard
Imitates external device by generating frames
Interface & white-box coverage
Programs the UART and transfers traffic
UART DUT (Verilog RTL) APB Verification Component
APB
Tx FIFO
txd
Transmitter
APB Interface
Mode Switch Rx FIFO
Receiver
Control/Status s
UART Serial
rxd
UART Verification Component
Control/Interrupt Logic
Multi-channel controller
Controls APB and UART components Test 1 4
© 2013 Cadence Design Systems, Inc. All rights reserved.
Test 2
Test 3
Tests indicate which stimulus sequences to execute
Functional Verification Methodologies Directed-Test Driven Results Test 1: Test 2: Test 3: Test 4:
DUT
x
Least effective in finding the hidden bugs
Test targets
Coverage Driven Results G
Coverage CG1 CG2 CG3 CG4
DUT
x
Adds quality & productivity, but difficult to estimate completion
Coverage targets
Metric Driven
Chip Features Results
G
Verif. Plan • Feature A • Feature B • Feature C • Feature D
5
DUT
A
B C D Feature-based Verification targets (Metrics for cov+checks)
© 2013 Cadence Design Systems, Inc. All rights reserved.
Feature-based plan with Feature A 50% extended Subset 1 …. 20%metrics Subset 2 …. 80% enables Feature B efficient Feature C 10% and accurate Feature D 70% 20% project closure
Metric-Driven Verification (MDV) Overview Planning with Unified Verification Metrics Metric-Based Executable Verification Plan
Yes
No
Signoff?
Done
Target Metrics Milestones Actual Metrics Achieved
VE Start
Module Set One
Module Set Two
Plan
Measure / Analyze
Successful Milestone Missed Milestone
Chip Integration
Prototype
Incisive VIP Portfolio
Construct
Production
Failure and Metric Analysis
Execute IEM
IFV IEV
ISX IES
SN
Testbench Simulation, Formal, HW/SW Co-Sim, LPV, MSV, Sim-Acceleration, Emulation 6
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Assertions Checks
Metric-Driven Verification Productivity Know Where You are Going and Get There Faster Focus and manage complex projects using Verification Planning
Functional and Design Specs
Create Verification Plan from Spec; Capturing Metrics
7
Find the most bugs early, and use analysis to identify and work on critical bugs first
vPlan
Scale Verification Throughput: Automated Parallel Verification Runs vs. Manual Engineering Work
© 2013 Cadence Design Systems, Inc. All rights reserved.
Analyze Results Debug, Fix, Resubmit
Coverage Is the Key for MDV • Coverage implemented to measure goals – Plan specifies Metrics required for DUT features. – After plan is developed, we need to instrument coverage into the verification environment.
• Coverage implemented to track progress – Plan provides feature-based tracking of progress – Regression results annotated back to Plan Features
• Let us dive deeper into coverage
8
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 9
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Metrics Code Coverage: method of assessing how well the test cases have exercised the design. Code Coverage : Block & Branch, Expression,Toggle, FSM Functional Coverage: focuses on functional aspects of a design and provides a very good insight on how the verification goals set by a test plan are being met. Functional Coverage : Assertion and CoverGroup
10
© 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage vs Functional Coverage •
Code coverage • Reflect how thoroughly the DUT code has been exercised • Enable detection of • Untested areas of DUT • Dead code or unused code • Most of the work are done by coverage tools automatically
APB UVC
•
Functional coverage • Measure what features of DUT were exercised • Lot of manual work: • Planning • What are the features? • Where/how to measure? • Coding of assertions and covergroups
11
© 2013 Cadence Design Systems, Inc. All rights reserved.
Scoreboard
DUT
UART UVC
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 12
© 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Block Coverage Identifies the lines of code that are executed during a simulation run. It helps you determine what areas of the DUT design are not exercised by the testbenches. A block is a statement or sequence of statements in Verilog/VHDL that executes with no branches or delays. Either none or all of the statements in a block are executed.
block coverage is an essential first step in the overall verification process.
13
© 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Block Coverage always @ (in ) begin $display("TRUE"); if ( in[1] == 1’b1 ) $display("IF1 TRUE"); else begin $display("IF1 FALSE"); $display("IF1 FALSE"); end; $display("TRUE"); if ( in[1] == 1’b1 ) $display("IF2 TRUE"); $display("TRUE"); end
BLK1
BLK2 BLK3
BLK4 BLK5 BLK6
Any construct that breaks execution flow creates a new block, for example: begin, if, else, case, wait, #, @, for, forever, repeat, while, disable 14
© 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Branch Coverage • Branch Coverage: Yields more precise coverage details than block coverage by obtaining coverage results for various branches individually. • With branch coverage, a piece of code is considered 100% covered when each branch of a conditional statement has been executed at least once.
assign out2 = (cond1) ? 1’b1 : 1’b0; The above statement has the following branches: - If cond1 evaluates to true, out2 is assigned 1’b1 - If cond1 evaluates to false, out2 is assigned 1’b0 Considered as a single block for block coverage. With branch coverage this statement is considered 100% covered when each branch of the statement has executed. 15
© 2013 Cadence Design Systems, Inc. All rights reserved.
Block Coverage Analysis Report
Notice that conditional assign statement was not scored with block coverage analysis
Score is 0 for the true part of if statement line 33 Score is 0 for the true part of if statement line 38
This block of statements was never executed This block of statements was never executed either
16
© 2013 Cadence Design Systems, Inc. All rights reserved.
Branch Coverage Analysis Report
With “Branch Coverage” enable, conditional assign statement is now scored.
17
© 2013 Cadence Design Systems, Inc. All rights reserved.
Code Coverage : Expression Coverage • Expression coverage measures how thoroughly the testbench exercises expressions in assignments and procedural control constructs (if/case conditions). It identifies each input condition that makes the expression true or false and whether that condition happened in simulation.
Before scoring expression coverage, make sure you have high block coverage in your regression.
18
© 2013 Cadence Design Systems, Inc. All rights reserved.
3 types of Expression Coverage • SOP – Sum of Products Scoring (Default Mode) • Does each term take a 0 and non-0 value? • Vector inputs are scored as logical (single bit) • Control Scoring • Does each term take a 0 and non-0 value *AND* • Does each term control the output result of the expression during simulation? • Vector Scoring • Does each term take a 0 and non-0 value *AND* • Does each bit of each term control the result of the expression? You can set different scoring modes for selected modules in your design. 19
© 2013 Cadence Design Systems, Inc. All rights reserved.
SOP Scoring Reduces expressions to a minimum set of expression inputs that make the expression both true and false, inherently first-level b
&
<1> Simulation Scoring Count
Coverage Hole
(c
|
d)
<2>
hit
rval
1
1
1
<1>
<3> <2>
<3>
1
-
1
1
1
1
-
0
0
-
0
0
1
0
0
-
-
Resulting value of the expression (either zero or non-zero)
20
© 2013 Cadence Design Systems, Inc. All rights reserved.
Expression Inputs <1>: b <2>: c <3>: d
Control Scoring Checks if each input has controlled the output value of the expression at some time during the simulation − Known as “sensitized condition coverage” or “focused condition coverage” − Breaks an expression into a hierarchy of sub-expressions, more accurate hit b <1>
&
(c
|
d)
<--
2
-->
<3>
<4>
&
21
Level 2: <3>: c <4>: d
© 2013 Cadence Design Systems, Inc. All rights reserved.
<2>
--------------
0
0
1
0
1
0
1
1
1
hit
<3>
<4>
| Level 1: <1>: b <2>:(c | d)
<1>
--------------
0
0
1
0
1
0
1
0
0
Vector Scoring Vector scoring mode is an extension of control scoring mode. Each bit of a multi-bit signal is scored and reported separately and lots of data to analyze. hit
reg [3:0] a, b;
&&
Wire [3:0] val = (a && b);
(a
&& <1>
Level 1: <1>: a <2>: b
Bit table for each vector operand
22
© 2013 Cadence Design Systems, Inc. All rights reserved.
b) <2>
Hit
<1>
<2>
--------------
0
0
1
0
1
0
1
1
1
T1[3] T2[3] [2] [1] hit [2][0] [1] [0]
0
0
0
0 0 0 0 1 0
1
0
0
0
0 0 1 0 0 1
0
0
0
0
1 0 0 1 0 0
0
1
1
1
0 1 0 0 0 0
0
0
0
0
0 0 0 0 0 0
0
Expression Coverage Analysis Report Source code window & selected expression
Detailed expression reports
(v1 == 1) was never false (v2 == 1) was never false Need additional test cases?
23
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Metrics – Toggle Coverage • What does toggle coverage do? – Collect and report design signal toggle activity
• What is a design signal toggle? – Normally a binary transition (and return after a finite delay) of a DUT signal. – Signals may transition through (but not terminate at) an unknown state.
• Why bother with toggle coverage? – It’s the only “code” coverage available for a gate-level netlist – that design interconnect is connected and “wiggles”
1
Z 0 partial 24
© 2013 Cadence Design Systems, Inc. All rights reserved.
full
set_toggle_includez
Code Coverage : FSM Coverage • FSMs are critical control logic of DUT • It is important that FSMs logic are exercised thoroughly by the testbench to ensure that there are no bugs – FSM coverage measures how well this logic is exercised
FSM Coverage Types - State Coverage - Transition Coverage - Arc Coverage
25
: reports what states were visited : reports what transitions occurred : reports why each transition occured
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Code Coverage What coverage metrics should I enable? • Add in stages code coverage to identify holes in test environment – First-order: block or branch – Second-order: expression
• Add in stages the simplest form of functional coverage in the form of FSM coverage – First-order: states and transitions – Second-order: arcs
Note: There is little value in scoring second-order coverages if simpler ones like block or branch still show low coverage
26
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 27
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data Two-Step Simulation Design
+
Testbench Compile, Elaborate and instrument for coverage
coverage options OR coverage configuration file
irun -c
irun -r
28
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Database(s)
Simulate, score and store coverage
Generating Coverage Data IRUN coverage options -coverage
Enable Block coverage
•T
Enable Toggle coverage
•E
Enable Expression coverage
•U
Enable Functional coverage
•F
Enable FSM coverage
•A
Enable ALL coverage types
-covdut
-cov_file
Instrument All coverage types for Module chipA % irun –f rtl_debug.f –coverage A –covdut chipA
29
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data irun runtime commands -covworkdir <workdir>
Basename for the work directory
Default work directory is cov_work -covscope <scope>
Specifies an alternate directory for storing coverage model files. Default is scope
-covtest
Test directory name. Default is test
-covoverwrite
Enable overwriting of coverage files
Invoke irun (during run time) with coverage instrumented during elaboration /home> irun -r <snapshot> Coverage model generated in /home/cov_work/scope (.ucm) Coverage data in /home/cov_work/design/test (.ucd) 30
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data Directory & file structure created for coverage Module Hierarchy
./
top
cov_work/ scope/
chipA
.ucm
mod2
testA/
mod1
.ucd testB/
.ucd % irun –c *.v –coverage B:E:T –covdut chipA
regr_10/
% irun –r top –covtest testA ...
scope/
.ucm
% irun -r top –covtest testB ...
testC/ .ucd 31
% irun –r top –covworkdir regr_10 –covtest testC
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data IRUN Coverage Summary of Commands
32
Switch name
Compile/elab/ runtime
Description
-coverage <string>
elaboration
Enable coverage instrumentation
-covfile
elaboration
Coverage instrumentation control file
-covdut
elaboration
Coverage DUT name
-covworkdir <workdir>
runtime
Coverage work directory
-covscope <scope>
runtime
Coverage scope name
-covtest
runtime
Coverage test name
-covoverwrite
runtime
Overwrite coverage output files
© 2013 Cadence Design Systems, Inc. All rights reserved.
Generating Coverage Data Defining a Coverage configuration file top chipA mod2 mod1 mod3
Enable B:E:T in chipA and all its descendents Matches all instances of mod and all of their descendents
covfile.ccf select_coverage –bet –module chipA…
Disable Toggle in module mod2… Enable Toggle in module mod3
deselect_coverage –t –module mod2… select_coverage –t –module mod3 # This is a comment
Score branches (ternary assignments)
set_branch_scoring
select_coverage -fsm –module mod1 select_functional
FSM Extraction in mod1 Score assertions / Covergroups
Run simulation with coverage according to configuration file covfile.ccf % irun *.v –cov_file covfile.ccf 33
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File • Coverage Configuration File Command
34
Command select_coverage
Applies to B:E:T:F
deselect_coverage
B:E:T:F
set_glitch_strobe set_hit_count_limit set_subprogram_scoring set_com set_com_interface set_code_fine_grained_m erging set_libcell_scoring set_merge_with_libname set_parameterized_modul e_coverage
B:E B:E B:E B:E:T B:E:T B:E ALL ALL ALL
© 2013 Cadence Design Systems, Inc. All rights reserved.
Description Select the design units or instances to score for coverage type Subtract from what is instrumented -remove_empty_instances remove empty instances from the coverage hierarchy Specify the glitch rejection time interval Sets the upper limit of hit counts Disable scoring of unused subprograms Identify items never happen and marks them ignored All ports of the modules will be marked as variables Provide merge resilience at concurrent block level instead of the default module/instance level Enable scoring of verilog library cells Match library name of module for merging Different modules dumped to different parameter values combinations used to instantiate the module
Coverage Configuration File • CCF Command apply only to Block Coverage Command
Description
set_implicit_block_scoring
set_branch_scoring
-off: disable scoring of implicit else and/or default case blocks -off: disable scoring of explicit else and/or default case blocks Enable scoring of verilog continous assignments Enable scoring of individual branches
set_statement_scoring
Enable statement coverage scoring
set_explicit_block_scoring set_assign_scoring
35
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File • CCF Command apply only to Expression Coverage Command
Description
set_expr_scoring
-all: enables scoring of all operators and assignments -sop/control/vector: specifies a scoring mode -event: enables scoring of verilog events -no_vhdl_shortcircuit: disables short-circuits evaluation of VHDL AND/NAND/OR/NOR operators -vlog_remove_redundancy: disables splitting of subexpressions -vhdl_not_as_operator: enables scoring of VHDL not operator set_expr_coverable_operators By default, coverage is scored only for Verilog logical operators (|| and &&) and VHDL logical operators (OR, AND, NOR, and NAND), and is scored only in condition expressions. This comand enables scoring coverage for all operators in condition expressions.
36
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File • CCF Command apply only to Toggle Coverage Command
Description
set_toggle_strobe
set_toggle_includex
Define a strobe interval for filtering glitches within the time window on nets Set maximum count for rise and fall transitions after which ncsim stop counting Record transitions x -> 0 and x -> 1.
set_toggle_includez
Record transitions z -> 0 and z -> 1.
set_toggle_noports
Disable scoring of module ports
set_toggle_portsonly
Score only module ports
set_toggle_excludefile
Exclude specific signals from the design during toggle simulation
set_toggle_limit
37
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File • CCF Command apply only to FSM Coverage Command
Description
set_fsm_attribute
Tag an FSM state vector
set_fsm_reset_scoring Enable scoring of reset states and transitions for identified FSMs. set_fsm_arc_scoring Enable scoring of arcs for identified FSMs. set_fsm_arc_termlimit set_fsm_scoring
38
Set the limit on the number of input created for an arc expression -hold_transition: enable scoring of FSM hold transitions.
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Configuration File • CCF Command apply only to Functional Coverage Command
Description
set_optimize
-ial_ovm_inst_asrt: treat instances of IAL and OVL components as individual assertions Enables scoring of all compiled assertions and covergroups -ams_control: enables scoring of assertions coverage in AMS modules Multiple switches that allow s to “fine tune” the collection and generation of coverage data; see ICC Guide for more details.
select_functional
set_covergroup
39
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 40
© 2013 Cadence Design Systems, Inc. All rights reserved.
Incisive Metrics Center (IMC) Key Features • Unified metrics center for all languages and all coverage metrics types – Code (block / expression / toggle) – FSM – Functional (Assertions / covergroup) • State of the art GUI capabilities to browse and analyze coverage – Instance and Type based views – Robust filtering and sorting capability – Intuitive page navigation • Unified batch mode Command Line Interface (CLI) • Good merge, rank, refinement and report(text, html) operations. • Exclusion
41
© 2013 Cadence Design Systems, Inc. All rights reserved.
2 modes to launch IMC GUI Mode: This invokes Incisive Metric Center, which is a graphical interface to observe coverage results, compare simulations, and refinement file % imc % imc -gui
Batch Mode: In this mode, imc command prompt is launched allowing to issue IMC commands interactively. Alternately, all commands can be placed in a
42
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Help Utilities
“How To”s display topics available for search
Favorites page displays list of pages added as favorites
Search “How To”s
43
© 2013 Cadence Design Systems, Inc. All rights reserved.
Analysis History page displays list of runs launched in this or previous IMC sessions
IMC Summary View
Info Tab Sub-elements
“Activity Center” Page Navigation
Unified Instance/Type Verification Metrics Tree
Sorting / Filtering
Tools & Options 44
© 2013 Cadence Design Systems, Inc. All rights reserved.
Details Tab
IMC Block Analysis View Block Coverage Analysis
45
© 2013 Cadence Design Systems, Inc. All rights reserved.
Green Indicates covered ; Red indicates uncovered
IMC Expression Analysis View Various Expressions
Selected Expression
46
© 2013 Cadence Design Systems, Inc. All rights reserved.
Source Code
Selected Expression Table
IMC Toggle Analysis View
Individual Vector bits
47
© 2013 Cadence Design Systems, Inc. All rights reserved.
Source Code
IMC FSM Analysis View
State Transitions / Arcs
48
© 2013 Cadence Design Systems, Inc. All rights reserved.
FSM bubble diagram
Source viewer with syntax highlighting
Lab 1: Code Coverage
49
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 50
© 2013 Cadence Design Systems, Inc. All rights reserved.
Functional Coverage – What are we measuring? • Stimulus completeness – – – –
All address blocks Packet sizes: short, jumbo, max All instructions Important sequences – Read-after-write – Store* - Branch – Read*
Scoreboard
APB UVC
DUT
• Typically, implemented with SV covergroups • Where should we implement these covergroups? – In data items (uvm_sequence_item, uvm_sequence) directly? – In monitors? – In scoreboards?
51
© 2013 Cadence Design Systems, Inc. All rights reserved.
UART UVC
Implementing Coverage for Stimulus – Unique Data Item • If “data item definition” is unique for each DUT port: – Covergroup(s) can be defined in the data item itself – Type-coverage measures the completeness of the stimulus
Scoreboard
APB UVC
apb_transferDUT
UART uart_frame UVC
• In the example above, APB and UART ports have unique data items – Covergroups can be defined directly in the apb_transfer, uart_frame – Type-coverage of apb_transfer measures the apb stimulus coverage – Type-coverage of uart_frame measures the uart stimulus overage
52
© 2013 Cadence Design Systems, Inc. All rights reserved.
Scoreboard
Covergroup Example – in data item APB UVC
apb_transfer
DUT
Covergroup in apb_transfer data item
53
© 2013 Cadence Design Systems, Inc. All rights reserved.
UART UVC
uart_frame
Implementing Coverage for Stimulus – “Shared” Data Item • If “data item definition” is not unique for each DUT port: – Covergroup should be defined in the monitor or scoreboard – Instance-coverage measures the completeness of the stimulus
Scoreboard Agent 1
APB Port1 apb_transfer
Mon
APB UVC
DUT
Agent 2
UART
uart_frame UVC
APB Port2 apb_transfer
Mon
• In the example above, DUT has two APB ports – Do not define covergroup in the apb_transfer – Covergroup of APB stimulus should be defined in the monitor – Instance-coverage of monitor in Agent1 = stimulus coverage of APB Port1 – Instance-coverage of monitor in Agent2 = stimulus coverage of APB Port2
54
© 2013 Cadence Design Systems, Inc. All rights reserved.
Covergroup Example – in monitor Agent1
Scoreboard
apb_transfer
APB Port1
APB UVC Agent2
DUT
uart_frame
apb_transfer APB Port2
Enable instance coverage
Use set_inst_name() to specify meaningful instance name
55
© 2013 Cadence Design Systems, Inc. All rights reserved.
UART UVC
Explicit naming of covergroup instances • Do not rely on the default (tool-generated) name – Covergroup instance could have multiple paths (object handles) – ==> default name may not always be logical enough for s to follow
– Default name could be different between tests => merge issue
• Two techniques are available for naming the covergroup instances explicitly – Use of set_inst_name() built-in method – Use of option.name
• We recommend use of set_inst_name() for naming the covergroup instances explicitly
56
© 2013 Cadence Design Systems, Inc. All rights reserved.
Functional Coverage – What else are we measuring? • DUT features – Operational modes
Scoreboard
– 1GB, 10GB – 720p, 1080p
– Corner cases
UVC1
– FIFO empty/full – Wrong branch prediction -> flushes of prefetch queues
DUT
• Typically, implemented with covergroups and assertions • Where should we implement covergroups & assertions? – model (UVM_RGM or UVM_REG) – Monitors, scoreboards – vunit
57
© 2013 Cadence Design Systems, Inc. All rights reserved.
UVC2
Covergroup Example – in Model
Generated automatically by package
58
© 2013 Cadence Design Systems, Inc. All rights reserved.
Covergroup Example – FIFO Corner Cases
Covergroups to check for corner cases of FIFOs
59
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 60
© 2013 Cadence Design Systems, Inc. All rights reserved.
Building a Functional Coverage Model • Coverage sampling is expensive ‒ ‒
Determine minimum number of variables required to a design feature Don’t sample every clock; use sample event or call .sample()
• For each feature of the DUT: – Determine what variables we need to sample to this feature – Determine where to implement covergroups for these variables – In data items definition? – In monitors? – In scoreboards?
– Group the values of each sample variable into meaningful “bins” – Name them explicitly => easier for report analysis
61
© 2013 Cadence Design Systems, Inc. All rights reserved.
Building a Functional Coverage Model • For each feature of the DUT (continue): – To determine if certain condition occurred, you might need to cross the value of two or more variables. – These variables need to be in same covergroup – Cross coverage can generate a lot of data; use “ignore_bins” and “illegal_bins” – If you don’t do it now, you will have to “exclude” those during analysis phase
– Check the “Covergroup type options” (e.g., weight, goal) – Name the covergroup using set_inst_name() to help with analysis and merging • Review your coverage model with the design team to ensure accuracy
62
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 63
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Assertion Analysis View Assertion Finished / Failed Counts
Source Code
List of Assertions 64
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC Covergroup Analysis View List of Covergroups
Coverpoints 65
© 2013 Cadence Design Systems, Inc. All rights reserved.
Bins
Incisive Metrics Center Search Capability Table search bar
Search on a tree
66
© 2013 Cadence Design Systems, Inc. All rights reserved.
Incisive Metrics Center Refinement Capability But can be hidden as well
Exclude items are gray out by default Exclusion Ability to save/read current state to/from a file
Exclude (recursively) Exclude Local (non-recursive)
Batch Mode also s ability to save/load refinement files Refinement can be applied on all metric kinds including block, Expression (sub-expressions), toggle signal (individual bits), Covergroup, Coverpoint, bins, cross-bins, assertions ,etc. 67
© 2013 Cadence Design Systems, Inc. All rights reserved.
Lab 2: Functional Coverage
68
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 69
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merging Coverage Data
Merging Coverage Data • Coverage analysis generally is performed on aggregated data from multiple tests run on one or more design configurations. • Coverage data must be merged to provide a comprehensive view of what is covered.
71
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merging Coverage Data with IMC “merge” command in IMC merge only ed in IMC batch mode ./ % imc -batch
cov_work/ scope/ test1/
imc> merge test1 test2 test3 –out all
OR imc> merge * –out all
test2/ test3/ all/
% imc -batch
./ regr_jan7_12 / system1/
72
72
IMC assumes tests are under cov_work/scope/
imc> merge regr_jan7_12/system1/* –out regr_jan7_12/system1/all
OR
test1/
imc> preferences –set workDir regr_jan7_12
test2/
imc> preferences –set scopeDir system1
test3/ all/
imc> merge * –out all
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Command merge -out
73
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Command Initial Model • Initial Model affects how the Merged coverage model looks like • -initial_model –
– union_all – Target model is the union of all test coverage models
– empty – creates a target model – Based on: -target, -targettype, -source, -sourcetype
74
© 2013 Cadence Design Systems, Inc. All rights reserved.
Initial Model:
imc> merge Run1 Run2 –initial_model primary_run –out all
75
© 2013 Cadence Design Systems, Inc. All rights reserved.
Initial Model: union_all
imc> merge Run1 Run2 –initial_model union_all –out all 76
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – System to Block level merge
77
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – Block to System level merge
merge_config –sourcetype A –targettype A merge Run1 Run2 –initial_model Run1 –out merged_run merge_config –source System.A1 –targettype A merge_config –source System.A2 –targettype A merge Run1 Run2 –initial_model Run1 –out merged_run 78
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – Create New Hierarchy
79
© 2013 Cadence Design Systems, Inc. All rights reserved.
Merge Config – Create New Hierarchy (one more)
80
© 2013 Cadence Design Systems, Inc. All rights reserved.
Ranking Coverage Data
Ranking runs in IMC • Ranking runs helps identify a subset of runs that have the lowest cost and still provide the same coverage grade as the full test suite. • By ranking the runs one can identify the runs that: – Make the most efficient test suite for regression testing eliminating the redundant runs. – Provide the best possible coverage considering the U time or the simulation time. – Help in achieving the target coverage goal.
• Ranking the runs involves: – Setting the Ranking Configuration using the rank_config command (optional) – Ranking the Runs using the rank command
• NOTE: Ranking runs does not require loading of runs.
82
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank_config Command • metric_type_weight_option – increases or decreases the weight of a given metric type. The default weight if not specified is 1 for all metrics. If weight is 0, the coverage metric type is not considered while ranking – – – – – –
-w_block <weight> -w_expression <weight> -w_toggle <weight> -w_fsm <weight> -w_assertion <weight> -w_covergroup <weight>
• -max
• -target <cumulative_coverage_goal> – Specify target goal to be achieved while ranking – ranking stops when goal hit
83
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank_config command (continued) • -initial_runfile
• -u | -sim – Specifies if U time or Simulation time should be considered for ranking. By default, neither U time nor simulation time are considered when ranking
84
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command •
• -output
• -run_order {as_specified | by_timestamp} – specifies the order in which the runs are processed during model union stage – as_specified : process runs in order specified in command line or runfile. This is the default. – by_timestamp : process runs in reverse chronical order, the latest database is processed first
85
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command (continued) • -use_prev_merge – helps rank runs that were merged earlier
• rank_element_option – specifies the elements for which coverage grading is calculated during ranking. By default ranking is done for all the elements in the design. – -type
86
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command examples Rank runs considering only the coverage grade of module/type ALU
• imc> rank test1 test2 test3 test4 type ALU
Rank runs to achieve • imc> rank_config -target 85 a cumulative coverage goal of • imc> rank test1 test2 test3 test4 85%
• imc> rank_config -w_block 0 Ignore block and w_expression 0 expression coverage • imc> rank test1 test2 test3 test4
87
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC rank command examples (continued) imc>rank test1 test2 test3 test4 –out rank_result Doing model union of IUS runs ... Ranking With primary run (cov_work/scope/test1): cov_work/scope/icc_59616df2_5bf8faed.ucm configuration IUS target model generated successfully. Coverage ranking options: ============================================================================ Rank elements: Model Weight: Block = 1, Expression = 1, Toggle = 1, Fsm = 1, Assertion = 1, Covergroup = 1 Ranking Cost: N.A Target Cumulative Grade(%): 100 Number of Best Runs: 4 Ranking of coverage runs:
Optimized Runs
Run Cum. Self Cum. Wt Self Wt Contrib. Run Id Grade(%) Grade(%) Grade(%) Grade(%) # items Cost Name ============================================================================ Optimized Runs: Redundant Runs 1 58.18 58.18 58.18 58.18 6634 0 cov_work/scope/test2 2 73.29 64.32 73.29 64.32 500 0 cov_work/scope/test1 Redundant Runs: 3 0.00 58.18 0.00 58.18 0 0 cov_work/scope/test3 4 0.00 58.18 0.00 58.18 0 0 cov_work/scope/test4
88
© 2013 Cadence Design Systems, Inc. All rights reserved.
HTML reporting
Analyzing Coverage Data Generating Text Reports in IMC % imc -batch
Load run “all” containing merged data from all tests imc> load -run all
Optional: Load refinement file (if any) imc> load –refinement my_allexclude.vRefine
Generate Summary Report for all instances imc> report -summary –inst [-html] –out myreportsum.html *…
Generate Detailed Report for all instances imc> report -detailed –inst [–html] -out myreport.det *…
90
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC: HTML reports
Example of a report – Top Level Summary Key data
Concise useful HELP is one click away
High-level summary Total (average)
Navigate hierarchy
Coverage Numbers Link to flat CoverGroup coverage – all CoverGroups in one table
91
© 2013 Cadence Design Systems, Inc. All rights reserved.
IMC: HTML reports Example of a report - Summary
Click to open detailed report for instance
Click column header to sort table
Click link to see details 92
© 2013 Cadence Design Systems, Inc. All rights reserved.
Click instance to descend
IMC: HTML reports Example of Global CoverGroup Summary Report
Actual coverage
Covergroup options
93
© 2013 Cadence Design Systems, Inc. All rights reserved.
Bin details
Covergroup instance name
IMC: HTML reports Example of Per Covergroup Coverage
Click on item to view details
94
© 2013 Cadence Design Systems, Inc. All rights reserved.
Lab 3 – Coverage Analysis
95
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Workshop Agenda • Introduction to Metric-Driven Verification • Coverage Metrics • Code coverage – Generating Code Coverage – Analyzing Code Coverage Using IMC – Lab 1: generating & analyzing code coverage
• Functional coverage – Building Functional Coverage Model – Analyzing Functional Coverage Using IMC – Lab 2: functional coverage collection & analysis
• IMC Merge/Rank/Refine/Report – Lab 3: merge tests & generate report
• Summary 96
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Guidelines Summary – Page 1 • Start with verification planning • Functional coverage design – For each design feature: – Determine minimum number of variables required – Select sampling event carefully – Check the “Covergroup type options” (e.g., weight, goal) to ensure correct grading – Name the covergroup using set_inst_name()
• Running tests and collect functional coverage • Once functional coverage reaches target & design is stable, start code coverage
97
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Guidelines Summary – Page 2 • Code coverage – Start with Block coverage + FSM coverage – Add branch coverage – Expression coverage for critical blocks; work with designers – Is SOP good enough? – If not, work with designers to determine if Control or Vector is required – Toggle coverage for interfaces between sub-blocks
• What about SOC level? – Functional coverage determined by the SOC verification plan – What about code coverage? – Since the concerns here are: interconnect between blocks and “glue” logic – Toggle coverage for interconnect between blocks – Block/branch/expression? coverage for “glue” logic
98
© 2013 Cadence Design Systems, Inc. All rights reserved.
Coverage Guidelines Summary – Page 3
• Be selective about what you measure & collect – Capture only what you really need otherwise…. – …you might have to spend lot of time analyzing results and writing exclusions!
99
© 2013 Cadence Design Systems, Inc. All rights reserved.
Additional Documentation / Tutorial IMC Guide:
© 2013 Cadence Design Systems, Inc. All rights reserved.
http://.cadence.com http://www.cadence.com
Related Documents 3h463d
Coverage Ws Overview 383bx
November 2022 0
Statement Coverage, Branch Coverage, Condition Coverage Tutorials For Software Testers 1f6q3f
October 2019 40
Code Coverage 461x3
July 2021 0
Statement Coverage, Branch Coverage, Condition Coverage Tutorials For Software Testers 1f6q3f
November 2019 35
Upcat Coverage 3z72c
January 2022 0
Functional Coverage 53g5
March 2021 0More Documents from "Hien Ha" 2x4845
Coverage Ws Overview 383bx
November 2022 0
Buildroot.pdf 1v528
March 2023 0
Christ Is Enough Chords By Hillsong Livetabs @ Ultimate Guitar Archive 665q4
November 2022 0
Compliancehandbook.pdf 421h2m
January 2022 0
The-incredible-life-of-a-himalayan-yogi-the-times-teachings-and-life-of-living-shiva-baba-lokenath-brahmachari-by-shuddhaanandaa-brahm.pdf 2h2925
December 2021 0