This post details the TFS ALM code quality strategic capabilities: static code analysis, code metrics, profiling, supported test types, unit test development, ordered tests, data driven unit tests, code coverage, test impact analysis, diagnostic data adapters, database unit tests, intelli-trace dumps for bug work items and coded UI tests.
rule groups, how to execute & enable on compile and delves into the introspection API.
· Includes DB analysis rules
· Rule groups: design , globalization, interop, maintainability, mobility, naming, perf, portability, reliability, security, usage
· To execute
o VSTS: Build > Run Code Analysis
o Results are displayed in Errors Window (can create WI from a warning or error)
o Suppress an error – adds SuppressMessageAttribute and an entry to StaticCodeAnalysis.SuppressMessages.xml file
o By default ignores code flagged with GeneratedCodeAttribute
o Results are stored at \bin\debug\{project}.CodeAnalysisLog.xml – open with Internet Explorer
o CLU - \Team Tools\Static Analysis Tools\FxCop\FxCopCmd.exe - Can be part of checkin policy, run as part of build. Use /project option to leverage settings in FxCopProject XML file
· To Enable on compile:
o VSTS: {Project} > properties > Code Analysis
o Enable code analysis on build checkbox – defines CODE_ANALYSIS variable
o Select Rule set
§ All
§ Basic correctness – logical errors & common mistakes
§ Basic design guideline – best practices
§ Extended correctness
§ Extended design guideline
§ Globalization
§ Minimum recommended - default
§ Security
o Enable / disable subsets in the tree
o Select action: warning (default), error, inherit, none – applies to active build config, but can specify which config (e.g. error in release, warning in debug)
· Introspection API – supports custom rules
o Advantage over reflection: faster, supports multithreaded ops, doesn’t lock analyzed file
o A rule has 2 components: 1) a code file to analyze target assembly 2) a rules.xml file containing descriptions and resolutions
o Learn by reflecting over existing rule assemblies
o Create a C# class library project and add references to \Team Tools\Static Analysis Tools\FxCop\FxCopSdk.dll and Microsoft.Cci.dll (common compiler infra goes beyond CodeDOM)
o Create a class that inherits from BaseIntrospectionRule and overloads Check method to specify target – when a rule is violated, retrieve resolution from embedded XML , create a new Problem instance and return the Problems collection
public class MyRule : BaseIntrospectionRule
{
Public MyRule (string name)
: base (name, "MyRules", typeof(MyRule).Assembly) {}
Public override ProblemCollection Check (Member member)
{
if (condition)
{
Problems.Add(new Problem(GetResolution(member.Name.Name)));
}
return Problems;
}
}
o Rules.Xml – add to Resources folder. in properties: Embedded Resource
<Rules …>
<Rule … >
<Name …>
<Description …>
<Resolution …>
<MessageLevel …>
<FixCategories …>
<url …>
<Email …>
<Owner …>
o To deploy - copy assembly to \Team Tools\Static Analysis Tools\FxCop\Rules subdirectory – the rule is added to All Rules by default.
o To debug – add XCopy command to Project > Properties > Build > PostBuild Requires VSTS restart
o To create a custom RuleSet - VSTS: File > New File > General > Rule Set File
how to evaluate code quality by examining code metrics.
· To run - Solution > Calculate Code metrics
· Cyclometric Complexity - # code paths
· Depth of inheritance
· Class coupling - # dependencies that a class has on other classes
· Lines of code – IL
· Maintainability index – combination value between 1 (bad) and 100 (good) – keep above 20!
Visual Studio profiling capabilities: profiler steps, how to create a performance analysis session, how to configure a session using Performance Explorer, session configuration properties, how to launch a session, viewing reports, and command line profiling.
· Overview
o Capabilities: Sampling (CPU / memory), instrumentation, concurrency
o Can integrate into build
o Sampling – periodic snapshots of call-stack. Relatively light. Use to determine which functions might be bottlenecks or critical paths.
o Instrumentation – inserts tracing probes into IL at start & end of each function
o Support for profiling on VMs, JavaScript
· Profiler steps:
o Create a perf session – select profiling method & targets
o Configure session - Use Perf Explorer
o Launch session – executing the target app with attached profiler
o Review results presented in perf reports
· Create a perf session
o 3 ways:
§ VSTS: Analyze > launch Perf Wizard
§ VSTS: Analyze > Profiler > New Performance Session à opens session in Perf Explorer
§ VSTS: Test > Windows > Test Results: {test} > Create Perf Session – when the perf session is run, the unit test is executed and the profiler will be activated to collect metrics
o Specify profiling method
§ CPU Sampling – recommended – measures CPU bound apps and has low overhead
§ Instrumentation – measure function call counts & timing
§ NET memory allocation sampling – track memory management, instantiated types & GC. Interrupts CPU for every managed object allocation
§ Concurrency – detect waiting threads, optionally collect resource contention and thread execution data
o Select target app
§ Project(s)
§ .exe
§ ASP.NET URL
o Launch on completion
· Configure Session – Performance Explorer
o Supports multiple sessions
o Toolbar button for switching between sampling & instrumentation
o 2 root folders in session tree: Targets and Reports
o { Session } Context menu:
§ Set as current
§ Remove
§ Launch with profiling
§ Launch with profiling paused
§ Attach / detach
§ Properties
o Default session name is "PerformanceX"
o Targets Context menu:
§ Add Target Binary
§ Add Target Project
§ Add Existing Web Site (ASP.NET URI)
o { Target } Context menu:
§ Remove
§ Collect Samples / Instrumentation
§ Set as launch – ifyou have multiple target assemblies, set the .exe to launch
§ Properties
o File > Open
§ open a .psess session file
§ open a .vsp report file
· Configure Session - Properties
o General
§ Profiling collection: sampling / instrumentation / concurrency
§ Collect resource contention data , thread execution data
§ Memory Profiling - Collect NET object allocation info – track # instances and amount used by them à Report > Allocation View
§ Memory Profiling - Collect NET object lifetime info – overhead impacts GC à Report > Object Lifetime View
§ Launch data collection control à Marks dialog for annotating live session and seeing marks later in report
§ Report – name, location
§ Automatically add new reports to session
§ Report number – auto-increment, use timestamp
o Launch – set target order – move up / down, Override project settings:
§ Executable to launch
§ Arguments
§ Working directory
o Sampling
§ Sample event: clock cycles / page faults (use if app is memory intensive) / system calls / perf counter (e.g. cache misses)
§ Sampling interval – reduce to minimize effect on production environment
o Binaries (Instrumentation only)
§ Relocate instrumented binaries – VS will copy original binaries, instrument them and place them in working folder
o Tier interactions
o Instrumentation
§ Exclude small functions
§ Profile JavaScript
§ Pre-instrument events: command line, exclude
§ Post-instrument events: command line, exclude – important for digitally signed assemblies: call sn.exe and exclude from instrumentation
o CPU Counters – select counters to track
o Windows Events – collect trace info: IO, NET CLR, IIS, ASP.NET
o Windows Counters – perf counters
o Advanced (Instrumentation only)
§ command line options for VSInstr.exe
§ CLR target version
· Launch a session
o For accurate results, make sure to set build config to Release not Debug!!!
o Perf Explorer: { Session } > Launch with profiling
o When app completes, the profiler will shut down and generate a report
o When profiling a web app, a browser instance is launched – not required to use – can leverage Web perf & load tests instead
· Review Perf Reports
o Performance Explorer: Reports
o Delta analysis - First profile a baseline session then compare profile sessions to it: {Report} > Compare Report > Comparison File
o {Report} > Export Report Data – choose views to export, set report prefix, export file format, location
o Choose columns – note elapsed time includes interrupt waits, application time excludes interrupt waits
o Notifications?
o Buttons: Filter, zoom , just my code
o Report buttons
§ Show trimmed call tree
§ Compare reports
§ Export
§ Save
§ Filter
§ Toggle full screen
o Summary view – sample / instrumentation
§ CPU usage chart
§ Hot path tree
§ Call tree
§ Most utilized functions tree
o Summary view - memory
§ CPU usage chart
§ Most memory-allocated functions tree
§ Most memory-allocated types tree
§ Most instantiated types tree
§ Object lifetimes tree (optional) – groups by generations: G2 contains oldest and should be large, G0 contains new instances and should be small. Optionally Filter by Instances Alive at End
o Functions support caller / callee view drilldown
o Allocations show totals and percentage
· Command line profiling
o Trigger from nightly build
o Flow:
§ Configure target & environment
§ Start data logging engine
§ Run target app
§ Stop data logging engine
§ Generate session report
o Tools – under install dir \Team Tools\Performance Tools\
§ Vsinstr.exe – instrument a binary
§ Vsperfcmd.exe – launch a profiling session
§ Vsperfmon.exe – monitor a profiling session
§ Vsperfreport.exe – generate a report from a profiling session
§ vsperfCLRenv.exe – set environment variables for session
· Common issues
o Debugging symbols
§ If the app cannot find symbols for some code block, you will see unhelpful entries such as [ntdll.dll]
§ To config a symbol server for .pdb program database files: VSTS: tools > options > debugging > symbols
§ Note: 1st time may take time.
o Code coverage conflicts with instrumentation
§ VSTS: Test > Edit Test Run Configs > Code Coverage – uncheck relevant binaries
supported test types and the capabilities of different versions of visual studio.
· Test Types
o Unit tests – coded MSTest
o Manual Tests
o Coded UI Tests – automated UI manipulation & validation
o Database Unit Tests
o Web Perf Tests
o Load Tests
o Ordered tests – (regression testing).
· VS 2010 premium / ultimate edition extra test features
o code coverage
o code impact analysis
o coded UI tests
o convert manual tests to automated tests
o web performance tests
o load tests
MSTest unit testing features and guidelines - the MSTest API, and how to use and configure MSTest unit tests, and diagnostic data adapters.
· Features
o Generate from code
o Test grouping à Regression tests
o Code coverage
o Test impact analysis - Identify unit tests affected by changesets
o Data driven tests – each test is executed once per record
· Guidelines
o Place in separate assembly
o Do not alter code to suit tests
o Each test should verify an atomic function
o Each test should be autonomous and isolated – avoid inter-test dependencies
o Test both expected behavior and error conditions
o Pattern: arrange, act assert
o Code coverage: removal of any line code should cause tests to fail
· MSTest API
o Assembly reference: Microsoft.VisualStudio.QualityTools.UnitTestFramework.dll
o Attributes:
§ TestClass, TestMethod
§ ExpectedException
§ TestProperty – decorative – for view in Test List Editor
§ TestCategory – decorative – for view in Test List Editor. E.g. "BVT"
§ DeploymentItem – specify source file for data driven tests
§ TestInitialize, TestCleanup
§ ClassInitialize, ClassCleanup – e.g. open / close a log file
§ AssemblyInitialize, AssemblyCleanup
o Methods with these attributes must have the signature: public, non-static, parameter-less, void
o TestContext class members:
§ Note: this is not static à need to instantiate as a field of the test class
§ WriteLine – add to output file
§ BeginTimer / EndTimer
§ AddResultFile – additional file for IO tests
§ DataRow – dictionary for data driven tests
o Assert members:
§ AreEqual / AreNotEqual – value equivalence
§ AreSame / AreNotSame – reference equivalence (i.e. same object)
§ IsTrue / IsFalse
§ IsNull / IsNotNull
§ IsInstanceOfType / IsNotInstanceOfType
§ Fail – immediately fail the test à AssertFailException
§ Inconclusive - default
o CollectionAssert members:
§ AllItemsAreInstancesOfType , AllItemsAreNotNull, AllItemsAreUnique
§ AreEqual / AreNotEqual – collection member reference equivalence
§ AreEquivalent / AreNotEquivalent – collection member value equivalence
§ Contains / DoesNotContain
§ IsSubsetOf / IsNotSubsetOf
o StringAssert members:
§ Contains
§ StartsWith / EndsWith
§ Matches / DoesNotMatch – regex
o PrivateObject / PrivateType – use for accessing non-public instance / static members – a wrapper around reflection code – e.g.:
[TestClass]
public class ExampleTest
{
const string Password = "xxx";
PrivateObject _objToTest;
[TestInitialize]
public void TestInitialize()
{
objToTest = new PrivateObject(typeof(Example));
}
[TestMethod]
public void TestPassword()
{
Assert.IsTrue((bool) objToTest.Invoke("xxx", Password) );
}
}
· Using MSTest
o To add test projects - Project > Add new Project > Test > Test project
o Generating Tests from code - From source (method or class level): Context > Create unit Tests
o main views: Test > Windows
§ Test View – compact view. Select, Group, Run, Debug. Advisable to add Test Categories column.
§ Test List Editor – additional options: Filter, Create New Test List
§ Test Results – Play / Pause / Stop. Publish. context menu: View Test Result Details to see item details, > Open Test to navigate to test code
o Running individual tests – from unit test method code > context menu, Ctrl+R, Ctrl+T
o Test results: stored as XML in \TestResults\*.trx – has viewer associated. Can contain error message & stack trace.
o Publishing results – publish .trx files to TFS DW for time based reporting: Quality Indicators, Code Coverage Details, Test Run Details. Test > Windows > Test Results: Publish à Publish Test Results Dialog: specify test runs, associated builds and whether to include code coverage data.
· Test run config:
o Test > Edit Test Settings à designer for solution level XML file: local.testsettings
§ General
§ Roles
§ Data & Diagnostics – data adapters
§ Deployment
§ Setup & Cleanup Scripts
§ Hosts
§ Test Timeouts
§ Unit Test
§ Web Test
o Test > Selective Active Test Settings – can have multiple configs – 2 out of box defaults:
§ 1) Local – smaller .trx files and no diagnostic adapters;
§ 2) Trace & Test Impact – enables system info, Trace and IntelliTrace adapters – use to file a bug from test results
o To create custom settings: Solution > Add > New item > Test Settings
· Diagnostic Data Adapters
o Adapter types:
§ Action log + action report – for manual tests, record steps.
§ Code coverage
§ IntelliTrace
§ Test impact analysis
§ Event Log
§ System Info
§ Video recorder
§ Network emulator – simulate bandwidth throttling
§ Custom
o
ordered tests and data driven unit tests.
· Ordered Tests
o Grouped regression tests.
o Stored in solution level project specific XML file: *.vsmdi.
o Can be leveraged by check in policy.
o To create: Test Project > Add > Ordered Test à ordered test editor dialog: select tests & move up / down
o Important Properties: Description, Test Enabled, Timeout, Continue After Failure
· Data Driven Unit Tests
o Automatically bind data input to unit tests. The unit test will be called once per row in the data source
o In Test List Editor – set a unit test's properties: set Data Connection String: ODBC (SQL, Excel), CSV, XML – added as a deployment item
o E.g. .csv file – 1st row converted to headers:
Username, Password
John, 123456
Bob, abcdefg
o XML elements: DataContextData, DataContextRow
o In test, this injects: DeploymentItemAttribute , TestContext.DataRow[""]
o Test result details show pass/fail for each row
o Use to test valid, invalid data
how to create them, how to use the Coded UI Test Builder tool, and the generated code.
· Functional UI testing – automatically execute & validate a test case.
· Generates code inside a Test Project to automate the UI & query the UI state
· Automated UI manipulation and validation for WPF, WinForms, ASP.NET, Silverlight (with 2010 sp1 beta).
· To create: Test Project > Add > Coded UI Test à 2 options
o Record new test
o Modify an existing test
· Coded UI Test Builder
o Appears at bottom of screen. Minimize all windows, click Record, then click shortcut to exe to start. Close the test builder to finish.
o Record / Pause
o Show Recorded Steps
o Add and generate – converts steps to code. Prerequisite to adding assertions. Also do after each assertion
o Add assertion – drag crosshairs onto UI. Select control, Assert method and comparison value
· Generated code example:
[TestMethod]
public void CodedUITestMethod1()
{
this.UIMap.EnterDataAndClickAdd();
this.UIMap.AssertAdd();
}
· Underlying API code constructs:
o WpfEdit
o WpfButton
o ApplicationUnderTest.Launch – specify file path
o Mouse.Click – static, takes absolute mouse position
· Unfortunately, the XAML Automation library is not leveraged!!!!
· Functions like any other unit test – run through test list editor, hook up data driven tests etc.
· Manual test work item steps recordedcan be converted into Coded UI Tests
how to unit test a database project.
· Prerequisite: Database Project, optional: data generation plan
· In Schema View: Schemas > dbo > Programmability > Stored Procedures. Select SPs and Context > Create Unit Tests à generates test project
· DB test project config:
o Data connection
o Deployment checkbox – check this à auto deploy DB before tests are run
o Deployment config – select Debug
o Generate test data before unit tests are run checkbox – check this & select plan file
o Clear DB before test data is generated checkbox – check this
· Test Initialize / Cleanup - specify pre / post test scripts (select from Common Scripts) to be executed before / after any given unit test to prime a DB with test data & cleanup after
· Test Conditions– validate execution e.g. row count, execution time, scalar value match etc.
· Custom test conditions– derive from Microsoft.Data.Schema.UnitTesting.Conditions
· Project files
o All test TSQL scripts are stored in a .resx file
o Connection strings, data generation plan are storedin app.config
o Deployment and execution of tests – instructions in DatabaseSetup.cs
· Recommendation: wrap each test in a transaction and rollback
common testing issues to watch out for.
|
Name
|
Description
|
Causes
|
|
Assertion Roulette
|
It is hard to tell which of several assertions within the same test method caused a test failure
|
Eager Test, Missing Assertion Message
|
|
Conditional Test Logic
|
A test contains code that may or may not be executed
|
Complex Teardown, Conditional Verification Logic, Flexible Test, Indented Test Code, Multiple Test Conditions, Production Logic in Test
|
|
Developers Not Writing Tests
|
Developers aren't writing automated tests
|
Not Enough Time, Wrong Test Automation Strategy
|
|
Erratic Test
|
One or more tests are behaving erratically; sometimes they pass and sometimes they fail
|
Interacting Test Suites, Interacting Tests, Lonely Test, Nondeterministic Test, Resource Leakage, Resource Optimism, Test Run War, Unrepeatable Test
|
|
Fragile Test
|
A test fails to compile or run when the SUT is changed in ways that do not affect the part the test is exercising
|
Behavior Sensitivity, Context Sensitivity, Data Sensitivity, Fragile Fixture, Interface Sensitivity, Over-coupled Test, Over-specified Software, Sensitive Equality
|
|
Frequent Debugging
|
Manual debugging is required to determine the cause of most test failures
|
Manual Debugging
|
|
Hard-to-Test Code
|
Code is difficult to test
|
Asynchronous Code, Hard-Coded Dependency, Highly Coupled Code, Untestable Test Code
|
|
High Test Maintenance Cost
|
Too much effort is spent maintaining existing tests
|
|
|
Manual Intervention
|
A test requires a person to perform some manual action each time it is run
|
Manual Event Injection, Manual Fixture Setup, Manual Result Verification
|
|
Obscure Test
|
It is difficult to understand the test at a glance
|
Complex Test, General Fixture, Hard-Coded Test Data, Indirect Testing, Irrelevant Information, Long Test, Mystery Guest, Verbose Test
|
|
Production Bugs
|
We find too many bugs during formal test or in production
|
Infrequently Run Tests, Lost Test, Missing Unit Test, Neverfail Test, Untested Code, Untested Requirement
|
|
Slow Tests
|
The tests take too long to run
|
Asynchronous Test, Slow Component Usage, Too Many Tests
|
|
Test Code Duplication
|
The same test code is repeated many times
|
Cut-and-Paste Code Reuse, Reinventing the Wheel
|
|
Test Logic in Production
|
The code that is put into production contains logic that should be exercised only during tests
|
Equality Pollution, For Tests Only, Test Dependency in Production
|
Code Coverage and Test Impact Analysis
how to assure code quality by leveraging code coverage and test impact analysis.
· Code Coverage
o Instruments code to monitor which lines are executed during test
o Test > Edit Test Settings > Configuration > Data & Diagnostics: check Code Coverage checkbox
o Click Configure button and select the row à Code Coverage Detail dialog. Add Assembly button – select assemblies to instrument
o Note: if working with signed assemblies, instrumentation invalidates. Specify resigning key file path.
o Code Coverage Results window – run tests to see. Blue / red highlight indicates code covered / not covered. Code coverage percentage will be indicated.
o Config build: Team Explorer > builds > {build} > Edit Build Definition > Process > Basic > Automated Tests > Test Assembly > Test Settings File – point to .testsettings file
· Test Impact Analysis
o Unproductive to run all tests. Determine what tests are affected by code change. Can then run just appropriate tests to ensure no new errors.
o To enable for a solution: Test > Edit Test Settings > Data & Diagnostics: check Test Impact checkbox
o To enable as part of TFS build: in Team Explorer, build definition: Process > Testing Category: set Analyze Test Impacts to true and from Test Container Test Settings File, browse to the .testsettings file
o Test > Windows > Test Impact View
o To view impacted tests: Show Impacted Tests
o To view tests for a particular method: Show Calling Tests. Select the method in the code editor
o Select tests and run selection
o To reset baseline: Run all impacted tests
how to configure and use intellitrace.
· Load and debug trace log files. Attach to Bug Work Items à No more "no repro"
· For 64bit support, install SP1 beta
· Configure
o Tools > Options > IntelliTrace:
o General
§ Check Enable IntelliTrace
§ 2 options:
§ 1) Events only
§ 2) Events and calls – captures call stacks at each event - can degrade perf, Edit & Continue is disabled
o Advanced
§ Recording location – default: c:\ProgramData\Microsoft VSTS\10.0\TraceDebugging\
§ Max space per recording
§ Display nav gutter in debug mode
o Events – select events
o Modules – select assemblies
· Use
o Recording Events - To load the window: Debug > Windows > IntelliTrace Events
o Also shows recorded exceptions
o When you select an event in the list, the appropriate source is shown in the editor and highlighted
o Can playback a recorded debug session: break and step back / forward
o Debug > Break All to pause the debugger, then navigate from Debug > Windows > IntelliTrace Events window
o When a trace log file is opened in VSTS, it shows the summary view, threads list and exceptions list. Click on an exception to start debugging from the point of throw.