Sas_di_developer_-_fast_track.pdf 53181f

[email protected]

. b. Check in all tables. Hint: The ODBC library and ODBC server are defined in the previous demonstration. If necessary, consider the following steps to replicate: ¾ ODBC Server: pg 3-42, steps 1-13 ¾ ODBC Library: pg 3-51, steps 9-19

3.2 ing Source Data Metadata

3-83

7. Defining Metadata for an External File A file, profit.txt (located in the directory S:\Workshop\dift\data), contains data that is stored in fixed column widths. Use the New Fixed Width External File wizard to create metadata. Use the following record layout and information for the necessary column metadata and use the default settings. Column Name

Length

Type

Informat

Format

Begin Position

End Position

1

22

Company

30

Char

YYMM

8

Num

YYMM5.

24

28

Sales

8

Num

DOLLAR13.

30

43

Cost

8

Num

DOLLAR13.

45

58

Salaries

8

Num

DOLLAR13.

61

84

Profit

8

Num

DOLLAR13.

87

100

$22.

Each column must be created using

.

• Name the metadata object representing the external file DIFT Profit Information and place the metadata object in the Data Mart Development Ö Orion Source Data folder. • After the external file metadata is defined in the project repository, be sure to check it in. 8. Defining Metadata for an ODBC Data Source Orion Star is considering purchasing a small store that sells various products over the internet and through catalog orders. Some data from the store was made available for analysis in the form of a Microsoft Access database DIFTExerciseData.mdb located in the directory S:\Workshop\dift\data. all three tables in metadata within this database. • The ODBC system data source needs to be created. Name the data source DIFT Workshop Data. • A library metadata object and a server metadata object need to be created. Name these as follows: Library name:

DIFT Workshop Microsoft Access Database

Server name:

DIFT Workshop Microsoft Access Database Server

• Place the metadata table objects in the Data Mart Development Ö Orion Source Data folder. • Change the default metadata names to DIFT . • Check in all objects.

3-84

Chapter 3 Creating Metadata for Source Data

3.3 Solutions to Exercises 1. Specifying Folder Structure If you did not follow along with the steps of the demonstration, complete steps 1-13 starting on page 3-5. 2. Specifying Orion Source Tables Library If you did not follow-along with the steps of the demonstration, complete steps 1-17 starting on page 3-10. 3. Specifying Library for Additional SAS Tables a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Source Data. that the Orion Source Data folder is selected. c. Select File Ö New Ö Library…. The New Library Wizard opens. d. Select SAS BASE Library as the type of library to create. e. Click

.

f. Type DIFT SAS Library as the value for the Name field. g. that the location is set to \Data Mart Development\Orion Source Data. h. Click

.

i. Select SASApp in the Available servers pane. j. Click k. Click

in the SASApp to selected servers pane. .

l. Type diftsas as the value for the Libref field. m. The desired path does not exist in the Available items pane. Click n. In the New Path Specification window, click

.

next to Paths.

o. In the Browse window, navigate to the desired path of S:\Workshop\dift\data. p. Click

to close the Browse window.

q. Click

to close the New Path Specification window.

3.3 Solutions to Exercises

r. the new path appears is the Selected items list.

s. Click

.

t. that the information is correct in the review window and then click The new library metadata object is found in the Checkouts tree.

.

3-85

3-86


4. Checking in New Library Objects a. Select Check Outs Ö Check In All. b. Type Adding two library objects as the value for the Title field. c. Type Checking in new library objects of DIFT Orion Source Tables Library and DIFT SAS Library as the value for the Description field.

d. Click

.


e. that both library objects are selected.

f. Click

.

3-87

3-88


g. Review the information in the Summary window.

h. Click

.

The library objects should no longer be in the Checkouts tree.


3-89

i. Click the Folders tab. j. Expand Data Mart Development Ö Orion Source Data. k. that both library objects are now checked in. 5. Defining Metadata for Additional SAS Source Tables a. Place the metadata table objects in the Data Mart Development Ö Orion Source Data folder. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Source Data. 3) that the Orion Source Data folder is selected. b. the three additional tables found in DIFT Orion Source Tables Library. 1) Select File Ö Tables…. The Tables wizard opens. 2) Select SAS as the type of table to import information. 3) Click 4) Click 5) Click

. The Select a SAS Library window opens. next to SAS Library field and then select DIFT Orion Source Tables Library. . The Define Tables and Select Folder Location window opens.

6) Select STAFF table, hold down the CTRL key and select ORDER_ITEM and ORDERS. 7) that /Data Mart Development/Orion Source Data is the folder listed for the Location field. 8) Click

. The review window opens.

9) that the information is correct and click

.

10) Right-click the STAFF metadata table object and select Properties. 11) Type DIFT at the beginning of the default name. 12) Click

to close the Properties window.

13) Right-click the ORDER_ITEM metadata table object and select Properties. 14) Type DIFT at the beginning of the default name. 15) Remove the default description. 16) Click


17) Right-click the ORDERS metadata table object and select Properties. 18) Type DIFT at the beginning of the default name.

3-90


19) Click


c. the five tables (SAS tables) found in the DIFT SAS Library. Change the default metadata names to DIFT <SAS-table-name>. 1) Select File Ö Tables…. The Tables wizard opens. 2) Select SAS as the type of table to import information. 3) Click 4) Click 5) Click 6) Click

. The Select a SAS Library window opens. next to SAS Library field and then select DIFT SAS Library. . The Define Tables and Select Folder Location window opens. .

7) that /Data Mart Development/Orion Source Data is the folder listed for the Location field. 8) Click

. The review window opens.

9) that the information is correct and click

.

The metadata objects for the table are found in the Checkouts tree.

10) Right-click the CUSTOMER_TRANS metadata table object and select Properties. 11) Type DIFT at the beginning of the default name.

12) Click


13) Right-click the CUSTOMER_TRANS_OCT metadata table object and select Properties.


14) Type DIFT at the beginning of the default name.

15) Click


16) Right-click the NEWORDERTRANS metadata table object and select Properties. 17) Type DIFT at the beginning of the default name.

18) Remove the default description. 19) Click


20) Right-click the STAFF_PARTIAL metadata table object and select Properties. 21) Type DIFT at the beginning of the default name.

22) Click


23) Right-click the VALIDPRODUSAOUTDOOR metadata table object and select Properties. 24) Type DIFT at the beginning of the default name.

25) Remove the default description. 26) Click


The metadata objects for the table are found in the Checkouts tree.

3-91

3-92


d. Check in all SAS tables. 1) Select Check Outs Ö Check In All. 2) Type Adding metadata for SAS source tables as the value for the Title field. 3) Type Added metadata for CUSTOMER_TRANS, CUSTOMER_TRANS_OCT, NEWORDERTRANS, ORDER_ITEM, ORDERS, STAFF, STAFF_PARTIAL, and VALIDPRODUSAOUTDOOR. as the value for the Description field. 4) Click

. that all table objects are selected.

5) Click

. Review the information in the Summary window.

6) Click

. The table objects should no longer be in the Checkouts tree.


3-93

6. Defining Metadata for an Existing ODBC Data Source a. Place the metadata table objects in the Data Mart Development Ö Orion Source Data folder. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Source Data. 3) that the Orion Source Data folder is selected. 4) Select File Ö Tables…. The Tables wizard opens. 5) Expand the ODBC Sources folder. 6) Select ODBC – Microsoft Access as the type of table to import information about. 7) Click . The ODBC window opens and displays the one ODBC library definition in the SAS Library field. 8) Click

.

9) Select s, hold down the CTRL key and select NewProducts. 10) Click

.

11) Click

.

The metadata objects are found in the Checkouts tree.

12) Right-click the s metadata table object and select Properties. 13) Type DIFT at the beginning of the default name.

14) Click


15) Right-click the NewProducts metadata table object and select Properties.

3-94


16) Type DIFT at the beginning of the default name.

17) Click


b. Check in all SAS tables. 1) Select Check Outs Ö Check In All. 2) Type Adding metadata for s and NewProducts tables as the value for the Title field. 3) Click


4) Click


5) Click

. The table objects should no longer be in the Checkouts tree.

7. Defining Metadata for an External File a. Name the metadata object representing the external file DIFT Profit Information and place the metadata object in the Data Mart Development Ö Orion Source Data folder. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Source Data. 3) that the Orion Source Data folder is selected. 4) Select File Ö New Ö External File Ö Fixed Width…. 5) Type DIFT Profit Information as the value for the Name field. 6) that the location is set to /Data Mart Development/Orion Source Data. 7) Click

. The External File Location window opens.

8) Click

to open the Select a file window.

9) Navigate to S:\Workshop\dift\data. 10) Select profit.txt. 11) Click

.


12) To view the contents of the external file, click

13) Click

.

to close the Preview File window.

14) Click

. The Parameters window opens.

15) Accept the default settings for Parameters and click

.

16) Define the column information. to add a new column specification. Enter the following information:

17) Click Column Name Company

Length 22

18) Click

Length 8

19) Click

$22.

1

End Position 22

Type Num

Begin Position

Format YYMM5.

24

End Position 28

to add a new column specification. Enter the following information:

Column Name Sales

Char

Begin Position

Informat


Column Name YYMM

Type

Length 8

Type Num

Begin Position

Format DOLLAR13.

30

End Position 43

3-95

3-96


20) Click


Column Name Cost

Length 8

21) Click

Length 8

22) Click

DOLLAR13.

Type

25) Click

58

Num

Begin Position

Format DOLLAR13.

Length 8

Type

61

End Position 84

Num

Begin Position

Format DOLLAR13.

23) Click the Data tab and then click 24) Click

45

End Position


Column Name Profit

Num

Begin Position

Format


Column Name Salaries

Type

87

End Position 100

. that the values are read in correctly.

. A Warning window opens.

. The review window displays general information for the external file.


26) Click

3-97

. The metadata object for the external file is found in the Checkouts tree.

b. After the external file metadata is defined in the project repository, be sure to check it in. 1) Select Check Outs Ö Check In All. 2) Type Adding metadata for profit information external file as the value for the Title field. 3) Click

. that the external file object is selected.

4) Click


5) Click

. The external file object should no longer be in the Checkouts tree.

8. Defining Metadata for an ODBC Data Source a. The ODBC system data source needs to be created. Name the data source DIFT Workshop Data. 1) Select Start Ö Control . 2) Open the istrative Tools by double-clicking istrative Tools. 3) In the istrative Tools window, double-click Data Sources (ODBC) to open the ODBC Data Source window. 4) In the ODBC Data Source window, click the System DSN tab. 5) Click

.

6) Select Microsoft Access Driver (*.mdb) as the driver type. 7) Click

. The ODBC Microsoft Access Setup window is opened.

8) Type DIFT Workshop Data as the value for the Data Source Name field. 9) Click

in the Database area.

10) Select s:Extension in the Drives area.

3-98


11) Navigate to s:\workshop\dift\data in the Directories area. 12) Select DIFTExerciseData.mdb in the Database Name area. 13) Click

to close the Select Database window.

14) Click

to close the ODBC Microsoft Access Setup window.

15) Click

to close the ODBC Data Source window.

16) Select File Ö Close to close the istrative Tools window. b. Access SAS Management Console using Ahmed’s credentials. 1) Select Start Ö All Programs Ö SAS Ö SAS Management Console 4.2. 2) Select My Server as the connection profile. 3) Click

to close the Connection Profile window and access the Log On window.

4) Type Ahmed as the value for the ID field and Student1 as the value for the field. 5) Click

to close the Log On window.

c. Define a new ODBC server. 1) Right-click on the Server Manager plug-in and select New Server….The New Server Wizard opens. 2) Select ODBC Server under Database Servers folder. 3) Click

.

4) Type DIFT Workshop Microsoft Access Database Server as the value for the Name field. 5) Click

.

6) Select ODBC – Microsoft Access as the value for the Data Source Type field. 7) Click

.

8) Select Datasrc. 9) Type ”DIFT Workshop Data” (the quotes are necessary since the ODBC data source name has spaces). 10) Click

.

11) Click

.

d. Select File Ö Exit to close SAS Management Console.


e. If necessary, access SAS Data Integration Studio as Barbara’s credentials. 1) Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. 2) Select Barbara’s Work Repository as the connection profile. 3) Click


4) Type Barbara as the value for the ID field and Student1 as the value for the field. 5) Click


f. the tables found in the Microsoft Access database. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Source Data. 3) that the Orion Source Data folder is selected. 4) Select File Ö Tables…. The Tables wizard opens. 5) Expand the ODBC Sources folder. 6) Select ODBC – Microsoft Access as the type of table to import information about. 7) Click 8) Click

. to define a new library object using the ODBC engine.

a) Type DIFT Workshop Microsoft Access Database as the value for the Name field. b) that the location is set to /Data Mart Development/Orion Source Data. c) Click

.

d) Select SASApp in the Available servers pane. e) Click

in the SASApp to Selected servers pane.

f) Type diftodbw as the value for the Libref field. .

g) Click

h) that DIFT Workshop Microsoft Access Database Server is the value for the Database Server field. i) Click

.

j) Click

. This finishes the metadata definition for the library object.

9) Click

.

3-99

3-100


10) Select Catalog_Orders , hold down the CTRL key, select PRODUCTS and then Web_Orders.

11) Click

.

12) Click .The metadata objects for the three tables as well as the newly defined library object are found in the Checkouts tree. g. Update the metadata for Catalog_Orders. 1) Right-click the Catalog_Orders metadata table object and select Properties. 2) Type DIFT Catalog_Orders as the new value for the Name field. 3) Click


h. Update the metadata for PRODUCTS. 1) Right-click the PRODUCTS metadata table object and select Properties. 2) Type DIFT PRODUCTS as the new value for the Name field. 3) Click



3-101

i. Update the metadata for Web_Orders. 1) Right-click the Web_Orders metadata table object and select Properties. 2) Type DIFT Web_Orders as the new value for the Name field. 3) Click


j. Check in the metadata objects. 1) that no objects are selected in the Checkouts tree. 2) Select Check Outs Ö Check In All. 3) Type Adding table metadata from a Microsoft Access database as the value for the Title field. 4) Click

.

5) that all objects are selected. 6) Click


7) Click

.

3-102


Chapter 4 Creating Metadata for Target Data 4.1

ing Target Data Metadata ................................................................................. 4-3 Demonstration: Defining the Product Dimension Table ......................................................... 4-6 Exercises .............................................................................................................................. 4-22

4.2

Importing Metadata ....................................................................................................... 4-23 Demonstration: Importing Relational Metadata.................................................................... 4-26

4.3

Solutions to Exercises ................................................................................................. 4-40

4-2

Chapter 4 Creating Metadata for Target Data

4.1 ing Target Data Metadata


Objectives „

Review features of the New Tables wizard.

3

New Tables Wizard The New Tables wizard enables metadata to be entered for a new table. In deg the new table, you can do these tasks: „ access metadata about any tables and columns already ed in a metadata repository „ override metadata that was imported (for example, change a column name) „ define new attributes for the table being created (for example, indexes)

4

4-3

4-4


Case Study Scenario The star schema goal that you are working toward has four dimension tables and one fact table. Organization Dimension

Customer Dimension

Order Fact Table

Product Dimension

Time Dimension

5

Case Study Scenario The star schema goal that you are working toward has four dimension tables and one fact table. Case Customer Dimension Study

Case Organization Dimension Study

Order

Exercise Fact Table

Product Demo Dimension

7

Case Time Dimension Study


Product Dimension Table Initial Diagram The ProdDim table process diagram was sketched to be this. The input table metadata objects are created. In the demonstration, you define the metadata for the target table, ProdDim.

8

ProdDim Computed Columns

Product_Category Product_Group Product_Line

Product_ List

Supplier

4-5

4-6


Defining the Product Dimension Table

This demonstration defines a metadata object for a single target file. The target is to be a SAS data set named ProdDim to be stored in DIFT Orion Target Tables Library (the library object needs to be created, as well). 1. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. 2. Log on using Barbara’s credentials to access her project repository. a. Select Barbara’s Work Repository as the connection profile. b. Click

to close the Connection Profile window and open the Log On window.

c. Type Barbara as the value for the ID field and Student1 as the value for the field. d. Click


3. Click the Folders tab. 4. Expand Data Mart Development Ö Orion Target Data. 5. that the Orion Target Data folder is selected. 6. Select File Ö New Ö Table….


The New Table wizard opens.

4-7

4-8


7. Type DIFT Product Dimension as the value for the Name field. 8. that the location is set to /Data Mart Development/Orion Target Data. The final specifications for the name and location window should be as follows:

9. Click

.


10. that the DBMS field is set to SAS. 11. Click

next to the Library field. The target tables library is not yet defined.

a. Type DIFT Orion Target Tables Library as the value for the Name field. b. that the location is set to /Data Mart Development/Orion Target Data. The final specifications for the name and location window are as follows:

c. Click

.

4-9

4-10


d. Click SASApp in the Available servers pane. e. Click

f.

Click

to move SASApp to Selected servers pane.

.

g. Type difttgt as the value for the Libref field.

h. Click

in the Path Specification area.

i.

In the New Path Specification window, click

j.

In the Browse window, navigate to S:\Workshop\dift.

k. Click l.

to add a new folder.

Type datamart and press ENTER.

m. Click the datamart folder. n. Click


next to Paths.


o. Click


p. that the newly specified path is found in the Selected items pane. The final settings for the library options window are shown here:

q. Click

.

4-11

4-12


The review window opens.

r.

the information is correct, and then click

.

The new library metadata object can be found in the Library field.


12. Type ProdDim as the value for the Name field. The final settings for the Table Storage Information window are shown below:

13. Click

.

4-13

4-14


14. Expand the Data Mart Development Ö Orion Source Data folder on the Folders tab. 15. From the Orion Source Data folder, expand DIFT PRODUCT_LIST table object. 16. Select the following columns from DIFT PRODUCT_LIST and click the Selected pane: Product_ID Product_Name Supplier_ID

to move the columns to


4-15

17. Collapse the list of columns for DIFT PRODUCT_LIST. 18. Expand the DIFT Supplier Information external file object. 19. Select the following columns from DIFT Supplier Information and click to the Selected pane: Supplier_Name Country

20. Click

.

to move the columns

4-16


The Change Columns/Indexes window opens.


21. Update column metadata. a. Update the name for the Country column to Supplier_Country. b. that the length of Supplier_ID is 4. 22. Select the last column (Supplier_Country). 23. Add new column metadata. to define an additional column.

a. Click

b. Enter the following information for the new column: Column Name

Description

Product_Category c. Click

Length

Product Category

25

Type Character

to define an additional column.

d. Enter the following information for the new column: Column Name

Description

Product_Group e. Click f.

Product Group

Length 25

Type Character


Enter the following information for the new column: Column Name Product_Line

Description Product Line

Length 25

Type Character

4-17

4-18


The final set of eight variables is shown below:


4-19

24. Define needed indexes. a. Click Product_Group. b. Click

. Define two simple indexes: one for Product_ID and one for

to add the first index.

c. Enter an index name of Product_ID and press ENTER.

Neglecting to press ENTER results in not having the name of the index saved, which produces an error when the table is generated because the name of the index and the column being indexed do not match.

d. Select the Product_ID column and move it to the Indexes by clicking e. Click f.

.

to add the second index.

Enter an index name of Product_Group and press ENTER.

g. Select the Product_Group column and move it to the Indexes by clicking requested indexes are defined in the Define Indexes window.

to close the Define Indexes window and return to the Target Table Designer.

h. Click 25. Click

. The two

.

4-20


26. Review the metadata listed in the finish window.

27. Click

.

The new table object and new library object appear on the Checkouts tab.


4-21

28. Select Check Outs Ö Check In All. a. Type Adding metadata for Product Dimension table as the value for the Title field. b. Type DIFT Orion Target Tables Library object was also created and checked in. as the value for the Description field. c. Click

. that all objects are selected.

d. Click


e. Click

.

The objects should appear in the Data Mart Development Ö Orion Target Data folders.

4-22


Exercises

1. Defining OrderFact Target Table Define metadata for the OrderFact table. Name the metadata object DIFT Order Fact. Specify that the table should be created as a SAS table with the physical name of OrderFact. Physically store the table in DIFT Orion Target Tables Library. Use the set of distinct columns from DIFT ORDER_ITEM and DIFT ORDERS. Store the metadata object in the Data Mart Development Ö Orion Target Data folder. Check in the new table object. 2. Defining Additional Target Tables Several additional tables must be defined for the demonstrations and exercises in subsequent sections. Check in all of the metadata table objects. • Create a target table metadata object named DIFT Recent Orders that defines column metadata for a SAS table that will be named Recent_Orders and stored in the DIFT Orion Target Tables Library. The columns in Recent_Orders should be the same columns that are defined in the OrderFact target table. Store the metadata object in the Data Mart Development Ö Orion Target Data folder. • Create a target table metadata object named DIFT Old Orders that defines column metadata for a SAS table that will be named Old_Orders and stored in the DIFT Orion Target Tables Library. The columns in Old_Orders should be the same columns that are defined in the OrderFact target table. Store the metadata object in the Data Mart Development Ö Orion Target Data folder. • (Optional) Create a target table metadata object named DIFT US Suppliers that defines column metadata for a SAS table that will be named US_Suppliers and stored in the DIFT Orion Target Tables Library. The columns in US_Suppliers should be the same columns that are defined in the DIFT Supplier Information external file object. Store the metadata object in the Data Mart Development Ö Orion Target Data folder.

4.2 Importing Metadata


Objectives „ „

Discuss SAS packages. Discuss importing and exporting of relational metadata.

13

Types of Metadata SAS Data Integration Studio enables you to import and export metadata for individual objects or sets of related objects. You can work with two kinds of metadata: Q SAS metadata in SAS Package format Q relational metadata (metadata for libraries, tables, columns, indexes, and keys) in formats that can be accessed with a SAS Metadata Bridge

14

4-23

4-24


SAS Package Metadata By importing and exporting SAS Package metadata, you can move the metadata for SAS Data Integration Studio jobs and related objects between SAS Metadata Servers. For example, you can create a job in a test environment, export it as a SAS Package, and import it into another instance of SAS Data Integration Studio in a production environment.

15

Relational Metadata By importing and exporting relational metadata in external formats, you can reuse metadata from third-party applications, and you can reuse SAS metadata in those applications as well. For example, you can use third-party data modeling software to specify a star schema for a set of tables. The model can be exported in Common Warehouse Metamodel (CWM) format. You can then use a SAS Metadata Bridge to import that model into SAS Data Integration Studio.

16


About SAS Metadata Bridges SAS Data Integration Studio can import and export relational metadata in any format that is ed by a SAS Metadata Bridge. For example, you can use third-party data modeling software to specify a star schema for a set of tables. The model can be exported in Common Warehouse Metamodel (CWM) format. You can then use a SAS Metadata Bridge to import that model into SAS Data Integration Studio.

17

Relational Metadata You can import and export relational metadata in any format that is accessible with a SAS Metadata Bridge. Relational metadata includes the metadata for the following objects: Q data libraries Q tables Q columns Q indexes Q keys (including primary keys and foreign keys)

18

4-25

4-26


Importing Relational Metadata

This demonstration illustrates importing metadata that was exported in CWM format from Oracle Designer. 1. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. 2. Log on using Barbara’s credentials to access her project repository. a. Select Barbara’s Work Repository as the connection profile. b. Click


c. Type Barbara as the value for the ID field and Student1 as the value for the field. d. Click


3. Click the Folders tab. 4. Expand Data Mart Development Ö Orion Target Data. 5. that the Orion Target Data folder is selected.


6. Select File Ö Import Ö Metadata…. The Metadata Importer wizard initializes and displays the window to enable the to select an import format. 7. Select Oracle Designer.

8. Click

.

4-27

4-28


9. Click

next to the File name field to open the Select a file window.

10. Navigate to S:\Workshop\dift. 11. Select OracleDesignerMetadata.dat. 12. Click

13. Click

.

to close the Select a file window.


4-29

14. that the folder location is set to /Data Mart Development/Orion Target Data. The final settings for the File Location window of the Metadata Importer wizard should be as shown:

15. Click

.

4-30


16. Review (and accept) the default settings.

17. Click

.


18. that Import as new metadata is selected.

19. Click

. The Metadata Location window is displayed.

4-31

4-32


20. Click

in the library field to assign a new library location.

21. On the Folders tab, navigate to Data Mart Development Ö Orion Target Data. 22. Select DIFT Orion Target Tables Library.

23. Click

to close Select a library window.


24. Review the settings for the Metadata Location window.

25. Click

.

4-33

4-34


26. The finish window displays the final settings. Review and accept the settings.

27. Click

.

28. An information window is opened.

29. Click

to not view the import log.


30. The Checkouts tree displays two new metadata table objects.

31. Right-click the Current Staff metadata table object and select Properties. 32. Type DIFT at the beginning of the default name.

4-35

4-36


33. Click the Columns tab to view some of the defined information. 34. Update the formats for each of the columns. Column Name

Column Format

Employee_ID

12.

Start_Date

Date9.

End_Date

Date9.

Job_Title

<none>

Salary

Dollar12.

Gender

$Gender6.

Birth_Date

Date9.

Emp_Hire_Date

Date9.

Emp_Term_Date

Date9.

Manager_ID

12.

35. Click



36. Right-click the Terminated Staff metadata table object and select Properties. 37. Type DIFT at the beginning of the default name.

4-37

4-38


38. Click the Columns tab to view some of the defined information. 39. Update the formats for each of the columns. Column Name

Column Format

Employee_ID

12.

Start_Date

Date9.

End_Date

Date9.

Job_Title

<none>

Salary

Dollar12.

Gender

$Gender6.

Birth_Date

Date9.

Emp_Hire_Date

Date9.

Emp_Term_Date

Date9.

Manager_ID

12.

40. Click



4-39

The metadata on the Checkouts tree now displays as the following:

41. Select Check Outs Ö Check In All. a. Type Adding metadata for Current and Terminated Staff tables as the value for the Title field. b. Click


c. Click


d. Click

. The table objects should no longer be on the Checkouts tab.

4-40


4.3 Solutions to Exercises 1. Defining the OrderFact Target Table a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Target Data. c. that the Orion Target Data folder is selected, and then select File Ö New Ö Table…. The New Table wizard opens. d. Type DIFT Order Fact as the value for the Name field. e. that the location is set to /Data Mart Development/Orion Target Data. f. Click

.

g. that the DBMS field is set to SAS. h. Select DIFT Orion Target Tables Library as the value for the Library field. i. Type OrderFact as the value for the Name field. j. Click

.

k. Expand the Data Mart Development Ö Orion Source Data folder on the Folders tab. l. From the Orion Source Data folder, click the DIFT ORDER_ITEM table object. m. Select all columns from DIFT ORDER_ITEM by clicking the Selected pane).

(all columns will be moved to

n. Collapse the list of columns for DIFT ORDER_ITEM. o. From the Orion Source Data folder, click the DIFT ORDERS table object. (all columns will be moved to the p. Select all columns from DIFT ORDERS by clicking Selected pane). An Error window opens saying that Order_ID will not be added twice. q. Click r. Click

. .

s. Accept the default attributes of the columns and then click t. Review the metadata listed in the finish window and then click appears on the Checkouts tab.

. . The new table object

u. Select Check Outs Ö Check In All. v. Type Adding metadata for Order Fact table as the value for the Title field. w. Click



x. Click


y. Click

. The table object should no longer be on the Checkouts tab.

4-41

2. Defining Additional Target Tables a. Define metadata for the DIFT Old Orders table. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Target Data. 3) that the Orion Target Data folder is selected, and then select File Ö New Ö Table…. The New Table wizard opens. 4) Type DIFT Old Orders as the value for the Name field. 5) that the location is set to /Data Mart Development/Orion Target Data. 6) Click

.

7) that the DBMS field is set to SAS. 8) Select DIFT Orion Target Tables Library as the value for the Library field. 9) Type Old_Orders as the value for the Name field. 10) Click

.

11) Expand the Data Mart Development Ö Orion Target Data folder on the Folders tab. 12) From the Orion Target Data folder, locate the DIFT Order Fact table object. 13) Select the DIFT Order Fact table object and click Selected pane. 14) Click

to move all columns to the

.

15) Accept the default attributes of the columns and then click 16) Review the metadata listed in the finish window and then click object appears on the Checkouts tab.

. . The new table

b. Define metadata for the DIFT Recent Orders table. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Target Data. 3) that the Orion Target Data folder is selected, and then select File Ö New Ö Table…. The New Table wizard opens. 4) Type DIFT Recent Orders as the value for the Name field. 5) that the location is set to /Data Mart Development/Orion Target Data.

4-42


6) Click

.

7) that the DBMS field is set to SAS. 8) Select DIFT Orion Target Tables Library as the value for the Library field. 9) Type Recent_Orders as the value for the Name field. .

10) Click

11) Expand the Data Mart Development Ö Orion Target Data folder on the Folders tab. 12) From the Orion Target Data folder, locate the DIFT Order Fact table object. 13) Select the DIFT Order Fact table object and click Selected pane.


.

14) Click

15) Accept the default attributes of the columns and then click 16) Review the metadata listed in the finish window and then click object appears on the Checkouts tab.

. . The new table

c. (Optional) Define metadata for the DIFT US Suppliers table. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Target Data. 3) that the Orion Target Data folder is selected, and then select File Ö New Ö Table…. The New Table wizard opens. 4) Type DIFT US Suppliers as the value for the Name field. 5) that the location is set to /Data Mart Development/Orion Target Data. 6) Click

.

7) that the DBMS field is set to SAS. 8) Select DIFT Orion Target Tables Library as the value for the Library field. 9) Type US_Suppliers as the value for the Name field. 10) Click

.

11) Expand the Data Mart Development Ö Orion Source Data folder on the Folders tab. 12) From the Orion Source Data folder, locate the DIFT Supplier Information table object. 13) Select the DIFT Supplier Information table object and click Selected pane. 14) Click

.



15) Accept the default attributes of the columns and then click

4-43

.

16) Review the metadata listed in the finish window and then click object appears on the Checkouts tab.

. The new table

d. Check in the newly created table objects. 1) Select Check Outs Ö Check In All. 2) Type Adding metadata for various target table objects as the value for the Title field. 3) Type Adding metadata for Old and Recent Orders, and US Suppliers as the value for the Description field. 4) Click


5) Click


6) Click


The metadata in the Orion Target Data folder should now resemble the following:

4-44


Chapter 5 Creating Metadata for Jobs 5.1

Introduction to Jobs and the Job Editor ...................................................................... 5-3 Demonstration: Populating Current and Terminated Staff Tables .......................................... 5-6

5.2

Using the SQL Transformation ........................................................................... 5-33 Demonstration: Populating the Product Dimension Table ............................................. 5-41 Exercises .............................................................................................................................. 5-68

5.3

Working with Jobs ........................................................................................................ 5-69 Demonstration: Investigate Mapping and Propagation Functionality .................................. 5-72 Demonstration: Chaining Jobs ............................................................................................. 5-99 Demonstration: Investigating Performance Statistics ........................................................ 5-105 Demonstration: Using the Reports Window ....................................................................... 5-120

5.4

Solutions to Exercises ............................................................................................... 5-129

5-2

Chapter 5 Creating Metadata for Jobs

5.1 Introduction to Jobs and the Job Editor


Objectives „ „

Define a job object. Discuss various features of jobs and the Job Editor window.

3

Overview At this point, metadata is defined for the following: „ various types of source tables „ desired target tables The next step is to load the targets in the data mart.

4

5-3

5-4


What Is a Job? A job is a collection of SAS tasks that creates output. SAS Data Integration Studio uses the metadata for each job to generate SAS code that reads sources and creates targets in physical storage.

5

Example Process Flow The following process flow diagram shows a job that reads data from a source table, sorts the data, and then writes the sorted data to a target table.

The components of this process flow perform the following functions: ƒ STAFF specifies metadata for the source table. ƒ Sort specifies metadata for the sort process. ƒ STAFF Sorted specifies metadata for the target table. 6


A Quick Example Before you proceed to further discussions on jobs and the Process Designer window, look at the creation of a simple job. The job creates two SAS data sets, one containing the current employees and the other containing the terminated employees.

(The aspects of creating a job are examined in more detail after some of the functionality is shown.) 7

Splitter Transformation The Splitter transformation is a transformation that can be used to create one or more subsets of a source.

8

5-5

5-6


Populating Current and Terminated Staff Tables

This demonstration shows the building of a job that uses the Splitter transformation. The final process flow diagram will look like the following:

1. If necessary, access SAS Data Integration Studio using Barbara’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. Select Barbara’s Work Repository as the connection profile. c. Click


d. Type Barbara as the value for the ID field and Student1 as the value for the field. e. Click



2. Click the Folders tab. 3. Expand Data Mart Development Ö Orion Jobs. 4. that the Orion Jobs folder is selected. 5. Select File Ö New Ö Job….

The New Job window opens.

5-7

5-8


6. Type DIFT Populate Current and Terminated Staff Tables as the value for the Name field. 7. that the Location is set to /Data Mart Development/Orion Jobs.

8. Click

If the Location is incorrect, click Data Mart Development Ö Orion Jobs.

.

and navigate to SAS Folders Ö


The Job Editor window opens:

5-9

5-10


9. Add the source data object to the process flow. a. If necessary, click the Folders tab. b. Expand Data Mart Development Ö Orion Source Data. c. Drag the DIFT Staff table object to the Diagram tab of the Job Editor.

When a job window is active, objects can be added to the diagram by right-clicking and selecting Add to Diagram.

10. Select File Ö Save to save diagram and job metadata to this point.


11. Add the Splitter transformation to the diagram. a. In the tree view, select the Transformations tab. b. Expand the Data grouping. c. Select the Splitter transformation.

d. Drag the Splitter transformation to the diagram. e. Center the Splitter so that it is lined up with the source table object.


5-11

5-12


13. Connect the source table object to the Splitter transformation. a. Place your cursor over the DIFT STAFF table object. A connection selector appears.

b. To connect the DIFT STAFF table object to the Splitter transformation, place your cursor over the connection selector until a pencil icon appears.

c. Click on this connection selector and drag to Splitter transformation.



5-13

15. Add the target table objects to the diagram. a. Click the Folders tab. b. If necessary, expand the Data Mart Development Ö Orion Target Data folder. c. Hold down the CTRL key and click on the two target table objects (DIFT Current Staff, DIFT Terminated Staff).

d. Drag the two objects to the Diagram tab of the Job Editor.

When a job window is active, objects can be added to the diagram by right-clicking and selecting Add to Diagram.

5-14


e. Arrange the source data objects so that they are separated.



5-15

17. Connect the Splitter transformation to the target table objects. The two target tables will be loaded with direct one-to-one column mappings of subsetted data. Therefore, no Table Loader transformation is needed for either of the target tables. Hence, the two temporary output table objects need to be deleted in order to connect the transformation to the table objects. a. Right-click on one of the temporary table objects of the Splitter transformation and select Delete.

5-16


b. Right-click on the second temporary table object of the Splitter transformation and select Delete.

18. Connect the Splitter transformation to each of the target table objects. a. Place your cursor over the Splitter transformation until a pencil icon appears.

b. When the pencil icon appears, click and drag the cursor to the first output table, DIFT Current Staff.


5-17

c. Again, place your cursor over the Splitter transformation until a pencil icons appears, and click and drag the cursor to the second output table, DIFT Terminated Staff.

The resulting process flow diagram should resemble the following:


5-18


20. Specify the properties of the Splitter transformation. a. Right-click on the Splitter transformation and select Properties.


b. Click the Row Selection tab. c. Specify the subsetting criteria for the DIFT Current Staff table object. 1) that the DIFT Current Staff table object is selected. 2) Select Row Selection Conditions as the value for the Row Selection Type field. 3) Click

below the Selection Conditions area.

The Expression window is opened.

5-19

5-20


4) Click the Data Sources tab. 5) Expand the STAFF table. 6) Select the Emp_Term_Date column. 7) Click

8) Click

.

in the operators area.

9) Type . (a period for a missing numeric value). The Expression Text area should now resemble the following:

10) Click

to close the Expression Builder window.


The Row Selection tab updates to the following:

5-21

5-22


d. Specify the subsetting criteria for the DIFT Terminated Staff table object. 1) that the DIFT Terminated Staff table object is selected. 2) Select Row Selection Conditions as the value for the Row Selection Type field. below the Selection Conditions area. The Expression window opens.

3) Click

4) Click the Data Sources tab. 5) Expand the STAFF table. 6) Select the Emp_Term_Date column. 7) Click 8) Click

. in the operators area.

9) Type . (a period for a missing numeric value). 10) Click

to close the Expression Builder window.

The Row Selection tab updates to the following:

e. Click the Mappings tab.


f.

5-23

that all Target Table columns have an arrow coming in to them (that is, all target columns will receive data from a source column).

g. Click

to close the Splitter Properties window.


5-24


22. Run the job. a. Right-click in background of the job and select Run.

A job can also be processed by selecting Actions Ö Run, or by clicking .

b. Click the Status tab in the Details area.

Note that all processes completed successfully.


23. Click

to close the Details view.

24. View the Log for the executed job. 25. Scroll to view the note about the creation of the DIFTTGT.CURRENT_STAFF table:

5-25

5-26


26. Scroll to view the note about the creation of the DIFTTGT.TERM_STAFF table:


27. View the data for the DIFT Current Staff table object. a. Right-click on the DIFT Current Staff table object and select Open.

b. When finished viewing the data, select File Ö Close to close the View Data window.

5-27

5-28


28. View the data for the DIFT Terminated Staff table object. a. Right-click on the DIFT Terminated Staff table object and select Open.

b. When finished viewing the data, select File Ö Close to close the View Data window. 29. Select File Ö Close to close the Job Editor. If necessary, save changes to the job. The new job object appears on the Checkouts tab. 30. Select Check Outs Ö Check In All. a. Type Adding job that populates current & terminated staff tables as the value for the Title field. b. Click


c. Click


d. Click



New Jobs New jobs are initialized by the New Job wizard. The New Job wizard names a job and the metadata location. (A description can optionally be specified.) Selecting creates an empty job.

10

Job Editor The Job Editor window enables you to create, maintain, and troubleshoot SAS Data Integration Studio jobs. To display this window for an existing job, right-click a job in the tree view and select Open. Job Editor Tabs:

11

Tab

Description

Diagram

Used to build and update the process flow for a job

Code

Used to review or update code for a job

Log

Used to review the log for a submitted job

Output

Used to review the output of a submitted job

5-29

5-30


Panes Used with the Job Editor Working with jobs in the Job Editor can be enhanced by using one or more panes available. Pane

Description

Details

Used to monitor and debug a job. To display, select View Ö Details from the desktop.

Runtime Manager Displays the run-time status of the current job, the last time that the job was executed in the current session, and the SAS Application Server that was used to execute the job. To display, select View Ö Runtime Manager from the Desktop. Actions History

Displays low-priority errors and warnings. To display, select View Ö Actions History from the Desktop.

12

Global Job Options Options can be specified for SAS Data Integration Studio that will affect all jobs.

13


Local Job Options Local options for jobs can be established using the job properties window. These options override global options for the selected job only – they do not affect any other jobs.

14

Process Flow Diagrams If SAS Data Integration Studio is to generate code for a job, you must define a process flow diagram that specifies the sequence of each source, target, and process in a job. In the diagram, each source, target, and process has its own metadata object.

15

5-31

5-32


Introduction to Transformations A transformation is a metadata object that specifies how to extract data, transform data, or load data into data stores. Each transformation that you specify in a process flow diagram generates or retrieves SAS code. You can also specify -written code in the metadata for any transformation in a process flow diagram.

16

Transformations Tree The Transformations tree organizes transformations into a set of folders. You can drag a transformation from the Transformations tree to the Job Editor, where you can connect it to source and target tables and update its default metadata. By updating a transformation with the metadata for actual sources, targets, and transformations, you can quickly create process flow diagrams for common scenarios. The display shows the standard Transformations tree.

17

5.2 Using the SQL Transformation


Objectives „

Discuss components of SQL ’s Designer window.

20

SQL Transformation The Splitter transformation was used in the previous job. As the course progresses, you will touch upon many of the available transformations. The next demonstration uses the SQL transformation, which can be used to create SQL queries that run in the context of SAS Data Integration Studio jobs. The SQL transformation features a graphical interface that provides a setting for building the statements and clauses that constitute queries. 21

5-33

5-34


“Open” the SQL Transformation The process of building the SQL query is performed in the Designer window. You access this window when you right-click the SQL transformation in a SAS Data Integration Studio job and select Open. You use the Designer window to create, edit, and review an SQL query.

22

SQL ’s Designer Window Designer Window Bar

Navigate pane

SQL Clauses pane

Properties pane

23


Navigate Pane: SQL

24

Navigate Pane: Table Selection

25

The Tables pane displays when a table object is selected in the Navigate pane, and when the Select keyword is selected in the Navigate pane. The Tables pane might also display when other aspects of particular s are requested (for instance, the surfacing of Having, Group by, and Order by information). The Tables pane is displayed in the same location as the SQL Clauses pane.

5-35

5-36


Navigate Pane: Select

26

Navigate Pane: Where

27


Loading the Product Dimension Table The DIFT Product Dimension table will be loaded from the defined metadata table objects, DIFT Product_List and DIFT Supplier Information, by using the SQL transformation. Three computed columns were designed: „ Product_Group „ Product_Category „ Product_Line The calculations for these columns are derived from a single variable, Product_ID, and take advantage of the -defined format PRODUCT. 28

Calculating Product_Group Using Product_ID Product_Group values are calculated by: Q

performing a grouping using Product_ID 210100100001 – 210100100050 group to 210100100000 210200100001 – 210200100019 group to 210200100000 210200200001 – 210200200050 group to 210200200000 210200300001 – 210200300125 group to 210200300000 …

Q

applying the PRODUCT. format to the result 210100100000 formats to Outdoor Things, Kids 210200100000 formats to A-Team, Kids 210200200000 formats to Bathing Suits, Kids 210200300000 formats to Eclipse, Kid’s Clothes …

29

This calculation creates a less fine view of the data.

5-37

5-38


Calculating Product_Category Product_Category values are calculated by: Q performing a grouping using Product_ID 210100100000 – 210100x00000 group to 210100000000 210200100000 – 210200x00000 group to 210200000000 220100100000 – 220100x00000 group to 220100000000 220200100000 – 220200x00000 group to 220200000000 … Q

applying the PRODUCT. format to the result 210100000000 formats to Children Outdoors 210200000000 formats to Children Sports 220100000000 formats to Clothes 220200000000 formats to Shoes …

30

This calculation creates an even less fine view of the data.

Calculating Product_Line Product_Line values are calculated by: Q performing a grouping using Product_ID 210100000000 – 210x00000000 group to 210000000000 220100000000 – 220x00000000 group to 220000000000 230100000000 – 230x00000000 group to 230000000000 240100000000 – 240x00000000 group to 240000000000 … Q

applying the PRODUCT. format to the result 210000000000 formats to Children 220000000000 formats to Clothes & Shoes 230000000000 formats to Outdoors 240000000000 formats to Sports …

31

This calculation creates an even less fine view of the data.


Computed Columns View of Data After they are defined, these new columns (Product_Line, Product_Category, Product_Group) can provide a wider or closer view of the data, depending on what is needed.

Product_Line (widest view) Product_Category (finer view) Product_Group (even finer view) Product_ID (finest view) 32

Computed Columns Calculations The necessary calculations for the three computed columns are as follows: Product_Group: put(int(product_id/100000)*100000,product.) Product_Category: put(int(product_id/100000000)*100000000,product.) Product_Line: put(int(product_id/10000000000)*10000000000,product.)

33

...

5-39

5-40


Computed Columns Calculations (Easier) The necessary calculations for the three computed columns are as follows: Product_Group: put(int(product_id/1e5)*1e5,product.) Product_Category: put(int(product_id/1e8)*1e8,product.) Product_Line: put(int(product_id/1e10)*1e10,product.)

34

...


Populating the Product Dimension Table

In this demonstration, you can take advantage of the SQL transformation to the DIFT Product_List and DIFT Supplier Information source tables to create the target table DIFT Product Dimension. In addition, three calculated columns are to be constructed. The final process flow diagram will look like the following:

1. If necessary, access SAS Data Integration Studio using Barbara’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. Select Barbara’s Work Repository as the connection profile. to close the Connection Profile window and access the Log On window.

c. Click

d. Type Barbara as the value for the ID field and Student1 as the value for the field. to close the Log On window.

e. Click

2. Click the Folders tab. 3. Expand Data Mart Development Ö Orion Jobs. 4. that the Orion Jobs folder is selected. 5. Select File Ö New Ö Job…. The New Job window opens. a. Type DIFT Populate Product Dimension Table as the value for the Name field. b. that the Location is set to /Data Mart Development/Orion Jobs. c. Click

.

5-41

5-42


The Job Editor window opens:


5-43

6. Add the source data objects to the process flow. a. If necessary, from the Folders tab, expand Data Mart Development Ö Orion Source Data. b. Select the DIFT PRODUCT_LIST table object and click and hold the CTRL key to also select DIFT Supplier Information external file object. c. Drag the objects to the Diagram tab of the Job Editor.

d. Arrange the source data objects so that they are separated.


5-44


8. Add the File Reader transformation to the diagram. a. In the tree view, click the Transformations tab. b. Expand the Access grouping. c. Select the File Reader transformation.

d. Drag the File Reader transformation to the diagram. e. Position the File Reader transformation so that it is next to (to the right of) the external file object, DIFT Supplier Information.

f.

Connect DIFT Supplier Information to the File Reader transformation.


9. Rename the temporary table object associated with the File Reader transformation. a. Right-click on the green temporary table object and select Properties.

b. Click the Physical Storage tab. c. Type FileReader as the value for the Name field.

5-45

5-46


d. Click

to close the File Reader Properties window.

10. Select File Ö Save to save diagram and job metadata to this point. 11. Add the SQL transformation to the diagram. a. In the tree view, click the Transformations tab. b. Expand the Data grouping. c. Select the SQL transformation.


5-47

d. Drag the SQL transformation to the diagram. e. Center the SQL so that it is in middle of the DIFT PRODUCT_LIST table object and the File Reader transformation.

12. Select File Ö Save to save diagram and job metadata to this point. 13. Add inputs to the SQL transformation. a. Place your cursor over the SQL transformation in the diagram to reveal the two default ports.

b. Connect the DIFT PRODUCT_LIST table object to one of the input ports for the SQL .

c. Connect the File Reader transformation (click on the temporary table icon, the File Reader and drag) to the other port of the SQL transformation.

, associated with

5-48


The diagram updates to the following:

14. Select File Ö Save to save diagram and job metadata to this point. 15. Add the DIFT Product Dimension table object as the output for the SQL . a. Right-click on the temporary table objects associated with the SQL transformation and select Replace.

b. In the Table Selector window, expand the Data Mart Development Ö Orion Target Data folders. c. Select DIFT Product Dimension table object. d. Select

.


The process flow diagram updates to the following:

16. Select File Ö Save to save diagram and job metadata to this point. 17. Review properties of the File Reader transformation. a. Right-click on the File Reader transformation and select Properties. b. Click the Mappings tab. c. that all target columns have a column mapping.

d. Click

to close the File Reader Properties.


5-49

5-50


19. Specify the properties of the SQL transformation. a. Right-click on the SQL transformation and select Open. The Designer window opens.

The Designer window initially has • a Diagram tab that you use to create SQL statements, configure the clauses that are contained in the statement, and edit the source to target mapping. The title of this tab changes to match the object selected in the Navigate pane. • a Navigate pane that enables navigation through various part of the Designer window • an SQL Clauses pane that enables you to add SQL clauses to the flow shown on the Diagram tab • a Properties pane that you use to display and update main properties of an object selected on the Diagram tab • a Tables pane that you use to review the list of columns in source and target tables


5-51

b. Select the item on the Diagram tab. that the is an Inner from the Properties pane.

The type of can also be verified and or changed by right-clicking on the item in the process flow of the SQL clauses. A pop-up menu appears that has the type of checked, but also enables selection of another type of .

5-52


c. Select the Where keyword in the Navigate pane to surface the Where tab. d. that the Inner will be executed based on the values of Supplier_ID columns from the sources being equal.

e. Add an additional Where clause to subset the data. 1) Click in the top portion of the Where tab. A row gets added with the logical AND as the Boolean operator.


2) Select Choose column(s)… from the drop-down list under the first Operand column.

The Choose Columns window opens. 3) Select Product_Level from the DIFT PRODUCT_LIST column listing.

4) Click

to close the Choose Columns window.

5-53

5-54


5) Type 1 (numeral one) in the field for the second Operand column and press ENTER.

The SQL where clause updates to the following:


f.

Select the Select keyword in the Navigate pane to surface the Select tab.

Note that four of the target columns map.

5-55

5-56


g. Map the Country column to Supplier_Country by clicking on the Country column and dragging to the Supplier_Country.

This leaves the three columns that are to be calculated. to fold the target table over the source table. This provides more room to work on the h. Click calculated expressions.

The Expression field is what needs to be filled in for the three columns.


5-57

i.

Re-order the columns so that Product_Group is first, then Product_Category, and then Product_Line.

j.

Specify an expression for Product_Group. 1) Locate the Product_Group column. In the Expression column, select Advanced… from the drop-down list.

5-58


The Expression window opens as displayed:

2) Locate and open the HelperFile.txt file in S:\Workshop\dift. 3) Copy the expression for Product_Group: put(int(product_list.product_id/1e5)*1e5,product.) 4) Paste the copied code in the Expression Text area.

5) Click

6) Click 7) Click

.

. to close the Expression window.


5-59

k. Specify an expression for Product_Category. 1) Locate the Product_Category column. 2) In the Expression column, select Advanced… from the drop-down list. The Expression window opens. 3) If necessary, access the HelperFile.txt file in S:\Workshop\dift. 4) Copy the expression for Product_Category: put(int(product_list.product_id/1e8)*1e8,product.) 5) Paste the copied code in the Expression Text area. 6) Click 7) Click

. .

8) Click l.

to close the Expression window.

Specify an expression for Product_Line. 1) Locate the Product_Line column. 2) In the Expression column, select Advanced… from the drop-down list. The Expression window is opened. 3) If necessary, access the HelperFile.txt file in S:\Workshop\dift. 4) Copy the expression for Product_Line: Put(int(product_list.product_id/1e10)*1e10,product.) 5) Paste the copied code in the Expression Text area. 6) Click 7) Click 8) Click

. . to close the Expression window.

5-60


The final settings for the calculated columns are as follows:

to fold the target table info back to the right side.

m. Click

n. Select File Ö Save to save changes to the SQL transformation. to return to the Job Editor.

o. Click 20. Run the job.

a. Right-click in background of the job and select Run.

A job can also be processed by selecting Actions Ö Run or by clicking on .


A warning occurred in the execution of the SQL . You see a change in the coloring of the transformation in the process flow and the symbol overlay.

b. Click the Status tab in the Details area.

5-61

5-62


c. Double-click the Warning for the SQL . The Warnings and Errors tab is moved forward with the warning message.


5-63

21. Edit the SQL transformation to fix the column mappings. a. Right-click on the SQL transformation and select Open. b. Click the Select keyword on the Navigate pane to surface the Select tab. Note the warning symbol,

, associated with each of the three calculated columns.

c. Map the Product_ID column to the Product_Group column (click on Product_ID in the Source table side and drag to Product_Group in the Target table side).

5-64


d. Map the Product_ID column to the Product_Category column. e. Map the Product_ID column to the Product_Line column.

f.

Select File Ö Save to save changes to the SQL transformation.

g. Click

to return to the Job Editor.


22. Run the job by right-clicking in background of the job and selecting Run.

The job runs without errors or warnings.

5-65

5-66


23. Click


24. View the log for the executed job by selecting the Log tab. 25. Scroll to view the note about the creation of the DIFTTGT.PRODDIM table:


5-67

26. Click the Diagram tab. 27. Right-click on the DIFT Product Dimension table object and select Open. 28. Scroll to the new columns and that data were calculated properly.

29. Select File Ö Close to close the View Data window. 30. Select File Ö Close to close the Job Editor window. The new job object appears on the Checkouts tab. 31. Select Check Outs Ö Check In All. a. Type Adding job that populates product dimension table as the value for the Title field. b. Click

. that the job object is selected.

c. Click


d. Click


5-68


Exercises

1. Loading the OrderFact Table Some specifics for creating the job to load the OrderFact table are show below: • Name the job DIFT Populate Order Fact Table. • Two tables should be ed together, DIFT ORDER_ITEM and DIFT ORDERS. • The SQL transformation will be used for the inner based on Order_ID from the input tables. • No additional processing is necessary beyond the SQL ; therefore, the targets can be loaded directly from the SQL transformation. • After ing that the table is created successfully (OrderFact should have 951,669 observations and 12 variables.), check in the job object. 2. Loading Recent and Old Orders Tables Some specifics for creating the job to load the Old Orders and Recent Orders tables are shown below: • Name the job DIFT Populate Old and Recent Orders Tables. • Use the SAS Splitter transformation to break apart the observations from the OrderFact table. • Old orders are defined to be orders older than those placed in 2005. An expression that can be used to find the observations for this data is the following: Order_Date < '01Jan2005'd • Recent orders are defined to be orders made within the years beginning in 2005. An expression that can be used to find the observations for this data is the following: Order_Date >= '01Jan2005'd • No additional processing is necessary beyond the Splitter; therefore, the targets can be loaded directly from Splitter transformation. • After ing that the tables are created successfully, check in the job object. Recent orders should have 615,396 observations and 12 variables, and old orders should have 336,273 observations and 12 variables.

5.3 Working with Jobs


Objectives „ „ „ „

Investigate mapping and propagation. Investigate chaining of jobs. Work with performance statistics. Generate reports on metadata for tables and jobs.

39

Automatic Mappings By default, SAS Data Integration Studio automatically creates a mapping when a source column and a target column have the same column name, data type, and length. Events that trigger automatic mapping include: Q connecting a source and a target to the transformation on the Diagram tab Q clicking Propagate on the toolbar or in the pop-up menu in the Job Editor window Q clicking Propagate on the Mappings tab toolbar and selecting a propagation option Q clicking Map all columns on the Mappings tab toolbar. 40

5-69

5-70


Disabling Automatic Mappings You can use the following methods to disable automatic mapping: Q disable automatic mapping globally for new SAS Data Integration Studio jobs using the Options window Q disable automatic mapping for the job using the Settings tool within the Job Editor window Q disable automatic mapping for the transformation in a job using the Settings tool at the top of the Mappings tab

If you disable automatic mapping for a transformation, you must maintain its mappings manually. 41

Automatic Propagation Automatic propagation sends column changes to tables when process flows are created. If you disable automatic propagation and refrain from using manual propagation, you can propagate column changes on the Mappings tab for a transformation that are restricted to the target tables for that transformation. Automatic propagation controls can occurs at various levels: Q Global Q Job Q Process flow Q Transformation

42


Level Global

Job

Control Automatically propagate columns in the Automatic Settings group box on the Job Editor tab in the Options window. (Click Options in the Tools menu to display the window.) This option controls automatic propagation of column changes in all new jobs. Automatically Propagate Job in the dropdown menu that displays when you click Settings in the toolbar on the Diagram tab in the Job Editor window. This option controls automatic propagation of column changes in the currently opened job.

5-71

Set Propagation Direction Select one of the following directions in the Propagation Direction group box: • From Beginning to End • From End to Beginning Select one of the following directions in the drop-down menu: • From Beginning to End • From End to Beginning Select one of the following directions in the pop-up menu: • To Beginning

Process flow

Propagate Columns in the pop-up menu on the Diagram tab in the Job Editor window. This option controls automatic propagation of column changes in the process flow in a currently opened • To End job.

Transformation

Include Transformation in Propagation in the drop-down menu that displays when you click Settings in the toolbar on the Mappings tab. This option controls automatic propagation of column changes in the selected transformation.

Not applicable

Transformation

Include Selected Columns in Propagation in the drop-down menu that displays when you click Settings in the toolbar on the Mappings tab to propagate changes to columns that you select in the source or target tables for the selected transformation.

Not applicable

5-72


Investigate Mapping and Propagation Functionality

This demonstration investigates automatic and manual propagation and mappings. The propagation is investigated from sources to targets only. Propagation can be done from targets to sources. 1. If necessary, access SAS Data Integration Studio using Bruno’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. that the connection profile is My Server. c. Click


d. Type Bruno as the value for the ID field and Student1 as the value for the field. e. Click


2. Create a new folder. a. Click the Folders tab in the tree view area. b. Right-click on Data Mart Development folder and select New Ö Folder. c. Type DIFT Additional Examples as the name for the new folder.


3. Create metadata table objects to demo mappings and propagation. a. Expand Data Mart Development Ö Orion Source Data. b. Locate the DIFT PRODUCTS metadata table object, right-click and select Copy. c. Right-click on the DIFT Additional Examples folder and select Paste. 4. Review the metadata for the DIFT PRODUCTS table. a. Right-click the DIFT PRODUCTS table object and select Properties. b. On the General tab, update the name of the metadata object to DIFT PRODUCTS (Copy).

c. Click the Columns tab.

Note that four columns are character and two are numeric. d. Click

to close the DIFT PRODUCTS Properties window.

5-73

5-74


5. Create a target table with the same attributes as the DIFT PRODUCTS (Copy) table. a. Right-click the DIFT PRODUCTS (Copy) table object and select Copy. b. Right-click the DIFT Additional Examples folder and select Paste.

c. Right-click the Copy of DIFT PRODUCTS (Copy) table object and select Properties. d. Type DIFT PRODUCTS Information as the value for the Name field (on the General tab).

e. Click the Physical Storage tab. f.

Click

next to the Library field.

g. Collapse the Orion Source Data folder. h. Expand the Orion Target Data folder. i.

Select DIFT Orion Target Tables Library.

j.

Click

to close the Select a library window.


k. that the DBMS appropriately updated to SAS with this new library selection.

l.

Type PRODUCTSInfo as the value for the Name field.

m. Click

to close the properties window.

The new table object appears under the DIFT Additional Examples folder.

6. Create a target table with different attributes as the DIFT PRODUCTS (Copy) table. a. Right-click the DIFT PRODUCTS (Copy) table object and select Copy. b. Right-click the DIFT Additional Examples folder and select Paste.

c. Right-click the Copy of DIFT PRODUCTS (Copy) table object and select Properties. d. Type DIFT PRODUCTS Profit Information as the value for the Name field (on the General tab).

e. Click the Columns tab. f.

Change attributes of the TYPE column. 1) Type Product Type as the value for the Description field. 2) Type 15 as the value for the Length field. 3) Type $15. as the value for the Informat field. 4) Type $15. as the value for the Format field.

5-75

5-76


g. Change attributes of the DESCRIP column. 1) Type Product Description as the value for the Description field. 2) Type 20 as the value for the Length field. 3) Type $20. as the value for the Informat field. 4) Type $20. as the value for the Format field. h. Change attributes of the PCODE column. 1) Type Product Code as the value for the Description field. 2) Select Numeric as the value for the Type field. 3) Delete the existing value for the Informat field. 4) Delete the existing value for the Format field. i.

Change attributes of the supplier column. 1) Type Product Supplier as the value for the Description field. 2) Type 32 as the value for the Length field. 3) Type $32. as the value for the Informat field. 4) Type $32. as the value for the Format field.

j.

Click

to add a new numeric column named Profit.


k. Click the Physical Storage tab. l.

Click


1) Collapse the Orion Source Data folder. 2) Expand the Orion Target Data folder. 3) Select DIFT Orion Target Tables Library. 4) Click


m. that the DBMS appropriately updated to SAS with this new library selection. n. Type PRODUCTSProfitInfo as the value for the Name field.

o. Click



5-77

5-78


7. Create a target table with no column definitions. a. Right-click on the DIFT Additional Examples folder and select New Ö Table. b. Type DIFT PRODUCTS Profit Information (2) as the value for the Name field.

c. Click

.

d. Select DIFT Orion Target Tables Library as the value for the Library field. e. Type PRODUCTSProfitInfo2 as the value for the Name field.

f.

Click

.

g. No columns will be selected, so click

.

h. No columns will be defined, so click

. A warning window opens as displayed.

to close the warning window.

i.

Click

j.

Review the final settings and click

.



5-79

8. that Automatic Mapping is set globally. a. Select Tools Ö Options. b. Click the Job Editor tab. c. that Automatically map columns and Automatically propagate columns are selected in the Automatic Settings area.

5-80


d. Click

to close the Options window.

9. Create initial job metadata to show automatic mappings. a. Right-click on DIFT Additional Examples folder and select New Ö Job. b. Type DIFT Mappings Example as the value for the Name field. c. that /Data Mart Development/DIFT Additional Examples is the value for the Location field.

d. Click

.

10. Add table objects and transformations to the Diagram tab of the Job Editor. a. Click and drag the DIFT PRODUCTS (Copy) table object to the Diagram tab of the Job Editor. b. Click and drag the DIFT PRODUCTS Information table object to the Diagram tab of the Job Editor. c. Click the Transformations tab. d. Expand the Data grouping of transformations. e. Click and drag the Extract transformation to the Diagram tab of the Job Editor. f.

Expand the Access grouping of transformations.

g. Click and drag the Table Loader transformation to the Diagram tab of the Job Editor.


5-81

h. Click the status indicator for the Extract transformation.

i.

Click

A similar message regarding no mappings defined can be found for the Table Loader transformation. to close the message.

11. Connect the objects in the process flow diagram and investigate the mappings. a. Connect the DIFT PRODUCTS (Copy) table object to the Extract transformation. b.

Connect the Extract transformation to the Table Loader transformation.

c. Connect the Table Loader transformation to the DIFT PRODUCTS Information table object. The process flow diagram updates to the following:

5-82


Mappings can be investigated by opening the Properties window for each transformation. Alternatively the Details section displays defined mappings for the selected transformation. d. If necessary, select View Ö Details to display the Details area within the Job Editor window. (Optionally, the Details section can be displayed by clicking the tool

in the Job Editor tools).

e. Click the Extract transformation on the Diagram tab. f.

Click the Mappings tab in the Details section.

Automatic mappings occur between a source column and a target column when these columns have the same column name, data type, and length. g. Click the Table Loader transformation on the Diagram tab. h. View the mappings in the Details section.

All mappings were automatically established between the source columns and the target columns.


5-83

12. Add additional table objects and transformations to the Diagram tab of the Job Editor. a. Click and drag the DIFT PRODUCTS (Copy) table object to the Diagram tab of the Job Editor. b. Click and drag the DIFT PRODUCTS Profit Information table object to the Diagram tab of the Job Editor. c. Click the Transformations tab. d. Expand the Data grouping of transformations. e. Click and drag the Extract transformation to the Diagram tab of the Job Editor. f.

Expand the Access grouping of transformations.

g. Click and drag the Table Loader transformation to the Diagram tab of the Job Editor.

13. Connect the new objects in the process flow diagram and investigate the mappings. a. Connect the DIFT PRODUCTS (Copy) table object to the Extract transformation. b. Connect the Extract transformation to the Table Loader transformation. c. Connect the Table Loader transformation to the DIFT PRODUCTS Profit Information table object. The process flow diagram updates to the following:

d. Select File Ö Save.

5-84


14. Investigate the mappings for the new transformations. a. If necessary, select View Ö Details to display the Details area within the Job Editor window. b. Click the newest Extract transformation on the Diagram tab. c. Click the Mappings tab in the Details section.

The columns are all automatically propagated, and mapped. d. Click the newest Table Loader transformation. e. View the mappings in the Details section.

Several points are to be noted: • The TYPE, DESCRIP and supplier columns did not map automatically because of the differing lengths. • The PCODE columns mapped, even though one is character and one is numeric. SAS Data Integration Studio does an automatic character-to-numeric conversion. • No columns mapped to the column Profit.


f.

5-85

Manually map the TYPE columns. A Warning will appear. An expression could be written to avoid this warning – otherwise the Log for the job will contain a WARNING message as well.

g. Manually map the DESCRIP columns. h. Manually map the supplier columns.

i.

Scroll in the target table side of the Mappings tab to view the automatic expression for the Sex column.

5-86


15. Turn off automatic mappings for the job and rebuild the connections for the first part of this process flow. a. Right-click in the background of the Job Editor and select Settings Ö Automatically Map Columns.

b. Click on each of the connections for the first flow, right-click and select Delete.


c. Re-connect the DIFT PRODUCTS (Copy) table object to the Extract transformation.

d. Click on the status indicator for the Extract transformation.

e. Click f.

to close the message.

Click the newest Extract transformation on the Diagram tab.

g. Click the Mappings tab in the Details section. Note that there are no mappings defined.

5-87

5-88


h. Click

on the Mappings tab tool set. All columns are mapped.

i.

Click

on the Mappings tab tool set. All columns are no longer mapped.

j.

Click

next to the Name column on the Source table side.

k. Click

next to the Name column on the Target table side.

l.

on the Mappings tab tool set, and that Include Selected Columns in Click Mapping is selected.

m. Click

on the Mappings tab tool set. The selected column is manually mapped.


16. Close the current job; no need to save any changes. a. Select File Ö Close. A warning window is displayed.

b. Click

.

17. Create initial job metadata to show propagation. a. Right-click on DIFT Additional Examples folder and select New Ö Job. b. Type DIFT Propagation Example as the value for the Name field. c. that /Data Mart Development/DIFT Additional Examples is the value for the Location field.

d. Click

.

18. Select View Ö Layout Ö Top to Bottom.

5-89

5-90


19. Add table objects and transformations to the Diagram tab of the Job Editor. a. Click and drag the DIFT PRODUCTS (Copy) table object to the Diagram tab of the Job Editor. b. Click and drag the DIFT PRODUCTS Profit Information (2) table object to the Diagram tab of the Job Editor. c. Click the Transformations tab. d. Expand the Data grouping of transformations. e. Click and drag the Extract transformation to the Diagram tab of the Job Editor. f.

Click and drag the Sort transformation to the Diagram tab of the Job Editor.

g. Expand the Access grouping of transformations. h. Click and drag the Table Loader transformation to the Diagram tab of the Job Editor.


5-91

20. Connect the source to the first transformation and investigate the propagation. a. Connect the DIFT PRODUCTS (Copy) table object to the Extract transformation. b. Click the Extract transformation on the Diagram tab. c. Click the Mappings tab in the Details section.

All columns from the source are propagated and mapped to the target. 21. Turn off automatic propagation and mappings for the job. a. Right-click in the background of the job and select Settings Ö Automatically Propagate Columns. b. Right-click in the background of the job and select Settings Ö Automatically Map Columns.

These selections can also be made from the Job Editor’s tool set.

5-92


22. Break the connection between the DIFT PRODUCTS (Copy) table object and the Extract transformation (click on the connection, right-click, and select Delete). 23. Remove the already propagated columns from the output table of the Extract transformation. a. Right-click on the output table object associated with the Extract transformation and select Properties.

b. Click the Columns tab. c. Select all columns (hold down the CTRL key and then click the A key). d. Right-click and select Delete. e. Click

. to close the properties window for the Extract output table.


24. Reconnect the source to the Extract transformation and investigate the propagation. a. Connect the DIFT PRODUCTS (Copy) table object to the Extract transformation. b. Click the Extract transformation on the Diagram tab. c. Click the Mappings tab in the Details section.

All columns from the source are NOT propagated and mapped to the target. 25. Manually propagate all columns. a. Click

to propagate all columns from source to targets.

b. Click

on the Mappings tab tool set. All columns are removed from the target table side.

5-93

5-94


26. Manually propagate selected columns. a. On the Source table side, click the TYPE column. b. Hold down the CTRL key and click both the DESCRIP, PRICE, and COST columns. c. Click

, and then select Selected Source Columns Ö To Targets.

Only the selected columns are propagated and mapped.


27. Manually propagate selected columns to the end. a. Connect the Extract transformation to the Sort transformation. b. Connect the Sort transformation to the Table Loader transformation. c. Connect the Table Loader transformation to the target table object, DIFT PRODUCT Profit Information (2).

d. Click the Extract transformation on the Diagram tab. e. Click the Mappings tab in the Details section. f.

Select all four of the target columns.

5-95

5-96


g. Click

, and then select Selected Target Columns Ö To End.

h. Click the Sort transformation on the Diagram tab. i.

Click the Mappings tab in the Details section. that only the three selected columns were propagated (and mapped).

j.

Click the Table Loader transformation on the Diagram tab.

k. Click the Mappings tab in the Details section. The propagation does not propagate to the target table. This needs to be done manually.


l.

Click

to propagate all columns from source to targets.

28. Save and close the job. a. Select File Ö Save. b. Select File Ö Close.

5-97

5-98


Chaining Jobs Existing jobs can be added to the Diagram tab of the Job Editor window. These jobs are added to the control flow in the order that they are added to the job. This sequence is useful for jobs that are closely related - however, the jobs do not have to be related. You can always change the order of execution for the added jobs in the Control Flow tab of the Details pane.

44

The added jobs are linked by dashed-line control flow arrows and not by solid-line data flow arrows.


Chaining Jobs

1. If necessary, access SAS Data Integration Studio using Bruno’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. that the connection profile is My Server. c. Click




2. Click the Folders tab. 3. Expand Data Mart Development Ö DIFT Additional Examples. 4. that the Jobs folder is selected. 5. Select File Ö New Ö Job…. The New Job window opens. a. Type DIFT Create a Job of Jobs as the value for the Name field. b. that the Location is set to /Data Mart Development/Orion Jobs.

c. Click

. The Job Editor window is displayed.

6. Return to the Folders tab, and expand the folders to Data Mart Development Ö Orion Jobs.

5-99

5-100


7. Select the DIFT Populate Old and Recent Orders Tables job and drag it to the Diagram tab of the job editor window. 8. Select the DIFT Populate Order Fact Table job and drag it to the Diagram tab of the job editor window. The first job connects automatically to the second job.

The OrderFact table, created in the DIFT Populate Order Fact Table job, is the source table for the DIFT Populate Old and Recent Orders Tables job. Therefore, DIFT Populate the Order Fact Table job should run first and then the DIFT Populate Old and Recent Orders Tables job.


5-101

9. Click the Control Flow tab in the Details pane. 10. Select the DIFT Populate Order Fact Table job. 11. Click

from the Control Flow tab’s toolbar.

The process flow diagram updates to:

12. Select View Ö Layout Ö Left to Right. The diagram updates with the correct ordering and in the horizontal view.

13. Select File Ö Save.

5-102


14. Click

to execute the job.

15. Click the Status tab in the Details pane and that the jobs both ran successfully in concurrence.

16. Select File Ö Close to close the job editor window.


Investigating Performance Statistics Performance statistics for a SAS Data Integration Studio job can be tracked as long as the following prerequisites are met: Q The logging facility is enabled on the SAS Workspace Server that executes the job. (The logging facility is enabled by default.) Q The collect runtime statistics option is enabled for the job.

46

Enabling Performance Statistics To collect runtime statistics for an existing job Q open the job in the Job Editor window Q right-click on the canvas and select Collect Runtime Statistics and, if desired, also turn on collection of table statistics by selecting Collect Table Statistics.

47

5-103

5-104


Enable Statistics in Options Window To collect runtime statistics for all new jobs: Q select Tools Ö Options… Q click the Job Editor tab Q On the Job Editor tab, in the Statistics area: Q click the check box for Collect runtime statistics Q click the check box for Collect table statistics.

48


Investigating Performance Statistics





2. Click the Folders tab. 3. Expand Data Mart Development Ö DIFT Demo. 4. Double-click on the job DIFT Test Job – OrderFact Table Plus to open it in the Job Editor.

5-105

5-106


5. If necessary, select View Ö Details. 6. Right-click in the background of the Diagram tab and select Collect Runtime Statistics. 7. Right-click in the background of the Diagram tab and select Collect Table Statistics. (The Collect Table Statistics menu item is dimmed unless the Collect Runtime Statistics item is selected.)


8. Click

to execute the job.

9. Click the Statistics tab in the Details pane.

The Collect Table Statistics choice populates the Records field - otherwise, a zero is listed. 10. Scroll to the right in the table of statistics.

5-107

5-108


11. Click

on the Statistics tab toolbar. The Save window is displayed.

12. Navigate to S:\Workshop\dift\reports. 13. Type DIFTTestJobStatsRun1 as the value for the File name field.

14. Click

.


5-109

15. Access a Windows Explorer window by selecting Start Ö All Programs Ö Accessories Ö Windows Explorer. 16. Navigate to S:\Workshop\dift\reports. 17. Double-click DIFTTestJobStatsRun1.csv. Microsoft Excel opens and displays the saved statistics.

18. Click

Ö

to close Microsoft Excel.

5-110


19. Click on the Statistics tab toolbar. The table view changes to a graphical view, which is a line graph by default.

20. Click default.

between Line Graph and Bar Chart. All of the reported statistics are selected by


21. Clear the following selections:

22. Click

Current Memory Operating System Memory Current I/O Operating System I/O

. The line graph updates to the following view of the requested statistics:

5-111

5-112



Records Real Time U Time

24. Click the following selections:

Current Memory Operating System Memory

25. Click




Current Memory Operating System Memory

27. Click the following selections:

Current I/O Operating I/O

28. Click


5-113

5-114


29. Click Bar Chart on the Statistics tab toolbar. The table view changes to a bar chart view.

The bar chart quickly tells us that the Table Loader transformation took almost three times as long as the rank transformation. The SQL transformation ran very quickly, compared to the other transformations. 30. Place your cursor over the bar for the SQL transformation. Tooltip text appears with summarized information about the processing of the SQL transformation.


31. Place your cursor over the bar for the Table Loader transformation. Tooltip text appears with summarized information about the processing of the Table Loader transformation.

5-115

5-116


32. Click

and choose the Table Loader.

The bar chart updates to a single bar for just the Table Loader transformation. The scaling for times is easier to read for this selected transformation.


33. Click

34. Click

and choose All transformations.

on Statistics tab toolbar. The Print window opens as displayed.

The graph can be written to a file, and then printed from the file. 35. Click

.

5-117

5-118


36. Click

on Statistics tab toolbar. The Save to File window opens as displayed.

37. If necessary, navigate to the S:\Workshop\dift\reports directory. 38. Accept the default name and type of file.

39. Click

.

40. With Job Editor window active, select File Ö Close. 41. If prompted, do not save any changes.


About Reports The reports featured in SAS Data Integration Studio can be used to generate reports - metadata for tables and jobs can be reviewed in a convenient format. Reports enable you to: Q find information about a table or job quickly Q compare information between different tables or jobs Q obtain a single file that contains summary information of all tables or jobs in HTML, RTF, or PDF format Q perform custom behaviors that are defined by created plug-in SAS code, Java code, or both

50

5-119

5-120


Using the Reports Window



d. Type Bruno as the value for the ID field and Student1 as the value for the field. e. Click 2. Click

to close the Log On window. on the SAS Data Integration Studio toolbar.

The Reports window opens as displayed.

The Reports window can also be accessed by selecting Tools Ö Reports.


3. Click Tables Report.

4. Click

from the Reports window’s toolbar. The Report Options window is displayed.

that the default report format is set to HTML. A valid CSS file can be specified as well as additional ODS HTML statement options.

5-121

5-122


5. Click

next to the Format field and choose RTF.

Valid ODS RTF statement options can be specified. 6. Click

next to the Format field and choose PDF.

Valid ODS PDF statement options can be specified. 7. Click 8. Click

to close the Report Options window. next to the Show field and choose Job.


The Reports window updates to the following:

5-123

5-124


9. Specify the location and name of the job reports. a. Click opened.

next to the Default Location field. The Select a directory window is

b. Navigate to S:\Workshop\dift\reports. c. Click

to create a new folder.

d. Type JobsReport as the name of the new folder and press ENTER. e. Navigate to the new JobsReport folder.

f.

Click

to close the Select a directory window.


10. Click Job Documentation and then select

A Report View window is displayed.

.

5-125

5-126


11. Click

. A browser opens with the report.

12. When done viewing the report, select File Ö Close from the browser window.


13. To create a document object, click Job Documentation and then select

.

The Save As Document window is opened. 14. Navigate to the Data Mart Development Ö DIFT Additional Examples folder. 15. Type Job Information as the value for the Name field.

16. Click

.

5-127

5-128


The document object appears on the Folders tab.

17. In the Reports window, click

.

18. Click the Folders tab. 19. If necessary, expand the Data Mart Development Ö DIFT Additional Examples folders. 20. Right-click on the Job Information (Document) and select Open. The document object is a link to the physical file. 21. Select File Ö Close from the browser window.


5-129

5.4 Solutions to Exercises 1. Loading the Order Fact Table a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Jobs. c. that the Orion Jobs folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. 1) Type DIFT Populate Order Fact Table as the value for the Name field. 2) that the Location is set to /Data Mart Development/Orion Jobs. 3) Click

. The Job Editor window opens.

e. Add the source data objects to the process flow. 1) Click the Folders tab. 2) If necessary, expand Data Mart Development Ö Orion Source Data. 3) Right-click on the DIFT ORDER_ITEM table object and select Add to Diagram. 4) Right-click on the DIFT ORDERS table object and select Add to Diagram. 5) Click on the DIFT ORDERS object on the Diagram tab of the Job Editor and drag it away from the DIFT ORDER_ITEM object (initially, these objects are one on top of the other). f. Select File Ö Save to save diagram and job metadata to this point. g. Add the SQL transformation to the diagram. 1) In the tree view, select the Transformations tab. 2) Expand the Data grouping. 3) Select the SQL transformation. 4) Drag the SQL transformation to the diagram. 5) Center the SQL so that it is in middle of the DIFT ORDER_ITEM table object and DIFT ORDERS table object. h. Add inputs to the SQL transformation. 1) Click on the connection selector for DIFT ORDER_ITEM and drag to one of the input ports for the SQL transformation. 2) Click on the connection selector for DIFT ORDERS and drag to other input port for the SQL transformation. i. Select File Ö Save to save diagram and job metadata to this point.

5-130


j. Add target table to the diagram. 1) Right-click on the temporary output table for the SQL and select Replace…. 2) the Folders tab is selected. 3) Expand the Data Mart Development Ö Orion Target Data folder. 4) Select DIFT Order Fact. 5) Click

.

k. Select File Ö Save to save diagram and job metadata to this point. l. Review the properties of the SQL transformation. 1) Right-click on the SQL transformation and select Open. The Designer window opens. 2) Select the item on the Diagram tab. that the is an Inner from the Properties pane. 3) that the Inner will be executed based on the values of Order_ID columns from the sources being equal. 4) Select the Select keyword on the Navigate pane to surface the Select tab. 5) that all target columns are mapped. m. Click


n. Right-click in the background of the job and select Run. o. that the job runs successfully. p. Select the Log tab and that DIFTTGT.ORDERFACT is created with 951669 observation and 12 variables. q. Click the Diagram tab. r. Right-click on DIFT Order Fact and select Open. s. Review the data then choose File Ö Close. t. Select File Ö Close to close the job editor. u. Select Check Outs Ö Check In All. 1) Type Adding job that populates order fact table as the value for the Title field. 2) Click


3) Click


4) Click



5-131

2. Loading Recent and Old Orders Tables a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Jobs. c. that the Orion Jobs folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. 1) Type DIFT Populate Old and Recent Orders Tables as the value for the Name field. 2) that the Location is set to /Data Mart Development/Orion Jobs. 3) Click


e. Add the source data object to the process flow. 1) If necessary, click the Folders tab. 2) Expand Data Mart Development Ö Orion Target Data. 3) Drag the DIFT Order Fact table object to the Diagram tab of the Job Editor. f. Select File Ö Save to save diagram and job metadata to this point. g. Add the Splitter transformation to the diagram. 1) In the tree view, select the Transformations tab. 2) Expand the Data grouping. 3) Select the Splitter transformation. 4) Drag the Splitter transformation to the diagram. 5) Center the Splitter so that it is lined up with the source table object. h. Select File Ö Save to save diagram and job metadata to this point. i. Connect the source table object to the Splitter transformation. 1) Place your cursor over the DIFT Order Fact table object. A connection selector appears. 2) To connect the DIFT Order Fact table object to the Splitter transformation, place your cursor over the connection selector until a pencil icon appears. Click on this connection selector and drag to the Splitter transformation. j. Select File Ö Save to save the diagram and job metadata to this point.

5-132


k. Add the target table objects to the diagram. 1) Right-click on one of the temporary output tables for the Splitter and select Replace…. 2) that the Folders tab is selected. 3) Expand the Data Mart Development Ö Orion Target Data folder. 4) Select DIFT Old Orders. 5) Click

.

6) Right-click on the other temporary output table for the Splitter and select Replace…. 7) that the Folders tab is selected. 8) Expand the Data Mart Development Ö Orion Target Data folder. 9) Select DIFT Recent Orders. 10) Click

.

11) If necessary, separate the two target table objects. The process flow diagram should resemble the following:

l. Select File Ö Save to save diagram and job metadata to this point.


5-133

m. Specify the properties of the Splitter transformation. 1) Right-click on the Splitter transformation and select Properties. 2) Click the Row Selection tab. 3) Specify the subsetting criteria for the DIFT Recent Orders table object. a) that DIFT Recent Orders table object is selected. b) Select Row Selection Conditions as the value for the Row Selection Type field. c) Click opened.

below the Selection Conditions area. The Expression window is

d) Click the Data Sources tab. e) Expand the OrderFact table. f) Select the Order_Date column. g) Click h) Click


i) Type ‘01jan2005’d. .

j) Click

k) Click l) Click

. to close the Expression Builder window.

The Selection Conditions area on the Row Selection tab updates to the following:

5-134


4) Specify the subsetting criteria for the DIFT Old Orders table object. a) that the DIFT Old Orders table object is selected. b) Select Row Selection Conditions as the value for the Row Selection Type field. below the Selection Conditions area. The Expression window is

c) Click opened.

d) Click the Data Sources tab. e) Expand the OrderFact table. f) Select the Order_Date column. g) Click h) Click


i) Type ‘01jan2005’d. .

j) Click

k) Click l) Click

. to close the Expression Builder window.

The Selection Conditions area on the Row Selection tab updates to the following:

5) Click the Mappings tab. 6) that all Target Table columns have an arrow coming in to them (that is, all target columns will receive data from a source column). 7) Click


n. Select File Ö Save to save the diagram and job metadata to this point.


5-135

o. Run the job. 1) Right-click in background of the job and select Run. 2) Select the Status tab in the Details area. Note that all processes completed successfully. 3) Click


4) View the Log for the executed Job. 5) Scroll to view the notes about the creation of the DIFTTGT.RECENT_ORDERS table and the creation of the DIFTTGT.OLD_ORDERS table:

6) View the data for the DIFT Recent Orders table object. a) Right-click on the DIFT Recent Orders table object and select Open. b) When finished viewing the data, select File Ö Close to close the View Data window. 7) View the data for the DIFT Old Orders table object. a) Right-click on the DIFT Old Orders table object and select Open. b) When finished viewing the data, select File Ö Close to close the View Data window. p. Select File Ö Close to close the Job Editor. The new job object appears on the Checkouts tab.

5-136


q. Select Check Outs Ö Check In All. 1) Type Adding job that populates old & recent orders as the value for the Title field. 2) Click


3) Click


4) Click


The Orion Jobs folder should resemble the following:

Chapter 6 Orion Star Case Study 6.1

Exercises ......................................................................................................................... 6-3 Define and Load the Customer Dimension Table ................................................................... 6-6 Define and Load the Organization Dimension Table .............................................................. 6-8 Define and Load the Time Dimension Table ......................................................................... 6-11

6.2

Solutions to Exercises ................................................................................................. 6-13 Define and Load the Customer Dimension Table ................................................................. 6-13 Define and Load the Organization Dimension Table ............................................................ 6-19 Define and Load the Time Dimension Table ......................................................................... 6-29

6-2

Chapter 6 Orion Star Case Study

6.1 Exercises

6.1 Exercises

Orion Star Target Star Schema (Review) One goal of this class is to produce, from the Orion Star source data, a dimensional data model that is a star schema. Organization Dimension

Customer Dimension

Order Fact Table

Product Dimension

Time Dimension

2

Order Fact Table Diagram The Order Fact table process was sketched to be this.

OrderFact Computed Columns

The source table and target table objects need to be defined in metadata, as well as the process flow.

NONE!

There are no computed columns defined for the target table. Orders

Order_Item

This table was loaded in a previous exercise. 3

6-3

6-4


Product Dimension Table Diagram The Product Dimension table process was sketched to be this.

ProdDim Computed Columns

The source table and target table objects need to be defined in metadata, as well as the process flow. There are three computed columns defined for the target table.

Product_Category Product_Group Product_Line

Product_List Supplier Information

This table was loaded in a previous demonstration. 4

Customer Dimension Table Diagram The Customer Dimension table process was sketched to be this.

CustDim Computed Columns

The source table and target table objects need Customer_Age to be defined in Customer_Age_Group metadata, as well as the process flow. There are two computed columns defined for the target table. Customer 5

Customer Types

6.1 Exercises

Organization Dimension Table Diagram The Organization Dimension table process was sketched to be this. The source table and target table objects need to be defined in metadata, as well as the process flow. There are four computed columns defined for the target table.

OrgDim

Company Department Group Section

Organization

Computed Columns

Staff

6

Time Dimension Table Diagram The Time Dimension table process was sketched to be this. The target table metadata object must be created, as well as the job that uses written code to load the target.

7

TimeDim Computed Columns All Columns!! (-written code generates the TimeDim table.)

6-5

6-6


In this exercise set you will define and load the three remaining dimension tables. For each of these “target” tables, you will need to define: • metadata for the source tables • metadata for the target table • metadata for the process flow to move the data from the source(s) to the target In addition, metadata objects for data sources needed for showing features of the software will be defined.

Define and Load the Customer Dimension Table 1. Defining the Customer Dimension Source Table Metadata Two source tables are needed for the customer dimension table, Customer and Customer Types. The Customer table is a SAS table located in the DIFT Orion Source Tables Library. The Customer Types table object was defined in metadata as part of a demonstration in Chapter 3. If necessary, complete these steps to define the Customer Types table object. Metadata objects for tables should be placed in the /Data Mart Development/Orion Source Data folder. Use the naming convention of DIFT for the metadata table objects. 2. Defining the Customer Dimension Target Table Metadata The following are some specifics for defining the metadata for the customer dimension table: • Name the metadata table object DIFT Customer Dimension. • Store the metadata object in the /Data Mart Development/Orion Target Data folder. • The physical table, a SAS table, is to be named CustDim and stored in the DIFT Orion Target Tables Library. • From the Customer table, import the following column metadata: ORIGINAL COLUMN (in Customer):

RENAME TO…

Customer_ID Country

Customer_Country

Gender

Customer_Gender

Customer_Name Customer_FirstName Customer_LastName Birth_Date

Customer_Birth_Date

• From the Customer Types table, import the following column metadata: ORIGINAL COLUMN (in Customer Types): Customer_Type Customer_Group

6.1 Exercises

6-7

• Add two new columns with the following properties: NAME Customer_Age

DESCRIPTION

LENGTH

Customer Age

Customer_Age_Group Customer Age Group

TYPE

8

Numeric

12

Character

• Create a simple index for Customer_ID. 3. Loading the Customer Dimension Target Table Some specifics for creating the job to load the DIFT Customer Dimension table are shown below: • Name the job DIFT Populate Customer Dimension Table. • Store the metadata object in the /Data Mart Development/Orion Jobs folder. • Two tables should be ed together, DIFT Customer and DIFT Customer Types. • The SQL transformation will be used for the , based on the Customer_Type_ID variable from the input tables. • The should include all rows from the DIFT Customer table and only matching rows from the DIFT Customer Types table. • The data should be filtered using the following expression: Customer_Type_ID>=0 • Two computed columns will be constructed, Customer_Age and Customer_Age_Group. The calculations for the columns are shown below: Column Customer_Age

Expression Floor(Yrdif(customer.birth_date, today(), 'actual'))

Customer_Age_Group Put(calculated customer_age,agegroup.)

The metadata for the columns can be imported from the DIFT Customer Dimension metadata object and the text for the expressions can be found in HelperFile.txt.

4. Check-In the Metadata Objects for the Customer Dimension After ing that the job ran successfully, check in all the objects from the project repository.

6-8


Define and Load the Organization Dimension Table 5. Defining the Organization Dimension Source Table Metadata Two source tables are needed for the organization dimension table, STAFF and ORGANIZATION. The ORGANIZATION table is a SAS table found in the DIFT Orion Source Tables Library. The STAFF table object was defined in metadata as part of the exercises at the end of Chapter 3 (Exercise #4, solution on pages 3-89 to 3-92). If necessary, complete these steps to define the Staff table object. Metadata objects for tables should be placed in the /Data Mart Development/Orion Source Data folder. Use the naming convention of DIFT for the metadata table objects. 6. Organization Dimension Target Table Metadata The following are some specifics for defining the metadata for the organization dimension table: • Name the metadata table object DIFT Organization Dimension. • Store the metadata object in the /Data Mart Development/Orion Target Data folder. • Name the physical table, a SAS table, OrgDim, and store it in the DIFT Orion Target Tables Library. • From the Organization table, import the following column metadata: ORIGINAL COLUMN (in Organization):

RENAME TO…

Employee_ID Org_Name

Employee_Name

Country

Employee_Country

• From the Staff table, import the following column metadata: ORIGINAL COLUMN (in Staff):

RENAME TO…

Job_Title Salary Gender

Employee_Gender

Birth_Date

Employee_Birth_Date

Emp_Hire_Date

Employee_Hire_Date

Emp_Term_Date

Employee_Term_Date

• Add four new columns with the following properties: NAME

LENGTH

TYPE

Group

40

Character

Section

40

Character

Department 40

Character

Company

Character

30

6.1 Exercises

6-9

• Create a simple index for Employee_ID. 7. Loading the Organization Dimension Target Table Some specifics for creating the job to load the DIFT Organization Dimension table are shown below: • Name the job DIFT Populate Organization Dimension Table. • Store the metadata object in the /Data Mart Development/Orion Jobs folder. • Two tables should be ed together, DIFT Organization and DIFT Staff. • The SQL transformation will be used for the , based on the Employee_ID variable from the input tables. • The Table Loader transformation will be used for the final loading of the OrgDim table. • The should include all rows from the Staff table and only matching rows from the Organization table. • The data should be subset using the following expression: Org_Level=1 • Four computed columns must be constructed (Company, Department, Section, and Group). Because of the nested complexities of the calculations for these columns, it is best to create four temporary variables and use the temporary variables to build the desired ones. • The four temporary columns that must be added, along with their length, type, and expressions, are shown below: Name

Length

Type

Expression

_Group

8

Numeric input(put(organization.employee_id,orgdim.),12.)

_Section

8

Numeric input(put(calculated _Group, orgdim.),12.)

_Department 8

Numeric input(put(calculated _Section, orgdim.),12.)

_Company

Numeric input(put(calculated _Department, orgdim.),12.)

8

The text for the expressions can be found in HelperFile.txt.

• The calculations for the desired computed columns are shown below: Name

Expression

Group

put(calculated _Group, org.)

Section

put(calculated _Section, org.)

Department put(calculated _Department, org.) Company

put(calculated _Company, org.)

The metadata for the columns can be imported from the DIFT Organization Dimension metadata object and the text for the expressions can be found in HelperFile.txt.

6-10


• that all columns for the Table Loader’s target table have a defined mapping. • The warning message regarding a compressed data set occurs because the DIFT Organization table has a COMPRESS=YES property set. Edit the table properties to set this to NO.

8. Check-In the Metadata Objects for the Organization Dimension After ing the job has run successfully, check-in all the objects from project repository.

6.1 Exercises

6-11

Define and Load the Time Dimension Table The time dimension will be created using -written code; therefore, no metadata objects for source tables are needed. 9. Defining the Time Dimension Target Table Metadata The time dimension table will be created with a -written SAS program. There are two ways to create the target table metadata: Method 1:

Use the New Table wizard and all the columns manually.

Method 2:

Run a modified version of the program that will be used to generate the table. (This generates a temporary version of the table.) When the table exists, the Tables wizard can be used to the table.

Choose one of the methods and create a metadata table object for DIFT Time Dimension. For Method 1: • Use the New Table wizard to create a metadata table object named DIFT Time Dimension. • Store the metadata object in the /Data Mart Development/Orion Target Data folder. • The following columns must be entered manually: NAME

LENGTH

TYPE

Date_ID

4

Numeric

WeekDay_Num

8

Numeric

WeekDay_Name

9

Character

Month_Num

8

Numeric

Year_ID

4

Character

Month_Name

9

Character

Quarter

6

Character

Holiday_US

26

Character

Fiscal_Year

4

Character

Fiscal_Month_Num 8

Numeric

Fiscal_Quarter

Character

6

FORMAT Date9.

• Name the physical table, a SAS table, TimeDim, and store it in the DIFT Orion Target Tables Library.

6-12


For Method 2: • In SAS Data Integration Studio, select Tools Ö Code Editor. • In the Enhanced Editor window, include the TimeDim.sas program from the S:\Workshop\dift\SASCode directory. • Submit the program and that no errors were generated in the Log window. • In SAS Data Integration Studio, invoke the Tables wizard. • Store the metadata object in the \Data Mart Development\Orion Target Data folder. • Select SAS as the source type, and select DIFT Orion Target Tables Library. • The TimeDim table should be available. • Set the name of the metadata table object to DIFT Time Dimension. • (update if necessary) that the length of Date_ID is 4. • In the Code Editor window, uncomment the PROC DATASETS step and run just that step. that the TimeDim table is deleted (check the Log). You will re-create it via a SAS Data Integration Studio job. • Close the Code Editor window. (Select File Ö Close.) Do not save any changes. 10. Loading the Time Dimension Target Table Some specifics for creating the job to load the DIFT Time Dimension table are shown below: • Name the job DIFT Populate Time Dimension Table. • Store the metadata object in the /Data Mart Development/Orion Jobs folder. • Use the Written Code transformation to specify the code to load this table. • Add the Table Loader transformation to the process flow for visual effect but specify to exclude this transformation from running.

• The code for creating the TimeDim table is found in S:\Workshop\dift\SASCode\timedimNoLibname.sas. • Open the Job Properties window and select the Precode and Postcode tab. • Enter the following LIBNAME statement in metadata: libname target "S:\Workshop\dift\datamart"; • Run the job. There should be 4384 observations and 11 variables. 11. Check-In the Metadata Objects for the Time Dimension After ing the job has run successfully, check-in all the objects from project repository.


6-13

6.2 Solutions to Exercises Define and Load the Customer Dimension Table 1. Defining the Customer Dimension Source Table Metadata a. Select File Ö Tables…. The Tables wizard opens. b. Select SAS as the type of table to import information. c. Click d. Select e. Click

. The Select a SAS Library window opens. next to the SAS Library field and then click DIFT Orion Source Tables Library. . The Define Tables and Select Folder Location window displays.

f. Select CUSTOMER table. g. that /Data Mart Development/Orion Source Data is the folder listed for the Location field. h. Click

. The review window displays.

i. that the information is correct and click

.

j. Right-click the CUSTOMER metadata table object and select Properties. k. Type DIFT at the beginning of the default name. l. Click


2. Defining the Customer Dimension Target Table Metadata a. Select the Folders tab. b. Expand Data Mart Development Ö Orion Target Data. c. that the Data folder is selected and select File Ö New Ö Table…. The New Table wizard opens. d. Type DIFT Customer Dimension as the value for the Name field. e. that the location is set to /Data Mart Development/Orion Target Data. f. Click

.

g. that the DBMS field is set to SAS. h. Select DIFT Orion Target Tables Library as the value for the Library field. i. Type CustDim as the value for the Name field. j. Click

.

6-14


k. Expand the Data Mart Development Ö Orion Source Data folder on the Folders tab. l. From the Data folder, expand DIFT Customer Types table object. m. Select the Customer_Type and Customer_Group columns from DIFT Customer Types and click to move the columns to the Selected pane. n. Select the Checkouts tab. o. Expand DIFT CUSTOMER table object. p. Select the following columns from DIFT CUSTOMER and click Selected pane: Customer_ID Country Gender Customer_Name Customer_FirstName Customer_LastName Birth_Date q. Click

.

r. Change the name of the following columns: ORIGINAL COLUMN (in Customer):

RENAME TO…

Country

Customer_Country

Gender

Customer_Gender

to move the columns to the


6-15

s. Add new column metadata. 1) Select the last column. to define an additional column.

2) Click

3) Enter the following information for the new column: Column Name

Description

Customer_Age

Customer Age

4) Click

Length 8

Type Numeric



Description

Customer_Age_Group

Customer Age Group

Length 12

Type Character

t. Define needed indexes: 1) Click

. to add the index.

2) Click

3) Enter an index name of Customer_ID and press ENTER. 4) Select the Customer_ID column and move it to the Indexes by clicking 5) Click u. Click

.

. .

v. Review the metadata listed in the finish window and then click appears on the Checkouts tab.

. The new table object

3. Loading the Customer Dimension Target Table a. Select the Folders tab. b. Expand Data Mart Development Ö Orion Jobs. c. that the Orion Jobs folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. 1) Type DIFT Populate Customer Dimension Table as the value for the Name field. 2) that the Location is set to /Data Mart Development/Orion Jobs. 3) Click


6-16


e. Add the source data objects to the process flow. 1) Click the Checkouts tab. 2) Right-click on the DIFT CUSTOMER table object and select Add to Diagram. 3) Select the Folders tab. 4) If necessary, expand Data Mart Development Ö Orion Source Data. 5) Right-click on the DIFT Customer Types table object and select Add to Diagram. f. Select File Ö Save to save diagram and job metadata to this point. g. Add the SQL transformation to the diagram. 1) In the tree view, select the Transformations tab. 2) Expand the Data grouping. 3) Select the SQL transformation. 4) Drag the SQL transformation to the diagram. 5) Center the SQL so that it is in middle of the DIFT CUSTOMER table object and DIFT Customer Types table object. h. Add inputs to the SQL transformation. 1) Click on the connection selector for DIFT CUSTOMER and drag to one of the input ports for the SQL transformation. 2) Click on the connection selector for DIFT Customer Types and drag to other input port for the SQL transformation. i. Select File Ö Save to save diagram and job metadata to this point. j. Add target table to the diagram. 1) Right-click on the temporary output table for the SQL and select Replace…. 2) Click the Checkouts tab. 3) Select DIFT Customer Dimension. 4) Click

.

k. Select File Ö Save to save diagram and job metadata to this point. l. Review the properties of the SQL transformation. 1) Right-click on the SQL transformation and select Open. The Designer window opens. 2) Select the item on the Diagram tab. 3) In the Properties pane, set the to Left.


4) Establish the criteria of the Customer_Type_ID columns from the sources being equal. a) Double-click the Left icon in the process flow diagram for SQL clauses. b) Click

.

c) Under the first Operand field, click

and then Choose column(s)….

d) Expand DIFT Customer Types and select Customer_Type_ID. e) Click

.

f) Under the Operator field, that the Operator is =. and then Choose column(s)….

g) Under the second Operator field, click

h) Expand DIFT Customer and select Customer_Type_ID. i) Click

.

5) Establish the subsetting criteria of Customer_Type_ID >= 0. a) Double-click the Where keyword on the Navigate pane to surface the Where tab. b) Click

.

c) Under the first Operand field, click

and then Choose column(s)….

d) Expand DIFT Customer Types and select Customer_Type_ID. e) Click

.

f) Under the Operator field, click

and then >=.

g) Under the second Operator field, click, type in 0, and then press ENTER. 6) mappings and establish new ones if necessary. a) Select the Select keyword on the Navigate pane to surface the Select tab. b) Manually map the Country source column to the Customer_Country column. c) Manually map the Gender source column to the Customer_Gender column. d) Click

to fold the target table over the source table.

e) Locate and open the HelperFile.txt file in S:\Workshop\dift\. 7) Copy the expression for Customer_Age. floor(yrdif(customer.birth_date, today(),'actual')) 8) Paste the copied code in the Expression area.

6-17

6-18


9) Copy the expression for Customer_Age_Group. put(calculated customer_age, agegroup.) 10) Paste the copied code in the Expression area. 11) Manually map the Birth_Date source column to the Customer_Age column. m. Click


n. Right-click in background of the job and select Run. o. that the job runs successfully. p. Select the Log tab and that DIFTTGT.CUSTDIM is created with 89,954 observation and 11 variables. q. Select the Diagram tab. r. Right-click on DIFT Customer Dimension and select Open. s. Review the data then choose File Ö Close. t. Select File Ö Close to close the job editor. 4. Check-In the Metadata Objects for the Customer Dimension a. Select the repository name on the Checkouts tab. b. Select Check Outs Ö Check In All. c. Type Adding source/target/job for Customer Dimension as the value for the Title field. d. Click


e. Click


f. Click



6-19

Define and Load the Organization Dimension Table 5. Defining the Organization Dimension Source Table Metadata a. Select File Ö Tables…. The Tables wizard opens. b. Select SAS as the type of table to import information. c. Click d. Select e. Click

. The Select a SAS Library window opens. next to the SAS Library field and then click DIFT Orion Source Tables Library. . The Define Tables and Select Folder Location window displays.

f. Select ORGANIZATION table. g. that /Data Mart Development/Orion Source Data is the folder listed for the Location field. h. Click

. The review window displays.

i. that the information is correct and click

.

j. Select the Checkouts tab. k. Right-click the ORGANIZATION metadata table object and select Properties. l. Type DIFT at the beginning of the default name. m. Click


6. Organization Dimension Target Table Metadata a. Select the Folders tab. b. Expand Data Mart Development Ö Orion Target Data. c. that the Data folder is selected and select File Ö New Ö Table…. The New Table wizard opens. d. Type DIFT Organization Dimension as the value for the Name field. e. that the location is set to /Data Mart Development/Orion Target Data. f. Click

.

g. that the DBMS field is set to SAS. h. Select DIFT Orion Target Tables Library as the value for the Library field. i. Type OrgDim as the value for the Name field. j. Click

.

6-20


k. Expand the Data Mart Development Ö Orion Source Data folder on the Folders tab. l. From the Data folder, expand DIFT STAFF table object. m. Select the following columns from DIFT STAFF and click Selected pane.

to move the columns to the

Job_Title Salary Gender Birth_Date Emp_Hire_Date Emp_Term_Date n. Select the Checkouts tab. o. Expand DIFT ORGANIZATION table object. p. Select the following columns from DIFT ORGANIZATION and click to the Selected pane. Employee_ID Org_Name Country q. Click

.

r. Change the name of the following columns: ORIGINAL COLUMN

RENAME TO…

Org_Name

Employee_Name

Country

Employee_Country

Gender

Employee_Gender

Birth_Date

Employee_Birth_Date

Emp_Hire_Date

Employee_Hire_Date

Emp_Term_Date

Employee_Term_Date

s. Add new column metadata. 1) Select the last column. 2) Click


3) Enter the following information for the new column: NAME Group

LENGTH 40

TYPE Character

to move the columns


4) Click

6-21


5) Enter the following information for the new column: NAME

LENGTH

Section

40

6) Click

TYPE Character



LENGTH

Department 40 8) Click

TYPE Character



LENGTH

Company

30

TYPE Character

t. Define needed indexes: 1) Click

. to add the index.

2) Click

3) Enter an index name of Employee_ID and press ENTER. 4) Select the Employee_ID column and move it to the Indexes by clicking 5) Click u. Click

.

. .

v. Review the metadata listed in the finish window and then click appears on the Checkouts tab.

. The new table object

6-22


7. Loading the Organization Dimension Target Table a. Select the Folders tab. 1) Expand Data Mart Development Ö Orion Jobs. 2) that the Orion Jobs folder is selected. b. Select File Ö New Ö Job…. The New Job window opens. 1) Type DIFT Populate Organization Dimension Table as the value for the Name field. 2) that the Location is set to /Data Mart Development/Orion Jobs. 3) Click

.

c. Add the source data objects to the process flow. 1) If necessary, expand Data Mart Development Ö Orion Source Data. 2) Select the DIFT STAFF table object and drag it to the Diagram tab of the Job Editor. 3) Click the Checkouts tab. 4) Select DIFT ORGANIZATION table object and drag it to the Diagram tab of the Job Editor. 5) Arrange the source data objects so that they are separated. d. Select File Ö Save to save diagram and job metadata to this point. e. Add the SQL transformation to the diagram. 1) In the tree view, select the Transformations tab. 2) Expand the Data grouping. 3) Select the SQL transformation. 4) Drag the SQL transformation to the diagram. 5) Center the SQL so that it is in middle of the two table objects. f. Select File Ö Save to save diagram and job metadata to this point. g. Add inputs to the SQL transformation. 1) Place your cursor over the SQL transformation in the diagram to reveal the two default ports. 2) To connect the DIFT STAFF table object to the SQL , place your cursor over the connection selector until a pencil icon appears. 3) Click on this connection selector and drag to one of the input ports for the SQL transformation.


6-23

4) To connect the DIFT ORGANIZATION table object to the SQL , place your cursor over the connection selector until a pencil icon appears. 5) Click on this connection selector and drag to one of the input ports for the SQL transformation. h. Select File Ö Save to save diagram and job metadata to this point. i. Add the Table Loader transformation to the diagram. 1) In the tree view, select the Transformations tab. 2) Expand the Access grouping. 3) Select the Table Loader transformation. 4) Drag the Table Loader transformation to the diagram. 5) Center the Table Loader so that it is to the right of the SQL transformation. j. Connect the SQL transformation (click on the temporary table icon, SQL and drag) to the Table Loader transformation.

, associated with the

k. Select File Ö Save to save diagram and job metadata to this point. l. Add the DIFT Organization Dimension table object to the process flow. 1) Click the Checkouts tab. 2) Drag the DIFT Organization Dimension table object to the diagram. 3) Center the table object so that it is to the right of the Table Loader transformation. m. Connect the Table Loader transformation to the DIFT Organization Dimension table object. n. Select File Ö Save to save diagram and job metadata to this point. o. Rename the temporary table object associated with the SQL transformation. 1) Right-click on the temporary table element attached to the SQL transformation and select Properties. 2) Select the Physical Storage tab. 3) Type SQL as the value for the Name field. 4) Click

to close the SQL Properties window.

p. Select File Ö Save to save diagram and job metadata to this point. q. Specify the properties of the SQL transformation. 1) Right-click on the SQL transformation and select Open. The Designer window opens. 2) Right-click the item on the Diagram tab and change the to a Left . 3) Double-click the Where keyword on the SQL Clauses pane.

6-24


4) Select the Where keyword on the Navigate pane to surface the Where tab. .

a) Click

b) Select Choose column(s)… from the drop-down list under the first Operand column. c) Expand the DIFT ORGANIZATION table and select Org_Level. d) Click

.

e) Operator is an equals sign (=). f) Type 1 for the second Operand field and press ENTER. 5) Select the Select keyword on the Navigate pane to surface the Select tab. a) Remove all target columns by right-clicking someplace over the target table side and choose Select All. b) Choose the delete tool

and then Delete Target Columns.

c) Import column definitions from the target table. d) Choose the import tool,

.

e) Select the Checkouts tab. f) Select the DIFT Organization Dimension table object. g) Click

to move columns to the Selected area.

h) Click

to close the Import Columns window

i) Map columns. (1) Right-click in the between source and target columns, and select Map All. Three of the target columns map. (2) Map the Country column to Employee_Country by clicking on the Country column and dragging to the Employee_Country. (3) Map the Gender column to Employee_Gender by clicking on the Gender column and dragging to the Employee_Gender. (4) Map the Org_Name column to Employee_Name by clicking on the Org_Name column and dragging to the Employee_Name. (5) Map the Birth_Date column to Employee_Birth_Date by clicking on the Birth_Date column and dragging to the Employee_Birth_Date. (6) Map the Emp_Hire_Date column to Employee_Hire_Date by clicking on the Emp_Hire_Date column and dragging to the Employee_Hire_Date.


6-25

(7) Map the Emp_Term_Date column to Employee_Term_Date by clicking on the Emp_Term_Date column and dragging to the Employee_Term_Date. This leaves four columns that are to be calculated. to fold the target table over the source table. This provides more room to work on j) Click the calculated expressions. (1) Select Employee_Country (or the last column before the four that need expressions) on the toolbar. and then click (2) Specify new column attributes. • Type _Group as the value for the Name. • Select Numeric as the value for Type. (3) Select _Group and then click

on the toolbar.

(4) Specify new column attributes. • Type _Section as the value for the Name. • Select Numeric as the value for Type. (5) Select _Section then click

on the toolbar.

(6) Specify new column attributes. • Type _Department as the value for the Name. • Select Numeric as the value for Type. (7) Select _Department then click

on the toolbar.

(8) Specify new column attributes. • Type _Company as the value for the Name. • Select Numeric as the value for Type. (9) Locate and open the HelperFile.txt file in S:\Workshop\dift\SASCode. (10) Locate the _Group column. In the Expression column, select Advanced… from the drop-down list. The Expression window opens as displayed: (11) Copy the expression for _Group from HelperFile.txt. Input(Put(organization.employee_id,orgdim.),12.) (12) Paste the copied code in the Expression Text area. (13) Select


(14) Locate the _Section column. In the Expression column, select Advanced… from the drop-down list. The Expression window opens as displayed.

6-26


(15) Copy the expression for _Section from HelperFile.txt. Input(Put(calculated _group,orgdim.),12.) (16) Paste the copied code in the Expression Text area. (17) Select


(18) Locate the _Department column. In the Expression column, select Advanced… from the drop-down list. The Expression window opens as displayed. (19) Copy the expression for _Department from HelperFile.txt. Input(Put(calculated _section,orgdim.),12.) (20) Paste the copied code in the Expression Text area. (21) Select


(22) Locate the _Company column. In the Expression column, select Advanced… from the drop-down list. The Expression window opens as displayed: (23) Copy the expression for _Company from HelperFile.txt. Input(Put(calculated _department,orgdim.),12.) (24) Paste the copied code in the Expression Text area. (25) Select


(26) Locate the Group column. In the Expression column, select Advanced… from the drop-down list. The Expression window opens as displayed. (27) Copy the expression for Group from HelperFile.txt. Put(calculated _group,org.) (28) Paste the copied code in the Expression Text area. (29) Select


(30) Locate the Section column. In the Expression column, select Advanced… from the drop-down list. The Expression window opens as displayed: (31) Copy the expression for Section from HelperFile.txt. Put(calculated _section,org.) (32) Paste the copied code in the Expression Text area. (33) Select


(34) Locate the Department column. In the Expression column, select Advanced… from the drop-down list. The Expression window opens as displayed.


6-27

(35) Copy the expression for Department from HelperFile.txt. Put(calculated _department,org.) (36) Paste the copied code in the Expression Text area. (37) Select


(38) Locate the Company column. In the Expression column, select Advanced… from the drop-down list. The Expression window opens as displayed. (39) Copy the expression for Company from HelperFile.txt. Put(calculated _company,org.) (40) Paste the copied code in the Expression Text area. (41) Select

k) Click


to fold the target table info back to the right side.

l) Map the Employee_ID column from the ORGANIZATION table to _Group. m) Select File Ö Save to save changes to the SQL transformation. 6) Click


r. Specify properties of the Table Loader transformation. 1) Right-click on the Table Loader transformation and select Properties. 2) Select the Mappings tab. 3) Right-click in the between the source columns and the target columns and select Map All. The mappings are updated. Note that all target columns are mapped. 4) Click

to close the Table Loader Properties window.

s. Run the job by right-clicking in background of the job and select Run. The job runs without Errors or Warnings. t. Click


u. View the Log for the executed Job by selecting the Log tab.

6-28


v. Scroll to view the note about the creation of the DIFTTGT.ORGDIM table:

8. Check-In the Metadata Objects for the Organization Dimension a. Select Check Outs Ö Check In All. b. Type Adding source/target/job for Organization Dimension as the value for the Title field. c. Click


d. Click


e. Click



6-29

Define and Load the Time Dimension Table 9. Defining the Time Dimension Target Table Metadata a. Select the Folders tab. b. Expand Data Mart Development Ö Orion Target Data. c. that the Data folder is selected and select File Ö New Ö Table…. The New Table wizard opens. d. Type DIFT Time Dimension as the value for the Name field. e. that the location is set to /Data Mart Development/Orion Target Data. .

f. Click

g. that the DBMS field is set to SAS. h. Select DIFT Orion Target Tables Library as the value for the Library field. i. Type TimeDim as the value for the Name field. j. Click

.

k. Click

.

l. Add the 11 new columns: 1) Click

to define the first column.


Description

Date_ID 3) Click

Length 4

Type Numeric

Format DATE9.


4) Enter the following information for the new column: Column Name WeekDay_Num

Description

Length 8

Type Numeric

6-30


5) Click



Description

WeekDay_Name 7) Click

Length 9

Type Character



Description

Month_Num 9) Click

Length 8

Type Numeric



Description

Year_ID 11) Click

Length 4

Type Character



Description

Month_Name 13) Click

Length 9

Type Character



Description

Quarter 15) Click

Length 6

Type Character



Description

Holiday_US 17) Click

Length 26


Type Character



Description

Fiscal_Year

Length 4

19) Click

Type Character



Description

Fiscal_Month_Num 21) Click

Length 8

Type Numeric



Description

Fiscal_Quarter m. Click

Length 6

Type Character

to define a simple index for Date_ID. to add the first index.

1) Click

2) Enter an index name of Date_ID and press ENTER. 3) Select the Date_ID column and move to the Indexes by clicking 4) Click n. Click

.

. .

a. Review the information. o. Click

.

10. Loading the Time Dimension Target Table a. Select the Folders tab. 1) Expand Data Mart Development Ö Orion Jobs. 2) that the Orion Jobs folder is selected. b. Select File Ö New Ö Job…. The New Job window opens. 1) Type DIFT Populate Time Dimension Table as the value for the Name field. 2) that the Location is set to /Data Mart Development/Orion Jobs. 3) Click

.

6-31

6-32


c. Add the target table metatdata object to the process flow. 1) Select Checkouts tab. 2) Select the DIFT Time Dimension table object and drag it to the Diagram tab of the Job Editor. d. Select File Ö Save to save diagram and job metadata to this point. e. Add the Written Code transformation to the diagram. 1) In the tree view, select the Transformations tab. 2) Expand the Data grouping. 3) Select the Written Code transformation. 4) Drag the Written Code transformation to the diagram and place it to the left of the DIFT Time Dimension table object. f. Select File Ö Save to save diagram and job metadata to this point. g. Add the Table Loader transformation to the diagram. 1) In the tree view, select the Transformations tab. 2) Expand the Access grouping. 3) Select the Table Loader transformation. 4) Drag the Table Loader transformation to the diagram. 5) Center the Table Loader so that it is to the right of the Written Code transformation. h. Connect the Written Code transformation to the Table Loader transformation. i. Select File Ö Save to save diagram and job metadata to this point. j. Add target table to the diagram. 1) Select Checkouts tab. 2) Select the DIFT Time Dimension table object. 3) Drag the DIFT Time Dimension table object to the diagram. 4) Connect the Table Loader transformation to DIFT Time Dimension table object. k. Select File Ö Save to save diagram and job metadata to this point. l. Specify properties for the Written code transformation. 1) Right-click on Written Code transformation and select Properties. 2) Select Code tab. 3) Select All written as the value for the Code generation mode field.


4) Click 5) Click

. The Open window is displayed. .

6) Navigate to S:\Workshop\dift\SASCode.

7) Select TimeDim.sas.

6-33

6-34


8) Click window.

9) Click 10) Click

. The path and filename of the code file are now listed in the Open

to close the Open window. to close the Written Properties window.

m. Specify properties of the Table Loader transformation. 1) Right-click on the Table Loader transformation and select Properties. 2) Click the Mappings tab. 3) Click

(propagate from target to sources tool) from the tool set on the Mappings tab.


6-35

4) that all columns are mapped. If not, right-click in the between the source columns and the target columns and select Map All. The mappings will be updated. 5) Click the Code tab. 6) Click Exclude transformation from run. 7) Click


n. Run the job by right-clicking in the background of the job and select Run. The job runs without Errors or Warnings.

o. View the Log for the executed Job by selecting the Log tab. p. Select File Ö Save to save diagram and job metadata to this point. q. Select File Ö Close to close the Job Editor. 11. Check-In the Metadata Objects for the Time Dimension a. Select Check Outs Ö Check In All. b. Type Adding target table & job for Time Dimension as the value for the Title field. c. Click


d. Click


e. Click


6-36


Chapter 7 Working with Transformations 7.1

Introduction..................................................................................................................... 7-3 Demonstration: Create Orion Reports Subfolders ................................................................. 7-5

7.2

Using Extract, Summary Statistics, and Loop Transformations ................................ 7-7 Demonstration: Using the Extract and Summary Statistics Transformation .......................... 7-9 Demonstration: Using the Loop Transformations ................................................................ 7-34 Exercises .............................................................................................................................. 7-54

7.3

Establishing Status Handling ...................................................................................... 7-57 Demonstration: Working with Transformation and Job Status Handling .............................. 7-61 Demonstration: Using the Return Code Check Transformation .......................................... 7-71

7.4

Using the Data Validation Transformation ................................................................. 7-77 Demonstration: Using the Data Validation Transformation .................................................. 7-79 Exercises .............................................................................................................................. 7-93

7.5

Using Transpose, Sort, Append, and Rank Transformations ................................... 7-95 Demonstration: Using the Transpose, Sort, Append, and Rank Transformations ............... 7-99

7.6

Basic Standardization with the Apply Lookup Standardization Transformation .. 7-123 Demonstration: Using the Apply Lookup Standardization Transformation......................... 7-125 Exercises ............................................................................................................................ 7-138

7.7

Solutions to Exercises ............................................................................................... 7-140

7-2

Chapter 7 Working with Transformations

7.1 Introduction

7.1 Introduction

Objectives „

List transformations that are discussed in this chapter.

3

Transformations Tree The Transformations tree organizes transformations into a set of folders. You can drag a transformation from the Transformations tree to the Job Editor, where you can connect it to source and target tables and update its default metadata. By updating a transformation with the metadata for actual sources, targets, and transformations, you can quickly create process flow diagrams for common scenarios.

4

7-3

7-4


Transformation Examples This chapter has examples that use a number of transformations available in SAS Data Integration Studio. Here is a partial listing of those transformations: Q Sort Q Rank Q Transpose Q Data Validation Q Extract Q Append Q Summary Statistics Q One-Way Frequency

5

7.1 Introduction

7-5

Create Orion Reports Subfolders

This demonstration creates a series of subfolders under the Orion Reports folder. The new folders will be used to organize the various metadata objects created and used in the subsequent sections of this chapter. 1. If necessary, access SAS Data Integration Studio using Bruno’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. that the connection profile is My Server. c. Click




2. Click the Folders tab in the tree view area. 3. Expand the Data Mart Development folder (click

in front of Data Mart Development).

4. Right-click on the Orion Reports folder and select New Ö Folder. 5. Type Extract and Summary as the name for the new folder. 6. Press ENTER. 7. Right-click on the Orion Reports folder and select New Ö Folder. 8. Type Loop Transforms as the name for the new folder. 9. Press ENTER. 10. Right-click on the Orion Reports folder and select New Ö Folder. 11. Type Data Validation as the name for the new folder. 12. Press ENTER. 13. Right-click on the Orion Reports folder and select New Ö Folder. 14. Type Transpose and Rank as the name for the new folder. 15. Press ENTER. 16. Right-click on the Orion Reports folder and select New Ö Folder. 17. Type Lookup Standardization as the name for the new folder. 18. Press ENTER.

7-6


19. Right-click on the Orion Reports folder and select New Ö Folder. 20. Type Status Handling as the name for the new folder. 21. Press ENTER. The final set of folders for Orion Reports should resemble the following:

7.2 Using Extract, Summary Statistics, and Loop Transformations


Objectives „

„

Discuss and use the Extract and Summary Statistics transformation. Discuss and use the Loop transformations.

9

Extract Transformation The Extract transformation is typically used to create a subset from a source. It can also be used to create columns in a target that are derived from columns in a source.

10

7-7

7-8


Summary Statistics Transformation The Summary Statistics transformation provides an interface to the MEANS procedure, which provides data summarization tools. By default, the MEANS procedure displays output but can also be used to store statistics in a SAS data set.

11


7-9

Using the Extract and Summary Statistics Transformation

This demonstration creates a report on customer order information for customers from the United States who placed orders in 2007. The customer dimension information first need to be ed to the order fact table and then subset, which is done in a separate job. A second job is created that will extract the desired rows, and then a summary statistics report will be created from this extracted data. 1. If necessary, access SAS Data Integration Studio using Bruno’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. that the connection profile is My Server. c. Click




Create the Job to Load the Customer Order Information Table 2. Create the initial job metadata. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Extract and Summary. c. that the Extract and Summary folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. e. Type DIFT Populate Customer Order Information Table as the value for the Name field. f.

that the Location is set to /Data Mart Development/ Orion Reports/Extract and Summary.

g. Click


3. Add source table metadata to the diagram for the process flow. a. Select the Data Mart Development Ö Orion Target Data Folders. b. Drag the DIFT Customer Dimension table object to the Diagram tab of the Job Editor. c. Drag the DIFT Order Fact table object to the Diagram tab of the Job Editor.

7-10


4. Add the SQL transformation to the process flow. a. Click the Transformation tab. b. Expand the Data folder and locate the SQL transformation template. c. Drag the SQL transformation to the Diagram tab of the Job Editor. Center it between the two table objects as follows:

d. Connect the DIFT Customer Dimension table object to one input port for the SQL transformation. e. Connect the DIFT Order Fact table object to the second input port for the SQL transformation.


7-11

5. Add the target table to the process flow. a. Right-click on the green temporary table object associated with the SQL transformation and select Table…. b. Type DIFT Customer Order Information as the value for the Name field. c. that the Location is set to /Data Mart Development/ Orion Reports/Extract and Summary.

7-12


d. Click the Physical Storage tab. e. Click f.


Expand Data Mart Development Ö Orion Target Data.

g. Click DIFT Orion Target Tables Library.

h. Click

.

The selected library is returned to the Library field.


i.

Type CustomerOrderInfo as the value for the Name field.

j.

Click

.

The process flow diagram should resemble the following:


7-13

7-14


7. properties for the SQL transformation. a. Right-click on the SQL transformation and select Open. b. that the type of is an inner by selecting the icon in the process flow, and view the Type in the Properties pane. c. Select the Where keyword in the Navigate pane. d. that the inner will be performed based on the matching (equality) Customer_ID columns from each of the sources.


e. Add an additional WHERE clause. 1) Click

.

2) Under the first Operand field, click window opens.

and then Advanced…. The Expression Builder

3) On the Functions tab, click the Date and Time folder under the Categories list. 4) On the Functions tab, under the Functions list, click YEAR(date). 5) Click

.

6) Click the Data Sources tab. 7) Expand the DIFT Order Fact table. 8) Click Order_Date. 9) Click 10) Click

. .

11) that = is set as the Operator. 12) In the second Operand field, click and type 2007 and then press ENTER.

f.

Select the Select keyword on the Navigate pane.

7-15

7-16


g. that all 22 target columns will be mapped one-to-one using a source column. h. Select

.

8. Select File Ö Save to save diagram and job metadata to this point. 9. Run the job. a. Right-click in background of the job and select Run. b. Click the Status tab in the Details area. Note that all processes completed successfully. c. Click


d. View the Log for the executed Job. Scroll to view the note about the creation of the DIFTTGT.CUSTOMERORDERINFO:

e. Click the Diagram tab of the Job Editor window. f.

Right-click on the DIFT Customer Order Information table and select Properties. 1) Click the Columns tab. 2) Update the Format field for the Customer_Country column to $COUNTRY20.. 3) Update the Format field for the Customer_Gender column to $GENDER6.. 4) Click

.

g. Right-click on the DIFT Customer Order Information table and select Open.


7-17

10. When you are finished viewing the DIFT Customer Order Information table, close the View Data window by selecting File Ö Close. 11. Save and close the Job Editor window. b. Select File Ö Save to save diagram and job metadata to this point. c. Select File Ö Close to close the Job Editor window.

Create the Job to Perform the Extract and to Create a Report. 1. Create the initial job metadata. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Extract and Summary. c. that the Extract and Summary folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. e. Type DIFT Create Report for US Customer Order Information as the value for the Name field. f.

that the Location is set to /Data Mart Development/ Orion Reports/Extract and Summary.

g. Click


2. Add source table metadata to the diagram for the process flow. a. If necessary, click the Folders tab. b. If necessary, expand Data Mart Development Ö Orion Reports Ö Extract and Summary. c. Drag the DIFT Customer Order Information table object to the Diagram tab of the Job Editor. 3. Add the Extract transformation to the process flow. a. Click the Transformation tab. b. Expand the Data folder and locate the Extract transformation template. c. Drag the Extract transformation to the Diagram tab of the Job Editor. Place the transformation next to the table object.

7-18


d. Connect the DIFT Customer Order Information table object to the Extract transformation.

4. Add the Summary Statistic transformation to the process flow. a. If necessary, click the Transformation tab. b. Expand the Analysis folder and locate the Summary Statistics transformation template. c. Drag the Summary Statistics transformation to the Diagram tab of the Job Editor. Place the transformation next to the table object.

d. Connect the Extract transformation to the Summary Statistics transformation.

5. Select File Ö Save to save diagram and job metadata to this point. 6. Specify properties for the Extract transformation. a. Right-click on the Extract transformation and select Properties. b. Click the Where tab. c. In bottom portion of the Where tab, Click the Data Sources tab. d. Expand CustomerOrderInfo table. e. Select Customer_Country. f.

Click

. The Expression Text area updates to the following:


g. In Expression Text area, type =”US”.

h. Click

to close the Extract Properties window.

7. Select File Ö Save to save diagram and job metadata to this point. 8. Specify properties for the Summary Statistics transformation. a. Right-click on the Summary Statistics transformation and select Properties. b. On the General tab, remove the default description.

7-19

7-20


c. Click the Options tab. in the Select analysis columns area to open the Select Data Source Items 1) Click window. 2) Select Total Retail Price, hold down the CTRL key and select Quantity, and then click

3) Click to close the Select Data Source Items window. The Select analysis columns area updates as displayed:

.


7-21

4) Click in the Select columns to subgroup data area to open the Select Data Source Items window. 5) Select Customer Gender, hold down the CTRL key and select Customer Age Group, and then click .

to close the Select Data Source Items window. The Select columns 6) Click to subgroup data area updates as displayed:

7-22


d. Options Basic in the selection pane. 1) Under Selected area, click Standard deviation (STD) and then 2) Under Selected area, click Number of observations (N) and then 3) Under Selected area, click Minimum (MIN) and then

.

. .


e. Click Percentiles in the selection pane. 1) Under Available area, click MEDIAN and then

.

7-23

7-24


f.

Select Other options in the selection pane. 1) Type nodate nonumber ls=80 for Specify other options for OPTIONS statement. 2) Type MAXDEC=2 NOLABELS for Other PROC MEANS options.


g. Select Titles and footnotes in the selection pane. 1) Type Customer Order Statistics for Heading 1. 2) Type (United States Customers) for Heading 2.

7-25

7-26


h. Select ODS options in the selection pane. 1) Select Use HTML for ODS result. 2) Click

in the Location area.

3) Navigate to S:\Workshop\dift\reports. 4) Type UnitedStatesCustomerInfo.html in the Name field.

5) Click i.

Click

to close the Select a File window. to close the Summary Statistics Properties window.



d. View the Log for the executed Job.


11. View the HTML document. a. Open a Windows Explorer by right-clicking on Start and selecting Explore. b. Collapse the C: drive listing. c. Expand to S:\Workshop\dift\reports.

d. Double-click UnitedStatesCustomerInfo.html to open the generated report. An information window appears as follows:

7-27

7-28


e. Click f.

Click

. to close the information bar in Internet Explorer window.

The generated report displays:

g. When done viewing the report, select File Ö Close to close Internet Explorer.


7-29

12. Select File Ö Save to save diagram and job metadata to this point. 13. Select File Ö Close to close the Job Editor window. The Extract and Summary folder displays the two jobs, and a target table:

7-30


Report for Each Country While the report on customer order information for United States is exactly what that country manager requested, other country managers are now requesting the same report but for their respective country. A separate job for each country could be created (all 42 of them). However, if a change occurred in the requested layout of the report, the management of all these jobs becomes unwieldy. A better solution would be to use an iterative process.

14

Loop and Loop End Transformation The Loop and Loop End transformations are used to mark the beginning and end, respectively, of an iterative job that is processing.

15


Steps for Using Loop Transformations You can fulfill this request in three steps: Step 1 Create the control table. Step 2 Create the parameterized job. Step 3 Create the iterative job. 16

Control Table The control table can be any table that contains rows of data that can be fed into an iteration. The creation of this table can be an independent job, or as part of the job flow containing the Loop transformations. In the previous example, we used that value of country in three places: Q The unformatted value was used to subset data in the Extract transformation. Q The formatted value was used as a title for the report. Q The compressed formatted value was used to build the name of the HTML file created. 17

7-31

7-32


Step 1: Create the Control Table This table will be populated with the distinct set of country values. The SQL transformation can be used to accomplish this task.

The code generated by the SQL will resemble the following: proc sql; create table difttgt.distinctcountries as select distinct put(customer_country,$country.) as countryname, compress(put(customer_country,$country.)) as ccountryname, put(customer_country,$2.) as countryvalue from difttgt.customerorderinfo; quit; 18

Step 2: Create the Parameterized Job The parameterized job will be used to extract data for each supplier country.

Distinct country values will be ed into this job using the control table and macro variables defined for the source table of this job.

19


Step 3: Create the Iterative Job The iterative job will use the control table as input to the Loop/Loop End transformations where the Loop/Loop End transformations execute the parameterized job in succession for each country value.

20

7-33

7-34


Using the Loop Transformations

This demonstration uses the loop transformations to iterate through the distinct customer country values and create a separate summary report for each of the countries. Three basic steps will be accomplished: • Step 1: Create the control table. • Step 2: Create the parameterized job. • Step 3: Create the iterative job.

Step 1: Create the Control Table 1. Define the table metadata object for the control table. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. c. that the Loop Transforms folder is selected. d. Select File Ö New Ö Table…. e. Type DIFT Control Table - Countries as the value for the Name field. f.

that the Location is set to /Data Mart Development/ Orion Reports/Loop Transforms.

g. Click

.

h. that the DBMS field is set to SAS. i.

Select DIFT Orion Target Tables Library as the value for the Library field.

j.

Type DistinctCountries as the value for the Name field.

k. Click l.

.

No column metadata will be selected from existing metadata objects. Click

.


m. Define three new columns. .

1) Click

2) Type CountryName as the Name of the new column. 3) Type Country Name as the Description of the new column. 4) Type 20 as the Length of the new column. 5) that the Type is set to Character. .

6) Click

7) Type CCountryName as the Name of the new column. 8) Type Compressed Country Name as the Description of the new column. 9) Type 20 as the Length of the new column. 10) that the Type is set to Character. .

11) Click

12) Type CountryValue as the Name of the new column. 13) Type 2-Character Country Value as the Description of the new column. 14) Type 2 as the Length of the new column. 15) that the Type is set to Character.

n. Click

.

o. Review the metadata listed in the finish window and click

.

7-35

7-36


2. Define the job metadata object to load the control table. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. c. that the Loop Transforms folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. e. Type DIFT Populate Control Table of Countries as the value for the Name field. f.


g. Click


3. Add the source table to the process flow. a. If necessary, click the Folders tab. b. If necessary, expand Data Mart Development Ö Orion Reports Ö Extract and Summary. c. Drag the DIFT Customer Order Information table object to the Diagram tab of the Job Editor. 4. Add the SQL transformation to the process flow. a. Click the Transformation tab. b. Expand the Data folder and locate the SQL transformation template. c. Drag the SQL transformation to the Diagram tab of the Job Editor. Place the transformation next to the table object.

d. Connect the DIFT Customer Order Information table object to the SQL transformation.

By default, the SQL expects at least two input tables. However, for this instance, we need just one input.


7-37

e. Click the status indicator on the SQL transformation to discover a source table is missing.

f.

Click

to close the message popup.

g. Right-click on the SQL transformation and select Ports Ö Delete Input Port. The status indicator now shows no errors.

5. Add the target table to the process flow. a. Right-click on the green temporary table object associated with the SQL transformation and select Replace…. b. Click the Checkouts tab. c. Select DIFT Distinct Set of Countries. d. Click

. The process flow updates to the following:

Again, the status indicator for the SQL shows that there is a problem. e. Click the status indicator on the SQL transformation to discover that mappings are needed.

f.

Click

to close the message popup.


7-38


7. Specify the properties for the SQL node. a. Right-click on the SQL transformation and select Open. b. Click the Select keyword in the Navigate pane. c. In the Select Properties pane, change the Distinct option from No to Yes.

d. On the Select tab, specify the following Expression information for the three target columns. Column Name

Expression

CountryName

put(customer_country,$country.)

CCountryName

compress(put(customer_country,$country.))

CountryValue

put(customer_country,$2.)


7-39

e. Map the Customer_Country column to each of the three calculated columns.

f.

to return to the Job Editor. Note that the status indicator associated with the SQL Click transformation now shows no errors.

8. Select File Ö Save to save diagram and job metadata to this point. 9. Run the job to generate the control table. a. Right-click in background of the job and select Run. b. that the job runs successfully. c. Click the Log tab and that DIFTTGT.DISTINCTCOUNTRIES is created with 45 observation and 3 variables.

7-40


d. Click the Diagram tab. e. Right-click DIFT Control Table - Countries and select Open.

f.

Review the data and then choose File Ö Close.

g. Select File Ö Close to close the job editor.


7-41

Step 2: Create the Parameterized Job The parameterized job will mimic the job that uses the Extract and Summary Statistics transformations. 1. Copy the job that extracts data and creates a summary report. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Extract and Summary. c. Right-click DIFT Create Report for US Customer Order Information and select Copy. d. Right-click the Loop Transforms folder and select Paste. e. Right-click DIFT Create Report for US Customer Order Information (the copied job located in the Loop Transforms folder) and select Properties. f.

Type DIFT Parameterized Job for Country Reports as the value for the Name field.

g. that the Location is set to /Data Mart Development/ Orion Reports/Loop Transforms. h. Click


2. Double-click the job DIFT Parameterized Job for Country Reports and it opens in the Job Editor window. 3. Edit the Extract transformation. a. Right-click on the Extract transformation and select Properties. b. Click the Where tab. c. Type &CtryValue in place of US (in the Expression Text area) – be sure double quotes are being used.

d. Click


7-42


4. Edit the Summary Statistics transformation. a. Right-click on the Summary Statistics transformation and select Properties. b. Click the Options tab. c. Click Titles and footnotes in the selection pane. d. In Heading 2 area, type (&Country Customers).

e. Click ODS options in the selection pane. f.

In Location area, update the filename to &CCountry.CustomerInfo.html.

g. Click

Be sure to type in the period that separates the parameter name from the rest of the text. to close the Summary Statistics Properties window.



6. Define job parameters. a. Right-click in the background of the job and select Properties. b. Click the Parameters tab. c. Click

.

1) Type Country as the value for the Name field. 2) Type Country Name as the value for the Displayed text field.

7-43

7-44


3) Click the Prompt Type and Values tab. 4) Type United States as the value for the Default value field.

5) Click d. Click

to close the New Prompt window. .

1) Type CCountry as the value for the Name field. 2) Type Compressed Country Name as the value for the Displayed text field. 3) Select Prompt Type and Values tab. 4) Type UnitedStates as the value for the Default value field. 5) Click

to close the New Prompt window.


e. Click

.

1) Type CtryValue as the value for the Name field. 2) Type Country Value as the value for the Displayed text field. 3) Select Prompt Type and Values tab. 4) Type US as the value for the Default value field. 5) Click


The final set of three parameters is listed on the Parameters tab.

7-45

7-46


f.

Click

to validate the prompts and default values.

g. Click

to close the Test the Prompts window.

h. Click

to close the DIFT Parameterized Job for Country Reports Properties window.

7. Select File Ö Save to save diagram and job metadata to this point. The icon for the job object in the Loop Transforms folder is now decorated with an ampersand to denote that the job is parameterized.


8. Run the job.

Parameterized jobs can be tested *if* all parameters are supplied default values.

a. Right-click in the background of the job and select Run. b. Click the Status tab in the Details area. Note that all processes completed successfully. c. Click


d. View the Log for the executed Job. e. Scroll towards the top of the Log and note that the parameters are all defined default values.

f.

Scroll towards end of the Summary Statistics code area and that the correct HTML file name is being generated, as well as the correct title2 text.

g. Additionally, that the data were subsetted correctly.

9. Select File Ö Close to close the Job Editor window.

7-47

7-48


10. View the HTML document. a. Open a Windows Explorer by right-clicking on Start and select Explore. b. If necessary, collapse the C: drive listing. c. Expand to S:\Workshop\dift\reports. d. that UnitedStatesCustomerInfo.html exists (you can also check the date-time stamp to that this HTML file was created with this job).


7-49

Step 3: Create the Iterative Job 1. Create the initial job metadata. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. c. that the Loop Transforms folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. e. Type DIFT Loop Job for Country Reports as the value for the Name field. f.


g. Click


2. Add control table metadata to the diagram for the process flow. a. Click the Folders tab. b. If necessary, expand Data Mart Development Ö Orion Reports Ö Loop Transforms. c. Drag the DIFT Control Table - Countries table object to the Diagram tab of the Job Editor. 3. Add the Loop transformation to the process flow. a. Click the Transformations tab. b. Expand the Control folder and locate the Loop transformation template. c. Drag the Loop transformation to the Diagram tab of the Job Editor. d. Connect the DIFT Control Table - Countries table object as input to the Loop transformation.

4. Add the parameterized job to the process flow. a. Click the Folders tab. b. If necessary, expand Data Mart Development Ö Orion Reports Ö Loop Transforms. c. Drag the DIFT Parameterized Job for Country Reports job to the Diagram tab of the Job Editor.

7-50


5. Add the Loop End transformation to the process flow. a. Click the Transformations tab. b. If necessary, expand the Control folder and locate the Loop End transformation template. c. Drag the Loop End transformation to the Diagram tab of the Job Editor. The process flow diagram should resemble the following:


7-51

6. Edit the properties of the Loop transformation. a. Right-click on the Loop transformation in the process flow diagram and select Properties. b. Click the Parameter Mapping tab.

c. For the Country Name parameter, select CountryName as the value for the Mapped Source Column. d. For the Compressed Country Name parameter, select CCountryName as the value for the Mapped Source Column. e. For the Country Value parameter, select CountryValue as the value for the Mapped Source Column.

f.

Click

to close the Loop Properties window.

7-52




d. View the Log for the executed Job. As the “inner” job (the parameterized job) executes, an observation is pulled from the control table and its values assigned to the parameters defined for the job. Scrolling in the Log reveals the various assignments. A few are listed here:

… and so on. For each of these parameter sets, the “inner” job is executed and this execution results in an HTML file.


9. that the html files were created. a. Access the Window Explorer. b. Navigate to S:\Workshop\dift\reports.

c. View any of the reports. d. Select File Ö Close to close any report viewed. e. Select File Ö Close to close the Windows Explorer. 10. Select File Ö Close to close the DIFT Loop Job for Country Reports. The final set of metadata objects in the Loop Transforms folder should resemble the following:

7-53

7-54


Exercises The Marketing Department has been asked to examine buying habits of various age groups across the genders. The same kind of marketing analysis will be applied to each distinct gender/age group combination. To make this task easier, a request has been made to create a separate SAS table for each of the distinct gender/age group combinations. You first use the Extract transformation to create one of the needed tables. This job can then be parameterized and used with the Loop transformations to create the series of desired tables. 1. Using the Extract Transformation to Create Table for Female Customers Aged 15-30 Years • Create a job that uses the Customer Dimension table to load a new table to contain just the female customers aged 15-30 years. • Place the job in the Data Mart Development Ö Orion Reports Ö Extract and Summary folder. • Name the job DIFT Populate Female15To30Years Table. • Use the Customer Dimension table as the source table for the job (the metadata for this table can be found in Data Mart Development Ö Orion Target Data). • Add the Extract transformation to the job and build the following WHERE clause: • Customer_Gender = “F” and Customer_Age_Group = “15-30 years” • the output table from the Extract transformation with the following attributes: Tab General

Physical Storage

Attribute

Value

Name:

DIFT Customers – Females 15-30 Years

Location:

/Data Mart Development/Orion Reports/ Extract and Summary

Library:

DIFT Orion Target Tables Library

Name:

Female15To30Years

• Run the job and that the new table has 12,465 observations and 11 variables. The final job flow should resemble the following:


7-55

2. Using the Loop Transformations to Create Gender-Age Group Customer Tables • Create metadata for the control table. – Place the table in the Data Mart Development Ö Orion Reports Ö Loop Transforms folder. – Name the table object DIFT Control Table - Gender Age Groups. – Name the physical table DistinctGenderAgeGroups, and have the table created in the DIFT Orion Target Tables Library. – The control table needs to have three columns with the following attributes: Name

•

Length

Type

To be used for:

GenVal

1

Character

WHERE clause processing

AgeGroup

12

Character

WHERE clause processing

GdrAgeGrp

20

Character

Name of new (subsetted) table

Create a separate job to create the control table. – Place the job in the Data Mart Development Ö Orion Reports Ö Loop Transforms folder. – Name the job DIFT Populate Control Table of Gender Age Groups. – Use the SQL transformation to populate the control table. The control table needs to contain the distinct combinations of all gender and age group values from the DIFT Customer Dimension. – Use the following table to help define the calculations for the columns: Column

Expression

GenVal

put(customer_gender,$1.)

AgeGroup

GdrAgeGrp compress(put(customer_gender,$gender.)|| tranwrd(customer_age_group,"-","To"))

•

– Run the job – the control table should have 8 observations and 3 columns. the data look appropriate. Create a table template to be used in the parameterized job. – Place the table in the Data Mart Development Ö Orion Reports Ö Loop Transforms folder. – Name the table object DIFT Table Template for Gender Age Group Table. – Name the physical table &GdrAgeGrp._Customers and have the table created in the DIFT Orion Target Tables Library as a SAS table. – The table template needs to have the same column specifications as the DIFT Customer Dimension table.

7-56


•

Create the parameterized job. – Use the Extract transformation to subset data from the DIFT Customer Dimension table using a compound WHERE clause (subset for gender and age group). – The target table object for this parameterized job will be the table template created in this exercise. – Create three parameters for the job: one for the gender value (used with WHERE processing), one for the age group value (also used with WHERE processing), and one for GdrAgeGrp (used to define the name of the table). Be sure to set up valid default values for the parameters (so this job can be tested outside of the Loop job). – Execute the job. the creation of the new table. • Create the iterative job. – Use the control table, the Loop transformation, the parameterized job, and the Loop End transformation to construct this iterative job. – Be sure to map the control table columns to the parameters from the parameterized job. – Run the job and check the Log to that eight tables are created. The 8 tables should contain the following: Table Name

Number of Observations

Number of Columns

Female15To30Years_Customers

12465 obs

11 cols


9263 obs

11 cols


9295 obs

11 cols


9266 obs

11 cols

Male15To30Years_Customers

15261 obs

11 cols


11434 obs

11 cols


11502 obs

11 cols


11468 obs

11 cols

7.3 Establishing Status Handling


Objectives „

„

Discuss return codes and how to “capture” a return code in a SAS Data Integration Studio job. Investigate where status handling is available.

25

Return Codes When a job is executed in SAS Data Integration Studio, a return code for each transformation in the job is captured in a macro variable. The return code for the job is set according to the least successful transformation in the job.

26

7-57

7-58


Example Actions The return code can be associated with an action that performs one or more of these tasks: „ terminate the job or transformation „ call a -defined SAS macro „ send a status message to a person, a file, or an event broker „ capture job statistics

27

Conditions and Actions SAS Data Integration Studio enables you to associate a return code condition, such as successful, with an action, such as Send Email or Send Job Status. In this way, you can specify how a return code is handled for the job or transformation. Example Conditions

Example Actions

Successful

Send Email

Errors

Send Entry to Text File

Warnings

Send Entry to a Data Set Send Job Status Save Exception Report

28

Example: If job is SUCCESSFUL, then SEND EMAIL.


Status Handling Tab A Status Handling tab is included in the Properties windows for jobs and for some transformations. s can select from a list of code conditions and actions on the Status Handling tab. However, the Properties windows for most transformations do not have a Status Handling tab. To return the status of a transformation that does not have a Status Handling tab, you can use a Return Code Check transformation to insert status-handling logic at a desired point in the process flow diagram for a job.

29

Status Handling Enabled If Status Handling has been defined for a transformation or the job, a macro definition (and call to it) will be added to the generated code to check for the specified condition(s). For example, if Status Handling has been defined for: „ the SQL transformation, then a definition and call for %etls_sqlrccheck will be generated in the code „ the Table Loader transformation, then a definition and call for %etls_loadercheck will be generated in the code „ the job, then a definition and call for %etls_jobrccheck will be generated in the code

30

7-59

7-60


Status Handling Code Conditions If Status Handling has been defined for a transformation or the job, the different conditions specified will generate different code segments for the status handling macros. For example, when checking for a condition of: SUCCESSFUL, the code will contain: %if (&trans_rc eq 0) %then %do; ERRORS, the code will contain: %if (&trans_rc ge 5) %then %do; WARNINGS, the code will contain: %if (&trans_rc eq 4) %then %do; 31

Return Code Check Transformation The Return Code Check transformation provides statushandling logic at a desired point in the process flow diagram for a job. It can be inserted between existing transformations and removed later without affecting the mappings in the original process flow.

32


7-61

Working with Transformation and Job Status Handling

This demonstration illustrates establishing Status Handling for an SQL transformation and for a job. Also illustrated is the use of the Return Code Check transformation. 1. If necessary, access SAS Data Integration Studio using Bruno’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. that the connection profile is My Server. c. Click




2. Locate and copy the Populate Customer Dimension Table job. a. Click the Folders tab. b. Navigate to Data Mart Development Ö Orion Jobs folder. c. Right-click DIFT Populate Customer Dimension Table and select Copy. d. Navigate to Data Mart Development Ö Orion Reports Ö Status Handling folder. e. Right-click Status Handling folder and select Paste. 3. Right-click DIFT Populate Customer Dimension Table and select Open. 4. the job runs successfully by right-clicking in the background and selecting Run. There should be 11 variables and 89954 observations.

5. Establish two “successful” status handling conditions for the SQL transformation. a. Click the Diagram tab. b. Right-click on the SQL transformation and select Properties. c. Click the Status Handling tab.

7-62


d. Click to add a new condition. By default, a Successful Condition is added with an Action of None.

There are two other conditions that can be tested for: Warnings and Errors. e. Click f.

in the Action area, next to None.

Select Send Entry to File.

The Action Options window is displayed.


7-63

The conditions of Warnings and Errors produce the same list of actions. The Errors condition has one additional option associated it – Abort. g. Specify S:\Workshop\dift\reports\SHforSQL.txt for the File Name field. h. Specify Successful running of SQL in Job &jobid for the Message field.

to close the Action Options window.

i.

Click

j.

Click to add a new condition. A second Successful Condition is added with an Action of None.

k. Click l.


Select Send Entry to a Dataset. The Action Options window opens.

m. Specify difttgt for the Libref field. n. Specify SHforSQL for the Dataset field. o. Specify Successful run for SQL in Job &jobid as the Message field.

p. Click


7-64


The two successful conditions appear on the Status Handling tab.

q. Click

to close the SQL Properties window.

6. Select File Ö Save to save diagram and job metadata to this point. 7. Re-run the job. a. Right-click in background of the job and select Run. b. Click the Status tab in the Details area. Note that all processes completed successfully. c. Click


d. View the Log for the executed Job.


The notes pertaining to the text file for a successful condition should resemble:

7-65

7-66


The notes pertaining to the data set for a successful condition should resemble the following:

8. Run the job two more times. a. Right-click in background of the job and select Run. b. Right-click in background of the job and select Run. 9. View the information in the text file. a. Open Windows Explorer by selecting Start Ö All Programs Ö Accessories Ö Windows Explorer. b. Navigate to S:\Workshop\dift\reports.


c. Double-click SHforSQL.txt.

The information in the text file is overwritten on each subsequent run. d. Select File Ö Exit to close the Notepad window. 10. View the SAS data set created if the job was successful. a. In the Windows Explorer, navigate to S:\Workshop\dift\datamart. b. Double-click SHforSQL.sas7bdat. The SAS data set opens in SAS Enterprise Guide.

The SAS data set received a new observation each time the job was run. c. Select File Ö Exit to close SAS Enterprise Guide and do not save any changes.

7-67

7-68


11. Establish two “successful” status handling conditions for the job. a. Right-click in the background of the job and select Properties. b. On the General tab, change the name to DIFT Pop Cust Dim Table (SH).

c. Click the Status Handling tab. d. Click e. Click f.

to add a new condition. in the Condition area, next to Successful, and select Send Job Status.

Select Send Job Status in the Action area. The Action Options window opens.

g. Specify difttgt for the Libref field. h. Specify SHforJob for the Dataset field.

i.

Click



7-69

The new condition appears on the Status Handling tab.

j.

Click

to close the DIFT Pop Cust Dim Table (SH) Properties window.

12. Select File Ö Save to save diagram and job metadata to this point. 13. Re-run the job. a. Right-click in background of the job and select Run. b. Click the Status tab in the Details area. Note that all processes completed successfully. c. Click


d. View the Log for the executed Job. The new job status data set is created with one observation.

14. Run the job two more times. a. Right-click in the background of the job and select Run. b. Right-click in the background of the job and select Run. 15. Click the Log tab and review the results. The job status data set now has three observations.

7-70


16. View the SAS data set. a. In the Windows Explorer, navigate to S:\Workshop\dift\datamart. b. Double-click SHforJob.sas7bdat. The SAS data set opens in SAS Enterprise Guide.

… The SAS data set received a new observation each time the job was run. The SAS data set can be used to gather some “total time” processing statistics for this job. c. Select File Ö Exit to close SAS Enterprise Guide and do not save any changes. 17. Close the DIFT Pop Cust Dim Table (SH) job and save any changes.


7-71

Using the Return Code Check Transformation

This demonstration shows the use of the Return Code Check transformation for transformations such as the Extract transformation that do not have a Status Handling tab. 1. Locate and open the Report on US Customer Order Information job. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Extract and Summary. c. Right-click DIFT Create Report for US Customer Order Information and select Copy. d. Expand Data Mart Development Ö Orion Reports Ö Status Handling. e. Right-click the Status Handling folder and select Paste. f.

Right-click the pasted job, DIFT Create Report on US Customer Order Information and select Properties.

g. On the General tab, change the name of the job to DIFT Create Report on US Cust Info (SH).

h. Click i.


Right-click DIFT Create Report for US Cust Order Info (SH) and select Open.

7-72


2. Right-click on the Extract transformation and select Properties. Note that the properties for this transformation do not have a Status Handling tab.

The Return Code Check transformation can be used to take advantage of the status handling features for those transformations that have no Status Handling tab. The Return Code Check transformation captures the status of the previous transformation in the process flow, in this case, the Extract transformation. 3. Click

to close the SAS Extract Properties window.

4. Add the Return Code Check transformation to the process flow diagram. a. Click the Transformations tab. b. Expand the Control group. c. Locate the Return Code Check transformation. d.

Drag the Return Code Check transformation in to the process flow diagram.


e.

Click the Control Flow tab in the Details area.

f.

Select the Return Code Check transformation and then click The processing order is rearranged.

.

7-73

7-74


5. Update the properties of the Return Code Check transformation. a. Right-click on the Return Code Check transformation and select Properties. b. Click the Status Handling tab. to add a new condition. By default, a Successful Condition is added c. Click with an Action of None. d. Click


e. Select Send Entry to File. The Action Options window opens. f.

Specify S:\Workshop\dift\reports\rccheck.txt for the File Name field.

g. Specify Successful running of Extract transformation in Job &jobid! for the Message field.

h. Click to close the Action Options window. The Return Code Check Properties window shows this one condition.

i.

Click

to close the Return Code Check transformation properties window.


6. Select File Ö Save to save changes to the job. 7. Right-click in the background of the job and select Run. 8. Click the Log tab and review the results. The notes pertaining to the text file for a successful condition should resemble the following:

7-75

7-76


9. View the information in the text file. a. Open Windows Explorer by selecting Start Ö All Programs Ö Accessories Ö Windows Explorer. b. Navigate to S:\Workshop\dift\reports. c. Double-click rccheck.txt.

d. Select File Ö Exit to close the Notepad window. 10. Save and close this job. a. Select the job editor window. b. Click the Diagram tab. c. Select File Ö Save. d. Select File Ö Close.

7.4 Using the Data Validation Transformation


Objectives „

Discuss and use the Data Validation transformation.

36

Data Validation Transformation The Data Validation transformation is used to improve the quality of operational data before that data is loaded into a data warehouse or data mart. With this transformation, error conditions and actions can be specified.

37

7-77

7-78


Using the Error Table

38

The error table contains the same columns as the source table, with the addition of a date/time column that identifies when the job was run. ¾ If the error table does not exist, it is created. ¾ The date/time column, named ETL_Error_JobRunTime, enables the same error table to be used each time that the job is run. ¾ New rows, with a new date/time value, are appended to existing rows. ¾ The libref and filename of the error table are specified on the Options tab of the Data Validation transformation. ¾ The error table differs from the exception report. The exception report is generated and distributed as specified on the Status Handling tab.


Using the Data Validation Transformation

The following tasks are completed in this demonstration: • Define the necessary target tables, one for valid items and one for error items. • Create a job that applies the Data Validation transformation to the source table to create the target table(s). 1. If necessary, access SAS Data Integration Studio using Bruno’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. that the connection profile is My Server. c. Click




Part 1: Defining the Metadata for the Target Tables 2. Specify metadata for a valid products table. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Data Validation. c. that the Data Validation folder is selected. d. Select File Ö New Ö Table…. e. Type DIFT Valid Products as the value for the Name field. f.

that the Location is set to /Data Mart Development/ Orion Reports/Data Validation.

g. Click

.



j.

Type Valid_Products as the value for the Name field.

k. Click l.

.

Expand the Data Mart Development Ö Orion Source Data folder on the Folders tab.

7-79

7-80


m. From the Orion Source Data folder, select DIFT NEWORDERTRANS table object. n. Click o. Click

to move all columns to the Selected pane. .

p. Accept the default column attributes and click

.

q. Review the metadata listed in the finish window and click

.

3. Specify metadata for an invalid products table. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Data Validation. c. that the Data Validation folder is selected. d. Select File Ö New Ö Table…. e. Type DIFT Invalid Products as the value for the Name field. f.


g. Click

.



j.

Type Invalid_Products as the value for the Name field.

k. Click l.

.

Expand the Data Mart Development Ö Orion Source Data folder on the Folders tab.

m. From the Orion Source Data folder, select DIFT NEWORDERTRANS table object. n. Click o. Click


p. Accept the default column attributes and click

.

q. Review the metadata listed in the finish window and click The metadata table objects appear in the Data Validation folder:

.


7-81

Part 2: Applying the Data Validation Transformation 4. Create the initial job metadata. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Data Validation. c. that the Data Validation folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. e. Type DIFT Populate Valid and Invalid Product Tables as the value for the Name field. f.


g. Click


5. Add source table metadata to the diagram for the process flow. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Source Data. c. Drag the DIFT NEWORDERTRANS table to the Diagram tab of the Job Editor. 6. Add the Data Validation transformation to the process flow. a. Click the Transformation tab. b. Expand the Data folder and locate the Data Validation transformation template. c. Drag the Data Validation transformation to the Diagram tab of the Job Editor. d. Connect the DIFT NEWORDERTRANS table object to the Data Validation transformation. e. Right-click DIFT NEWORDERTRANS and select Properties. f.

On the General tab, remove the description.

g. Click

.

7-82


7. Add a target table to the process flow. a. Right-click on the green temporary table object associated with the Data Validation transformation and select Replace…. b. Expand Data Mart Development Ö Orion Reports Ö Data Validation. c. Select DIFT Valid Products. d. Click

.

8. Specify properties for the Data Validation transformation. a. Right-click on the Data Validation transformation and select Properties. b. Click the Options tab. c. Locate the Enter an error table name (libref.tablename) option. d. Type difttgt.invalid_products as the value for this option.


e. Click the Status Handling tab. 1) Click

to add a new condition.

2) Data Exception is the Condition. 3) Select Save Report as the Action. 4) In the Action Options window, type S:\Workshop\dift\reports\ValidInvalidProdExceptionReport.txt as the value for File Name.

5) Click


6) Click


7) Data Exception is the Condition. 8) Select Send Entry to File as the Action. 9) In the Action Options window, type S:\Workshop\dift\reports\ValidInvalidProdEntryToFile.txt as the value for File Name.

10) Click


7-83

7-84


11) Click


12) Data Exception is the Condition. 13) Select Send Entry to a Dataset as the Action. 14) In the Action Options window, type difttgt as the value for the Libref field. 15) Type ProdDataExcept as the value for the Dataset Name.

16) Click


The Status Handling tab shows the following:


f.

Click the Invalid Values tab.

g. Click

to add a new row.

h. Select Product_Name as the value for Column Name. i.

Click

j.

Expand the DIFT VALIDPRODUSAOUTDOOR table in the Data Mart Development Ö Orion Source Data folder and select Product_Name.

k. Click

l.

Click

next to Lookup Table.

to move Product_Name to the Selected list.

to close the Lookup Table and Column window.

7-85

7-86


The following is populated in the Invalid Values window:

m. Keep the default Action if invalid value, Move row to error table. n. Click

to close the Invalid Values window. The Invalid Values tab shows the following:


o. Click the Missing Values tab. p. Click

to add a new row.

q. Select Product_ID as the value for Column Name. r. that Move row t o error table is set as the value for Action if missing.

s. Click following:

to close the Missing Values window. The Missing Values tab shows the

t. Click

to close the Data Validation Properties window.

7-87

7-88


9. Add a sticky note to the job. (A sticky note is a way to visual document within a job.) a. Click

on the Job Editor toolbar.

The sticky note is added to the upper-left corner of the diagram.

b. Drag the sticky note and place it under the Data Validation transformation. c. Double-click the sticky note to expand to add some text. d. Type The Invalid_Products table is populated through the execution of the Data Validation transformation. as the text for the sticky note.



7-89

11. Run the job. a. Right-click in background of the job and select Run. b. Click the Status tab in the Details area. Note that all processes completed successfully. c. Click


d. View the Log for the executed Job. e. Scroll to view the note about the creation of the DIFTTGT.VALID_PRODUCTS table and the creation of the DIFTTGT.INVALID_PRODUCTS table:

7-90


12. View the DIFT Valid Products table. a. Click the Diagram tab of the Job Editor window. b. Right-click the DIFT Valid Products table object and select Open.

c. When you are finished viewing the DIFT Valid Products data set, close the View Data window by selecting File Ö Close. 13. View the DIFT Invalid Products table. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Data Validation. c. Right-click the DIFT Invalid Products table object and select Open.

d. When you are finished viewing the Invalid Products data set, close the View Data window by selecting File Ö Close.


14. View the exception report. a. Open Windows Explorer and navigate to S:\Workshop\dift\reports. b. Open the file ValidInvalidProdExceptionReport.txt to view the generated exception report.

c. When you are finished viewing the file, select File Ö Exit to close it. 15. View ValidInvalidProdEntryToFile.txt. a. Open Windows Explorer and navigate to S:\Workshop\dift\reports. b. Open the file ValidInvalidProdEntryToFile.txt to view the generated exception report.

c. When you are finished viewing the file, select File Ö Exit to close it. 16. View the data set difttgt.proddataexcept. a. Copy the LIBNAME statement for the DIFT Orion Target Tables Library. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Target Data. 3) Right-click on the DIFT Orion Target Tables Library object and select View Libname. 4) Right-click in the background of the Display Libname window and select Select All.

7-91

7-92


5) Right-click in the background of the Display Libname window and select Copy. 6) Click

to close the Display Libname window.

b. Select Tools Ö Code Editor. 1) Right-click in the background of the Code Editor window and select Paste. 2) Enter the following PROC PRINT steps: options ls=72 nodate nonumber; proc print data=difttgt.DataExcept; run; 3) Click

to run the code.

4) Click the Output tab to view the information in the data set.

5) Select File Ö Close to close the Code Editor. 6) Click

to not save changes.

17. Select File Ö Save to save the job metadata. 18. Select File Ö Close to close the job editor.


7-93

Exercises

3. Using the Data Validation Transformation Use the Data Validation transformation to load the Valid Customers table from the DIFT Customer Dimension table. The process flow diagram should be as follows:

•

•

•

Create metadata for the target table named DIFT Valid Customers. – The target table should be physically stored in the DIFT Orion Target Tables Library with a name of Valid_Customers. – The table object should contain the exact same columns as the DIFT Customer Dimension table object found in the Data Mart Development Ö Orion Target Data folder. – The target table object should end up in the Data Mart Development Ö Orion Reports Ö Data Validation folder Create metadata for the target table named DIFT Invalid Customers. – The target table should be physically stored in the DIFT Orion Target Tables Library with a name of Invalid_Customers. – The table object should contain the exact same columns as the DIFT Customer Dimension table object found in the Data Mart Development Ö Orion Target Data folder. – The target table object should end up in the Data Mart Development Ö Orion Reports Ö Data Validation folder Create the job that will load Valid Customers from the DIFT Customer Dimension table using the Data Validation transformation.

7-94


•

For the Data Validation transformation properties: – Specify difttgt.Invalid_Customers as the error table. – that the Customer_Type values are all valid by comparing to the DIFT Customer Types table’s Customer_Type column (the DIFT Customer Types table object can be found in the Orion Source Tables folder). If an invalid value is found for Customer_Type, change it to “Unknown Customer Type” and write the information to an error table. – If missing values are found for the column Customer_ID then abort the job. – If duplicate values are found for the columns Customer_Name and Customer_Birth_Date then move the rows to an error table. – For status handling, establish an exception report (S:\Workshop\dift\reports\ValidInvalidCustExceptReport.txt).

How many rows were moved to the error table because the value for Customer_Type was invalid? Were missing values found for Customer ID? Were there any duplicate values found for Customer_Name and Customer_Birth_Date?

7.5 Using Transpose, Sort, Append, and Rank Transformations


Objectives „

Discuss and use the Rank, Transpose, Append, List, and Sort transformations.

43

Transpose Transformation The Transpose transformation creates an output data set by restructuring the values in a SAS data set, and transposing selected variables in to observations. The Transpose transformation is an interface to the TRANSPOSE procedure.

44

7-95

7-96


Sort Transformation The Sort transformation provides an interface for the SORT procedure. The transformation can be used to read data from a source, sort it, and write the sorted data to a target in a SAS Data Integration Studio job.

45

Append Transformation The Append transformation can be used to create a single target by appending or concatenating two or more sources.

46


Rank Transformation The Rank transformation uses the RANK procedure to rank one or more numeric variables in the source and store the ranks in the target.

47

List Data Transformation The List Data transformation is used to create a report on specified columns from a source table. The report can be an HTML report.

48

7-97

7-98


Completed Process Flow The demonstration will take advantage of each of the Rank, Transpose, Append, List, and Sort transformations.

49


7-99

Using the Transpose, Sort, Append, and Rank Transformations A staff table (SAS data set) was defined with only partial staff information (DIFT STAFF_PARTIAL). Additional staff was entered in an external file, but the structure of the external file does not match that of the partial staff table. The following job reads the partial staff information from the SAS data set and reads additional partial staff information from the external file. The external file result, read through a File Reader transformation, then needs to be sorted and transposed so that it can be appended to the SAS data set of partial staff information. The final data is sorted and ranked, with a report listing the top ten salaries for staff. The final process flow should resemble the following:





7-100


2. that the DIFT STAFF_PARTIAL metadata table object exists and has data loaded. a. Click the Folders tab. b. Navigate to the Data Mart Development Ö Orion Source Data folder. c. Right-click DIFT STAFF_PARTIAL and select Open.

d. Select File Ö Close to close the View Data window. 3. Define metadata for the external file containing additional staff. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Transpose and Rank. c. that the Transpose and Rank folder is selected. d. Select File Ö New Ö External File Ö Delimited…. The New Delimited External File wizard opens: 1) Type DIFT Additional Staff Information as the value for the Name field. 2) that the Location is set to /Data Mart Development/ Orion Reports/Transpose and Rank. e. Click

. The External File Location window is displayed.


f.

Click

7-101


1) Navigate to S:\Workshop\dift\data. 2) Change the Type field to Delimited Files (*.csv). 3) Select AddlStaff.csv. 4) Click

.

g. To view the contents of the external file, click

.

Previewing the file shows the first record contains column names and that the values are commadelimited and not space delimited. h. Click i.

to close the Preview File window.

Click

. The Delimiters and Parameters window is displayed.

1) Clear Blank. 2) Click Comma. j.

Click 1) Click

. .The Auto Fill Columns window opens.

2) Type 2 (the number two) as the value for the Start record field. 3) Click to close the Auto Fill Columns window. The top portion of the Column Definitions window populates with three columns, one of them numeric and two of them character.

7-102


4) Click

. The Import Column Definitions window opens.

5) Select Get the column names from column headings in this file. 6) that 1 is set as the value for The column headings are in the file record field. 7) Click

. The Name field populates with all the column names.

k. Change the length, informat, and format for the Job_Title column. 1) In the top portion of the Column Definitions window, locate the Job_Title column. 2) Update the length to 25. 3) Update the informat to $25.. 4) Update the format to $25.. l.

Accept the defaults for the rest of the column settings.

m. Click

.

n. Click

. The metadata object for the external file is found on the Checkouts tab.

4. Create a target table object that is a duplicate of metadata from DIFT STAFF_PARTIAL. a. Click the Folders tab. b. Navigate to Data Mart Development Ö Orion Source Data. c. Right-click DIFT STAFF_PARTIAL and select Copy to Folder. d. Navigate to Data Mart Development Ö Orion Reports Ö Transpose and Rank. e. Click

f.

Click

. A Warning window is displayed.

. The copied object is now in the Transpose and Rank folder.


5. Update metadata for the copied DIFT STAFF_PARTIAL object. a. Right-click the Copy of DIFT STAFF_PARTIAL table object and select Properties. b. On the General tab, change the name of the metadata object to DIFT Full Staff.

c. Click the Physical Storage tab. d. Click

next to the Library field. The Select a library window is displayed.

1) Collapse the Orion Source Data folder. 2) Expand the Orion Target Data folder. 3) Click DIFT Orion Target Tables Library. 4) Click

.

e. Type STAFF_FULL as the value for the Name field.

f.

Click


7-103

7-104


6. Create initial job metadata. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Transpose and Rank. c. that the Transpose and Rank folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. e. Type DIFT Populate Full Staff & Create Rank Report as the value for the Name field. f.

that the Location is set to /Data Mart Development/ Orion Reports/Transpose and Rank.

g. Click


7. Add source table metadata to the diagram for the process flow. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Source Data. c. Drag the DIFT STAFF_PARTIAL table object to the Diagram tab of the Job Editor. d. Collapse Orion Source Data. e. Expand Data Mart Development Ö Orion Reports Ö Transpose and Rank. f.

Drag the DIFT Additional Staff Information external file object to the Diagram tab of the Job Editor.


8. Add the File Reader transformation to the process flow. a. Click the Transformation tab. b. Expand the Access folder and locate the File Reader transformation template. c. Drag the File Reader transformation to the Diagram tab of the Job Editor. 9. Connect the DIFT Additional Staff Information external file object to the File Reader transformation. 10. Add the Sort transformation to the process flow. a. Click the Transformation tab. b. Collapse the Access folder. c. Expand the Data folder and locate the Sort transformation template. d. Drag the Sort transformation to the Diagram tab of the Job Editor. 11. Connect the File Reader transformation to the Sort transformation. The process flow diagram at this point should resemble the following:

12. Add the Transpose transformation to the process flow. a. Click the Transformation tab. b. Expand the Data folder and locate the Transpose transformation template. c. Drag the Transpose transformation to the Diagram tab of the Job Editor. 13. Connect the Sort transformation to the Transpose transformation. 14. Add the Append transformation to the process flow. a. Click the Transformation tab. b. Expand the Data folder and locate the Append transformation template. c. Drag the Append transformation to the Diagram tab of the Job Editor. 15. Connect the Transpose transformation to one of the ports for the Append transformation. 16. Connect DIFT STAFF_PARTIAL table object to the other default port for the Append transformation.

7-105

7-106



17. Add target table to the diagram. a. Right-click on the temporary output table for the Append and select Replace…. The Table Selector window is displayed. b. Expand Data Mart Development Ö Orion Reports Ö Transpose and Rank. c. Click DIFT Full Staff. d. Click

.

18. Select File Ö Save to save diagram and job metadata to this point. 19. Add the Rank transformation to the process flow. a. Click the Transformation tab. b. Expand the Data folder and locate the Rank transformation template. c. Drag the Rank transformation to the Diagram tab of the Job Editor. 20. Connect the DIFT Full Staff table object to the Rank transformation. 21. Select File Ö Save to save diagram and job metadata to this point. The process flow diagram should resemble the following:


7-107

22. the properties for the File Reader transformation. a. Right-click on the File Reader transformation and select Properties. b. Click the Mappings tab. c. that all columns are mapped properly.

d. Click

.

23. Change the name of work table that is output for the File Reader transformation. a. Right-click on the green temporary table object associated with the File Reader transformation and select Properties. b. Click the Physical Storage tab. c. Type FileReader as the value for the Name field. d. Click

.

7-108


24. Specify properties for the first Sort transformation. a. Right-click on the Sort transformation and select Properties. b. Click the Sort By Columns tab. c. Select Job_Title under Available columns area. d. Click

e. Click

to move the Job_Title column to the Sort by columns area.

.

25. Change the name of work table that is output for the first Sort transformation. a. Right-click on the green temporary table object associated with the Sort transformation and select Properties. b. Click the Physical Storage tab. c. Type SortedExtFile as the value for the Name field. d. Click

.


7-109

26. Change the name of work table that is output for the Transpose transformation. a. Right-click on the green temporary table object associated with the Transpose transformation and select Properties. b. On the General tab, remove the description. c. Click the Physical Storage tab. d. Type TransposedData as the value for the Name field. e. Click

.

27. Specify properties for the Transpose transformation. a. Right-click on the Transpose transformation and select Properties. b. On the General tab, remove the description. c. Click the Mappings tab. 1) Remove all target table columns. a) Right-click in the background of the Target table area and select Select All. b) Click

and then select Delete Target Columns.

7-110


2) Import correct target columns. a) Right-click in the background of the Target table area and select Import Columns…. b) Expand Data Mart Development Ö Orion Reports Ö Transpose and Rank. c) Select DIFT Full Staff table object. d) Select

e) Click

to move all columns from the DIFT Full Staff table to the Selected area.

.


3) Map Job_Title from Source table area to Job_Title in the Target table area.

d. Click the Options tab. 1) that Assign columns is selected in the selection pane.

7-111

7-112


2) Establish ColValues for the Select columns to transpose area. in the Select columns to transpose area to open the Select Data a) Click Source Items window. b) Select ColValues and then click

.

to close the Select Data Source Items window. The Select c) Click analysis columns area updates as displayed:


7-113

3) Establish ColNames for the Select a column for output column names area. a) Click in the Select a column for output column names area to open the Select a Data Source Item window. b) Select ColNames.

to close the Select a Data Source Item window. The Select a c) Click column for output column names area updates as displayed:

7-114


4) Establish Job_Title for the Select columns whose values define groups of records to transpose area. in the Select columns whose values define groups of a) Click records to transpose area to open the Select Data Source Items window. b) Select Job_Title and then click

.

c) Click to close the Select Data Source Items window. The Select columns whose values define groups of records to transpose area updates as displayed:

e. Click the Precode and Postcode tab. 1) Click Postcode. 2) Type the following SAS code: data work.transposed (drop=gdrnum); set &syslast(rename=(gender=gdrnum)); if gdrnum=0 then gender='M'; else if gdrnum=1 then gender='F'; run; f.

Click

to close the Transpose Properties window.


28. Specify properties for the Append transformation. a. Right-click on the Append transformation and select Properties. b. Click the Mappings tab. c. Click

d. Click

to map all source columns to target columns.

.

7-115

7-116


29. Change the name of work table that is output for the Rank transformation. a. Right-click on the green temporary table object associated with the Rank transformation and select Properties. b. Click the Physical Storage tab. c. Type RankedData as the value for the Name field. d. Click

.

30. Specify properties for the Rank transformation. a. Right-click on the Rank transformation and select Properties. b. Click the Mappings tab. 1) On the target table side, right-click and select New Column. 2) Type RankedSalary as the name of the new column. 3) Select Numeric as the Type of the new column. 4) Select the following columns (click on one, then hold down the CTRL key and click on each subsequent column): Start_Date End_Date Birth_Date Emp_Hire_Date Emp_Term_Date Manager_ID 5) Click

and then select Delete Target Columns.

6) Re-order the columns so that Salary and RankedSalary are the last two columns (click on the row number and drag a column to desired ordering).

7) that all columns are mapped properly (the RankedSalary column will not have a oneto-one column mapping). 8) Right-click on the RankedSalary column and select Propagate From Targets To End.


7-117

c. Click the Rank Variable Columns tab. 1) Select Salary in the Available source columns area, and then click Salary to the Selected source columns area.

to move

2) Select RankSalary in the Available target columns area, and then click move RankSalary to the Selected target columns area.

d. Click

to

to close the Rank Properties window.

31. Right-click in the background of the job and select Settings Ö Automatically Propagate Columns (effectively disabling the automatic propagation for this job, from this point of edit on). 32. Add the Sort transformation to the process flow. a. Click the Transformation tab. b. Expand the Data folder and locate the Sort transformation template. c. Drag the Sort transformation to the Diagram tab of the Job Editor.

7-118


33. Connect the Rank transformation to the Sort transformation. 34. Add the List Data transformation to the process flow. a. Click the Transformation tab. b. Expand the Output folder and locate the List Data transformation template. c. Drag the List Data transformation to the Diagram tab of the Job Editor. 35. Connect the Sort transformation to the List Data transformation. 36. Select File Ö Save to save diagram and job metadata to this point. 37. Change the name of work table that is output for the second Sort transformation. a. Right-click on the green temporary table object associated with the second Sort transformation and select Properties. b. Click the Physical Storage tab. c. Type SortedRanks as the value for the Name field. d. Click

.

38. Right-click on the Rank transformation and select Propagate Columns Ö From Selected Transformation’s Targets Ö To End.


39. Specify properties for the second Sort transformation. a. Right-click on the second Sort transformation and select Properties. b. Click the Sort By Columns tab. c. Select RankedSalary under Available columns area. d. Click

to move the RankSalary column to the Sort by columns area.

e. Change the Sort order to Descending.

f.

Click

.

7-119

7-120


40. Specify properties for the List Data transformation. a. Right-click on the List Data transformation and select Properties. b. On the General tab, remove the description. c. Click the Options tab. 1) Select Report formatting in the selection pane. 2) Select Use column labels as column headings (LABEL) and then click

.

3) Select Other options in the selection pane. 4) Type nodate nonumber ls=80; as the value for the System options area. 5) Type NOOBS as the value for the Additional PROC PRINT options area. 6) Type format salary dollar12.; as the value for the Additional PROC PRINT statements area.


7-121

7) Select the Columns to report on item in the selection pane. 8) Establish Job_Title, Gender, and Salary for the Select other columns to print area. in the Select other columns to print area to open the Select Data a) Click Source Items window. b) Select Job_Title, hold down the CTRL key and select Gender, hold down the CTRL key and select Salary, and then click .

to close the Select Data Source Items window. The Select other c) Click columns to print area updates as displayed:

9) Select the Titles and footnotes item in the selection pane. 10) Type Top Ten Paying Positions at Orion Star as the value for Heading 1. d. Click the Precode and Postcode tab. 1) Click Precode. 2) Type options obs=10;. 3) Click Postcode. 4) Type options obs=MAX;.

7-122


e. Click

to close the List Data Properties window.



d. View the Log for the executed Job. e. Click the Output tab to view the report.

43. Select File Ö Close to close the job editor.

7.6 Basic Standardization with the Apply Lookup Standardization Transformation


Objectives „

„

Discuss and use the Apply Lookup Standardization transformation. Discuss and use the One-Way Frequency transformation.

53

One-Way Frequency Transformation The One-Way Frequency and Frequency transformations can be used to generate frequency statistics. These transformations are based on the FREQ procedure.

54

7-123

7-124


Apply Lookup Standardization Transformation The Apply Lookup Standardization transformation is used to apply one or more schemes to one or more columns in a source table. Applying schemes modifies the source data according to rules that are defined in the schemes.

55


7-125

Using the Apply Lookup Standardization Transformation

A table of potential customers was defined in metadata (DIFT s). This demonstration creates a job that initially reports on this data by creating a one-way frequency reports for two of the columns. The job is then updated by adding the Apply Lookup Standardization transformation to apply two predefined standardization schemes. The final step is reporting on the newly transformed data. The final process flow should resemble the following:





7-126


2. that the DIFT s metadata table object exists and has data available. a. Click the Folders tab. b. Navigate to the Data Mart Development Ö Orion Source Data folder. c. Right-click on DIFT s and select Open.

d. Select File Ö Close to close the View Data window.


7-127

3. Create initial job metadata. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Lookup Standardization. c. that the Lookup Standardization folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. e. Type DIFT Standardize and Report on s Table as the value for the Name field. f.

that the Location is set to /Data Mart Development/ Orion Reports/Lookup Standardization.

g. Click


4. Add source table metadata to the diagram for the process flow. a. Click the Folders tab. b. Navigate to the Data Mart Development Ö Orion Source Data folder. c. Drag the DIFT s table object to the Diagram tab of the Job Editor. 5. Add the One-Way Frequency transformation to the process flow. a. Click the Transformation tab. b. Expand the Analysis folder and locate the One-Way Frequency transformation template. c. Drag the One-Way Frequency transformation to the Diagram tab of the Job Editor. 6. Connect the DIFT s table object to the One-Way Frequency transformation. 7. Select File Ö Save to save diagram and job metadata to this point.

7-128


8. Specify properties for the One-Way Frequency transformation. a. Right-click on the One-Way Frequency transformation and select Properties. b. Click the Options tab. 1) that Assign columns is selected in the selection pane. 2) Establish OS and Database in the Select columns to transpose area. in the Select columns to perform a one-way frequency a) Click distribution on area to open the Select Data Source Items window. b) Select OS, hold down the CTRL key and select DATABASE, and then click

.

c) Click to close the Select Data Source Items window. The Select columns to perform a one-way frequency distribution on area updates as displayed:

3) Select Specify other options in the selection pane.


7-129

4) Type nocum nopercent as the value for Specify other options for TABLES statement area. 5) Select Titles and footnotes in the selection pane. 6) Type OS and Database Frequency Counts as the value for the Heading 1. c. Click

to close the One-Way Frequency Properties window.



d. View the Log for the executed Job. e. Click the Output tab to view the report. Partial Output – OS frequency report:

7-130


Partial Output – DATABASE frequency report:

A quick glance verifies the initial suspicion that the OS and Database columns have not had any standards imposed for data values. 11. Select File Ö Close to close the job editor.


Two schemes have been pre-built for these types of column data. The next steps will • establish the necessary options to access these schemes • add the Apply Lookup Standardization • re-run the One-Way Frequency task against the standardized table. 1. Select Tools Ö Options. 2. Select Data Quality tab. 3. that the following fields are set appropriately in the Data Quality area: Default Locale: ENUSA DQ Setup Location: C:\Program Files\SAS\SASFoundation\9.2\dquality\sasmisc\dqsetup.txt Scheme Repository Type: Scheme Repository:

4. Click

dfPower scheme (BFD)

C:\Program Files\DataFlux\QltyKB\CI\2008A\scheme


7-131

7-132


5. Re-open the job editor. a. Click the Folders tab. b. Expand Data Mart Development Ö Orion Reports Ö Lookup Standardization. c. Double-click on the job DIFT Standardize and Report on s Table. The job editor will be opened. 6. Break the connection between the DIFT s table object and the One-Way Frequency transformation. a. Select the connection line between the table and the transformation.

b. Right-click on the selected connection and select Delete.

7. Add the Apply Lookup Standardization transformation to the process flow. a. Click the Transformation tab. b. Expand the Data Quality folder and locate the Apply Lookup Standardization transformation template. c. Drag the Apply Lookup Standardization transformation to the Diagram tab of the Job Editor. 8. Connect the DIFT s table object to the Apply Lookup Standardization transformation. 9. Connect the Apply Lookup Standardization transformation to the One-Way Frequency transformation.


10. If necessary, re-order the processing for the transformations. a. Click the Control Flow tab in the Details area.

b. Select the Apply Lookup Standardization transformation and then click

c. On the Diagram tab, click

.

to auto-arrange the elements in the process flow.



7-133

7-134


12. Specify properties for the Apply Lookup Standardization transformation. a. Right-click on the Apply Lookup Standardization transformation and select Properties. b. Click the Standardizations tab. c. For the DATABASE column, select DIFT Database Scheme.sch.qkb as the value for the Scheme field. d. For the OS column, select DIFT OS Scheme.sch.qkb as the value for the Scheme field. e. For the DATABASE column, select Phrase as the value for the Apply Mode field. f.

For the OS column, select Phrase as the value for the Apply Mode field.

g. Click

to close the Apply Lookup Standardization Properties window.



7-135

14. Specify properties for the One-Way Frequency transformation. a. Right-click on the One-Way Frequency transformation and select Properties. b. Click the Options tab. 1) that Assign columns is selected in the selection pane. 2) Establish OS and Database in the Select columns to transpose area. in the Select columns to perform a one-way frequency a) Click distribution on area to open the Select Data Source Items window. b) Select OS, hold down the CTRL key and select DATABASE, and then click

.

c) Click to close the Select Data Source Items window. The Select columns to perform a one-way frequency distribution on area updates as displayed:

7-136


3) Select Specify other options in the selection pane. 4) Type nocum nopercent as the value for Specify other options for TABLES statement area. 5) Select Titles and footnotes in the selection pane. 6) Type OS and Database Frequency Counts as the value for the Heading 1. c. Click




d. View the Log for the executed Job. e. Click the Output tab to view the report. Output – OS frequency report:


Output – DATABASE frequency report:

7-137

7-138


Exercises

4. Standardizing Catalog Orders A table from a prospective subsidiary was defined in metadata (DIFT Catalog_Orders). A column (CATALOG) in this file has information on the types of products purchased. • Create a job to create a one-way frequency report based on the column CATALOG from the DIFT Catalog_Orders table. Partial Output:

• that the data are not “clean”. • Use the Apply Lookup Standardization transformation to apply a pre-defined scheme to the CATALOG column. The scheme is named DIFT Catalog Orders.


• Create a one-way frequency report based on the standardized column CATALOG.

The final process flow should resemble the following:

7-139

7-140


7.7 Solutions to Exercises 1. Using the Extract Transformation to Create Table for Female Customers Aged 15-30 Years a. Create the initial job metadata. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Extract and Summary. 3) that the Extract and Summary folder is selected. 4) Select File Ö New Ö Job…. The New Job window opens. 5) Type DIFT Populate Female15To30Years Table as the value for the Name field. 6) that the Location is set to /Data Mart Development/ Orion Reports/Extract and Summary. 7) Click


b. Add source table metadata to the diagram for the process flow. 1) Select the Data Mart Development Ö Orion Target Data folder. 2) Drag the DIFT Customer Dimension table object to the Diagram tab of the Job Editor. c. Add the Extract transformation to the process flow. 1) Click the Transformation tab. 2) Expand the Data folder and locate the Extract transformation template. 3) Drag the Extract transformation to the Diagram tab of the Job Editor. Place it next to the DIFT Customer Dimension table object. 4) Connect the DIFT Customer Dimension table object to the Extract transformation. d. Add the target table to the process flow. 1) Right-click on the green temporary table object associated with the Extract transformation and select Table…. 2) Type DIFT Customers - Females 15-30 Years as the value for the Name field. 3) that the Location is set to /Data Mart Development/ Orion Reports/Extract and Summary. 4) Click the Physical Storage tab. 5) Click


6) Expand Data Mart Development Ö Orion Target Data. 7) Click DIFT Orion Target Tables Library.


8) Click


9) Type Female15To30Years as the value for the Name field. 10) Click

to close the Table window.


e. Select File Ö Save to save diagram and job metadata to this point. f. Specify properties for the Extract transformation. 1) Right-click on the Extract transformation and select Properties. 2) Click the Where tab. 3) Construct the following expression:

4) Click the Mappings tab. 5) that all columns are mapped. 6) Click


g. Run the job. 1) Right-click in background of the job and select Run. 2) the job completed successfully. The target table should have 12,465 observations and 11 columns.

7-141

7-142


2. Using the Loop Transformations to Create Gender-Age Group Customer Tables a. Define the table metadata object for the control table. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. 3) that the Loop Transforms folder is selected. 4) Select File Ö New Ö Table…. The New Job window opens. 5) Type DIFT Control Table – Gender Age Groups as the value for the Name field. 6) that the Location is set to /Data Mart Development/ Orion Reports/ Loop Transforms. 7) Click

.

8) that the DBMS field is set to SAS. 9) Select DIFT Orion Target Tables Library as the value for the Library field. 10) Type DistinctGenderAgeGroups as the value for the Name field. 11) Click

.

12) No column metadata will be selected from existing metadata objects. Click

.

13) Define three new columns. a) Click

.

b) Type GenVal as the Name of the new column. c) Type Single-character value for Gender as the Description of the new column. d) Type 1 as the Length of the new column. e) that the Type is set to Character. f) Click

.

g) Type AgeGroup as the Name of the new column. h) Type Age Group values as the Description of the new column. i)

Type 12 as the Length of the new column.

j)

that the Type is set to Character.


k) Click l)

7-143

.

Type GdrAgeGrp as the Name of the new column.

m) Type Concatenated Gender/AgeGroup values as the Description of the new column. n) Type 18 as the Length of the new column. o) that the Type is set to Character.

14) Click

.

15) Review the metadata listed in the finish window and click

.

b. Define the job metadata object to load the control table. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. 3) that the Loop Transforms folder is selected. 4) Select File Ö New Ö Job…. The New Job window opens. 5) Type DIFT Populate Control Table of Gender-Age Groups as the value for the Name field.

7-144


6) that the Location is set to /Data Mart Development/ Orion Reports/ Loop Transforms. 7) Click


c. Add source table metadata to the diagram for the process flow. 1) Expand Data Mart Development Ö Orion Target Data. 2) Drag the DIFT Customer Dimension table object to the Diagram tab of the Job Editor. d. Add the SQL transformation to the process flow. 1) Click the Transformation tab. 2) Expand the Data folder and locate the SQL transformation template. 3) Drag the SQL transformation to the Diagram tab of the Job Editor. Place it next to the DIFT Customer Dimension table object. 4) Connect the DIFT Customer Dimension table object to the SQL transformation. 5) Right-click on the SQL transformation and select Ports Ö Delete Input Port. The status indicator now shows no errors. e. Add the target table to the process flow. 1) Right-click on the green icon (output table icon) for the SQL transformation and select Replace. 2) Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. 3) Click the DIFT Control Table – Gender Age Groups table object. 4) Click

.

f. Specify the properties for the SQL node. 1) Right-click on the SQL transformation and select Open. 2) Click the Select keyword in the Navigate pane. 3) In the Select Properties pane, change the Distinct option from No to Yes. 4) On the Select tab, specify the following Expression information for the three target columns: GenVal

put(customer_gender,$1.)

AgeGroup

GdrAgeGrp

compress(put(customer_gender,$gender.)|| tranwrd(customer_age_group,"-","To"))


7-145

5) Map the Customer_Gender column to both GenVal and GdrAgeGrp. 6) Map the Customer_Age_Group column to both AgeGroup and GdrAgeGrp. 7) Click to return to the Job Editor. Note that the status indicator associated with the SQL transformation now shows no errors.

g. Select File Ö Save to save diagram and job metadata to this point. h. Run the job to generate the control table. 1) Right-click in background of the job and select Run. 2) that the job runs successfully. 3) Click the Log tab and that DIFTTGT.DISTINCTCOUNTRIES is created with 45 observation and three variables.

4) Click the Diagram tab. 5) Right-click on DIFT Control Table – Gender Age Groups and select Open.

6) Review the data and then choose File Ö Close. 7) Select File Ö Close to close the job editor.

7-146


i. Create a table object template that will be used to generate the individual gender-age group tables. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. 3) that the Loop Transforms folder is selected. 4) Select File Ö New Ö Table…. 5) Type DIFT Table Template for Gender Age Group Table as the value for the Name field. 6) that the Location is set to /Data Mart Development/ Orion Reports/ Loop Transforms. 7) Click

.

8) that the DBMS field is set to SAS. 9) Select DIFT Orion Target Tables Library as the value for the Library field. 10) Type &GdrAgeGrp._Customers as the value for the Name field. 11) Click

.

12) Click the Folders tab. 13) Expand Data Mart Development Ö Orion Target Data. 14) Select DIFT Customer Dimension and click 15) Click

.

16) Click

.

.


.

j. Define the parameterized job metadata object to load the holding table. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. 3) that the Loop Transforms folder is selected. 4) Select File Ö New Ö Job…. The New Job window opens. 5) Type DIFT Parameterized Job for Gender Age Group Tablesas the value for the Name field. 6) that the Location is set to /Data Mart Development/ Orion Reports/ Loop Transforms. 7) Click



7-147

8) Add source table metadata to the diagram for the process flow. a) Expand Data Mart Development Ö Orion Target Data. b) Drag the DIFT Customer Dimension table object to the Diagram tab of the Job Editor. 9) Add the Extract transformation to the process flow. a) Click the Transformation tab. b) Expand the Data folder and locate the Extract transformation template. c) Drag the Extract transformation to the Diagram tab of the Job Editor. Place it next to the DIFT Customer Dimension table object. d) Connect the DIFT Customer Dimension table object to the Extract transformation. 10) Add the target table to the process flow. a) Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. b) Drag DIFT Table Template for Gender Age Group Tables table object to the Diagram tab of the Job Editor. c) Right-click on the output table object (green icon) for the Extract transformation and select Delete. d) Connect the Extract transformation to the DIFT Table Template for Gender Age Group Tables table object. 11) Select File Ö Save to save diagram and job metadata to this point. 12) Specify properties for the Extract transformation. a) Right-click on the Extract transformation and select Properties. b) Select Where tab. c) In bottom portion of the Where tab, click the Data Sources tab. d) Expand CustDim table. e) Select Customer_Gender. f) Click

.

g) In Expression Text area, type ="&genval" AND (that is, an equals sign, the text “&genval”, a space, the text AND, and another space). h) On the Data Sources tab, double-click Customer_Age_Group to add this to the Expression Text area. i) In the Expression Text area, type ="&AgeGroup" following the Customer_Age_Group column.

7-148


The Expression Text area should be as shown below:

j) Click


13) Select File Ö Save to save diagram and job metadata to this point. 14) Define job parameters. a) Right-click in the background of the job and select Properties. b) Click the Parameters tab. c) Click

.

d) Type GenVal as the value for the Name field. e) Type Gender Value as the value for the Displayed text field. f) Click the Prompt Type and Values tab. g) Type F as the value for the Default value field. h) Click i) Click


j) Type AgeGroup as the value for the Name field. k) Type Age Group Value as the value for the Displayed text field. l) Click the Prompt Type and Values tab. m) Type 15–30 years as the value for the Default value field. n) Click o) Click


p) Type GdrAgeGrp as the value for the Name field. q) Type Gender – Age Group Value as the value for the Displayed text field. r) Click the Prompt Type and Values tab. s) Type Female15To30years as the value for the Default value field. t) Click



u) Click

7-149

to validate the prompts and default values.

v) Click

to close the Test the Prompts window.

w) Click


15) Select File Ö Save to save diagram and job metadata to this point. 16) Run the job. a) Right-click in background of the job and select Run. b) Click the Status tab in the Details area. Note that all processes completed successfully. c) Click


d) View the Log for the executed Job. Specifically, locate the note about the FEMALE15TO30YEARS_CUSTOMERS table.

e) Select File Ö Close to close the Job Editor window.

7-150


k. Create the job metadata for the iterative job. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. 3) that the Loop Transforms folder is selected. 4) Select File Ö New Ö Job…. The New Job window opens. 5) Type DIFT Loop Job for Gender Age Group Tables as the value for the Name field. 6) that the Location is set to /Data Mart Development/ Orion Reports/ Loop Transforms. 7) Click


l. Add control table metadata to the diagram for the process flow. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. 3) Drag the DIFT Control Table – Gender Age Groups table object to the Diagram tab of the Job Editor. m. Add the Loop transformation to the process flow. 1) Click the Transformations tab. 2) Expand the Control folder and locate the Loop transformation template. 3) Drag the Loop transformation to the Diagram tab of the Job Editor. 4) Connect the DIFT Control Table - Gender-Age Groups table object as input to the Loop transformation. n. Add the parameterized job to the process flow. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Loop Transforms. 3) Drag the DIFT Parameterized Job for Gender Age Group Tables job to the Diagram tab of the Job Editor.


7-151

o. Add the Loop End transformation to the process flow. 1) Click the Transformations tab. 2) Expand the Control folder and locate the Loop End transformation template. 3) Drag the Loop End transformation to the Diagram tab of the Job Editor. The process flow diagram should resemble the following:

p. Edit the properties of the Loop transformation. 1) Right-click on the Loop transformation in the process flow diagram and select Properties. 2) Click the Parameter Mapping tab. 3) For the Gender Value parameter, select GenVal as the value for the Mapped Source Column. 4) For the Age Group Value parameter, select AgeGroup as the value for the Mapped Source Column. 5) For the Gender – Age Group Value parameter, select GdrAgeGrp as the value for the Mapped Source Column.

6) Click

to close the Loop Properties window.

7-152


q. Select File Ö Save to save diagram and job metadata to this point. r. Run the job. 1) Right-click in background of the job and select Run. 2) Click the Status tab in the Details area. Note that all processes completed successfully. 3) Click


4) View the Log for the executed Job. s. that the tables were created. 1) Access the Window Explorer. 2) Navigate to S:\Workshop\dift\datamart. 3) that the data sets were created.

4) Select File Ö Close to close the Windows Explorer.


7-153

3. Using the Data Validation Transformation a. Specify metadata for a valid products table. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Data Validation. 3) that the Data Validation folder is selected. 4) Select File Ö New Ö Table…. 5) Type DIFT Valid Customers as the value for the Name field. 6) that the Location is set to /Data Mart Development/ Orion Reports/Data Validation. 7) Click

.

8) that the DBMS field is set to SAS. 9) Select DIFT Orion Target Tables Library as the value for the Library field. 10) Type Valid_Customers as the value for the Name field. 11) Click

.

12) Expand the Data Mart Development Ö Orion Source Data folder on the Folders tab. 13) From the Orion Source Data folder, select DIFT Customer Dimension table object. 14) Click


15) Click

16) Accept the default column attributes and click

.


.

b. Specify metadata for an invalid products table. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Data Validation. 3) that the Data Validation folder is selected. 4) Select File Ö New Ö Table…. 5) Type DIFT Invalid Customers as the value for the Name field. 6) that the Location is set to /Data Mart Development/ Orion Reports/Data Validation. 7) Click

.

7-154


8) that the DBMS field is set to SAS. 9) Select DIFT Orion Target Tables Library as the value for the Library field. 10) Type Invalid_Customers as the value for the Name field. 11) Click

.

12) Expand the Data Mart Development Ö Orion Source Data folder on the Folders tab. 13) From the Orion Source Data folder, select DIFT Customer Dimension table object. 14) Click 15) Click


16) Accept the default column attributes and click

.


.

c. Create the initial job metadata. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Data Validation. 3) that the Data Validation folder is selected. 4) Select File Ö New Ö Job…. The New Job window opens. 5) Type Populate Valid and Invalid Customer Tables as the value for the Name field. 6) that the Location is set to /Data Mart Development/ Orion Reports/Data Validation. 7) Click


d. Add source table metadata to the diagram for the process flow. 1) Click the Data Mart Development Ö Orion Target Data Folders. 2) Drag the DIFT Customer Dimension table to the Diagram tab of the Job Editor. e. Add the Data Validation transformation to the process flow. 1) Click the Transformation tab. 2) Expand the Data folder and locate the Data Validation transformation template. 3) Drag the Data Validation transformation to the Diagram tab of the Job Editor. 4) Connect the DIFT Customer Dimension table object to the Data Validation transformation.


7-155

f. Add a target table to the process flow. 1) Right-click on the green temporary table object associated with the Data Validation transformation and select Replace…. 2) Expand Data Mart Development Ö Orion Reports Ö Data Validation. 3) Select DIFT Valid Customers. 4) Click

.

g. Specify properties for the Data Validation transformation. 1) Right-click on the Data Validation transformation and select Properties. 2) Click the Options tab. a) Locate the Enter an error table name (libref.tablename) option. b) Type difttgt.invalid_customers as the value for this option. 3) Click the Status Handling tab. a) Click


b) that Data Exception is the Condition. c) Select Save Report as the Action. d) In the Action Options window, type S:\Workshop\dift\reports\ValidInvalidCustExceptReport.txt as the value for File Name. e) Click


4) Click the Invalid Values tab. a) Click

to add a new row.

b) Select Customer_Type as the value for Column Name. c) Click

next to Lookup Table.

d) Expand the DIFT Customer Types table in the Data Mart Development Ö Data folder and select Customer_Type. e) Click

to close the Lookup Table and Column window.

f) Select Change value to as the value for the Action if invalid field.

7-156


g) Type “Unknown Customer Type” as the value for the New Value field. The following is populated in the Invalid Values window:

h) Click

to close the Invalid Values window.

5) Click the Missing Values tab. a) Click

to add a new row.

b) Select Customer_ID as the value for Column Name. c) that Abort job is set as the value for Action if missing. d) Click

to close the Missing Values window.


6) Click the Duplicate Values tab. a) Click

to add a new row.

b) Select both Customer_Name and Birth_Date.

c) Click 7) Click

to close the Missing Values window. to close the Data Validation Properties window.

7-157

7-158


h. Add a sticky note to the job. (A sticky note is a way to visually document within a job.) 1) Click

on the Job Editor toolbar.

2) Drag the sticky note and place it under the Data Validation transformation. 3) Double-click the sticky note to expand to add some text. 4) Type The Invalid_Customers table is populated through the execution of the Data Validation transformation. as the text for the sticky note.

i. Select File Ö Save to save the job metadata. j. Run the job. 1) Right-click in background of the job and select Run. 2) Click the Status tab in the Details area. Note that all processes completed successfully. 3) Click


4) View the Log for the executed Job. 5) Scroll to view the note about the creation of the DIFTTGT.INVALID_CUSTOMERS table and the creation of the DIFTTGT.VALID_CUSTOMERS table.


7-159

k. View the exception report. 1) Open Windows Explorer and navigate to S:\Workshop\dift\reports. 2) Open the file ValidInvalidCustExceptReport.txt to view the generated exception report. The exception report shows that all error exceptions were due to duplicate values.

3) When you are finished viewing the file, select File Ö Exit to close it. l. View the DIFT Valid Customers table. 1) Click the Diagram tab of the Job Editor window. 2) Right-click on the DIFT Valid Customers table and select Open. Identify the duplicate observations. 3) When you are finished viewing the DIFT Valid Customers data set, close the View Data window by selecting File Ö Close. m. Select File Ö Save to save the job metadata. n. Select File Ö Close to close the job editor. Questions: How many rows were moved to the error table because the value for Customer_Type was invalid? None Were missing values found for Customer ID? No Were there any duplicate values found for Customer_Name and Customer_Birth_Date? Yes – four pairs.

7-160


4. Standardizing Catalog Orders. a. Create initial job metadata. 1) Click the Folders tab. 2) Expand Data Mart Development Ö Orion Reports Ö Lookup Standardization. 3) that the Lookup Standardization folder is selected. 4) Select File Ö New Ö Job…. The New Job window opens. 5) Type DIFT Standardize and Report on Catalog Orders Table as the value for the Name field. 6) that the Location is set to /Data Mart Development/ Orion Reports/Lookup Standardization. 7) Click


b. Add source table metadata to the diagram for the process flow. 1) Click the Folders tab. 2) Navigate to the Data Mart Development Ö Orion Source Data folder. 3) Drag the DIFT Catalog_Orders table object to the Diagram tab of the Job Editor. c. Add the One-Way Frequency transformation to the process flow. 1) Click the Transformation tab. 2) Expand the Analysis folder and locate the One-Way Frequency transformation template. 3) Drag the One-Way Frequency transformation to the Diagram tab of the Job Editor. d. Connect the DIFT Catalog_Orders table object to the One-Way Frequency transformation. e. Select File Ö Save to save diagram and job metadata to this point. f. Specify properties for the One-Way Frequency transformation. 1) Right-click on the One-Way Frequency transformation and select Properties. 2) Click the Options tab. 3) that Assign columns is selected in the selection pane. 4) Establish CATA the Select columns to perform a one-way frequency distribution on area. a) Click in the Select columns to perform a one-way frequency distribution on area to open the Select Data Source Items window. b) Select CATALOG and then click c) Click

.

to close the Select Data Source Items window.


5) Select Specify other options in the selection pane. a) Type nocum nopercent as the value for Specify other options for TABLES statement area. 6) Select Titles and footnotes in the selection pane. a) Type CATALOG Frequency Counts as the value for the Heading 1. 7) Click


g. Select File Ö Save to save diagram and job metadata to this point. h. Run the job. 1) Right-click in background of the job and select Run. 2) Click the Status tab in the Details area. Note that all processes completed successfully. 3) Click


4) View the Log for the executed Job. 5) Click the Output tab to view the report.

7-161

7-162


i. Add the Apply Lookup Standardization transformation to the process flow. 1) Click the Transformation tab. 2) Expand the Data Quality folder and locate the Apply Lookup Standardization transformation template. 3) Drag the Apply Lookup Standardization transformation to the Diagram tab of the Job Editor. j. Connect the DIFT Catalog_Orders table object to the Apply Lookup Standardization transformation. k. Click the connection line between the table and the One-Way Frequency transformation, rightclick, and select Delete. l. Connect the Apply Lookup Standardization transformation to the One-Way Frequency transformation. m. If necessary, re-order the processing for the transformations. 1) Click the Control Flow tab in the Details area. 2) Select the Apply Lookup Standardization transformation and then click

.

n. Select File Ö Save to save diagram and job metadata to this point. o. Specify properties for the Apply Lookup Standardization transformation. 1) Right-click on the Apply Lookup Standardization transformation and select Properties. 2) Click the Standardizations tab. 3) For the CATALOG column, select DIFT Catalog Orders.sch.qkb as the value for the Scheme field. 4) For the CATALOG column, select Phrase as the value for the Apply Mode field. 5) For the OS column, select Phrase as the value for the Apply Mode field. 6) Click

to close the Apply Lookup Standardization Properties window.

p. Select File Ö Save to save diagram and job metadata to this point. q. Specify properties for the One-Way Frequency transformation. 1) Right-click on the One-Way Frequency transformation and select Properties. 2) Click the Options tab. 3) that Assign columns is selected in the selection pane.


7-163

4) Establish CATA the Select columns to transpose area. in the Select columns to perform a one-way frequency a) Click distribution on area to open the Select Data Source Items window. b) Select CATALOG and then click c) Click

.

to close the Select Data Source Items window.

5) Select Specify other options in the selection pane. a) Type nocum nopercent as the value for Specify other options for TABLES statement area. 6) Select Titles and footnotes in the selection pane. a) Type CATALOG Frequency Counts as the value for the Heading 1. 7) Click


r. Select File Ö Save to save diagram and job metadata to this point. s. Run the job. 1) Right-click in background of the job and select Run. 2) Click the Status tab in the Details area. Note that all processes completed successfully. 3) Click


4) View the Log for the executed Job. 5) Click the Output tab to view the report.

7-164


Chapter 8 Working with Tables and the Table Loader Transformation 8.1

Basics of the Table Loader Transformation ................................................................. 8-3

8.2

Load Styles of the Table Loader Transformation ........................................................ 8-8

8.3

Table Properties and Load Techniques of the Table Loader Transformation ......... 8-14

8-2

ƒ

Chapter 8 Working with Tables and the Table Loader Transformation

8.1 Basics of the Table Loader Transformation


Objectives „ „

Discuss available table loader techniques. Discuss reasons to use the Table Loader transformation.

3

Loader Transformations SAS Data Integration Studio provides three specific transformations to load data. These Loader transformations are designed to output to permanent, ed tables (that is, tables that are available in the Folder or Inventory Tree). Loaders can do the following: „ create and replace tables „ maintain indexes „ do updates and appends „ be used to maintain constraints 4

8-3

8-4


Available Loader Transformations SAS Data Integration Studio provides the following transformations for loading data into permanent output tables: „ SPD Server Table Loader „ Table Loader „ SCD Type 2 Loader

5

SPD Server Table Loader Information The SPD Server Table Loader reads a source and writes to an SPD Server target and is automatically added to a process flow when an SPD Server table is specified as a source or as a target. Options that are specific to SPD Server tables can be specified with this transformation.

6

The SPD Server Table Loader is not discussed in this class.


Table Loader Information The Table Loader is a general loader that reads a source table and writes to a target table and can be used to load SAS and most DBMS tables, as well as Excel spreadsheets. The code generated by this transformation includes syntax that is specific to the output data type.

7

The Table Loader transformation is discussed in more detail in this chapter.

SCD Type 2 Loader Information The SCD Type 2 Loader loads source data into a dimension table, detects changes between source and target rows, updates change tracking columns, and applies generated key values. This transformation implements slowly changing dimensions.

8

The SCD Type 2 Loader is discussed in more detail in a later chapter.

8-5

8-6


No Table Loader? SAS Data Integration Studio data transformations can perform a simple load of that transformation’s output table. The transformations will drop and then replace the table.

9

Why Use the Table Loader? A Table Loader transformation can also be used to load a permanent output table. The options of the Table Loader transformation control how data is loaded into the target table.

10


Why Use the Table Loader? A separate Table Loader transformation might be desirable under the following conditions: „ loading a DBMS table with any technique other than drop and replace „ loading tables that contain rows that must be updated upon load (instead of dropping and re-creating the table each time the job is executed) „ creating primary keys, foreign keys, or column constraints „ performing operations on constraints before or after the loading of the output table „ performing operations on indexes other than after the loading of the output table 11

8-7

8-8


8.2 Load Styles of the Table Loader Transformation

Objectives „

Discuss various load styles provided by the Table Loader transformation.

13

Important Step An important step in an ETL process usually involves loading data into a permanent physical table that is structured to match your data model. The designer or builder of an ETL process flow must identify the type of load that the process requires in order to: „ append all source data to any previously loaded data „ replace all previously loaded data with the source data „ use the source data to update and add to the previously loaded data based on specific key column(s) 14


8-9

Load Style In SAS Data Integration Studio, the Table Loader transformation can be used to perform any of the three load types (the Load style field on the Load Technique tab).

When the type of load is known, various techniques and options can be selected to maximize the step’s performance. 15

Add New Rows The Table Loader transformation provides two techniques for adding new rows for all three load styles.

New rows can be appended using: „ PROC APPEND with the FORCE option (default) „ PROC SQL with the INSERT statement

16

The APPEND procedure with the FORCE option is the default. If the source is a large table and the target is in a database that s bulk load, PROC APPEND can take advantage of the bulk-load feature. Consider bulk loading the data into database tables by using the optimized SAS/ACCESS engine bulk loaders. It is recommended that you use native SAS/ACCESS engine libraries instead of ODBC libraries or OLE DB libraries for relational database data. SAS/ACCESS engines have native access to the databases and have superior bulk-loading capabilities. PROC SQL with the INSERT statement performs well when the source table is small (because the overhead needed to set up bulk loading is not incurred). PROC SQL with the INSERT option adds one row at a time to the database.

8-10


Load Style - Replace The default load style is Replace. Based on the type of target table that is being loaded, three or four selections are listed for the Replace field.

Replace

Description

All rows using delete

uses PROC SQL with DELETE * to remove all rows.

Entire table

replaces the entire table using PROC DATASETS.

Simulating truncate

uses DATA step with SET and STOP statements to remove all rows (available only for SAS tables).

All rows using truncate uses PROC SQL with TRUNCATE to remove all rows (only available for some databases). 17

When Entire table is selected, the table is removed and disk space is freed. Then the table is re-created with 0 rows. Consider using this option unless your security requirements restrict table deletion permissions (a restriction that is commonly imposed by a database on database tables). Also, avoid this method if the table has any indexes or constraints that SAS Data Integration Studio cannot re-create from metadata (for example, check constraints). If available, consider using All rows using truncate. Both “All rows using …” selections enable you to keep all indexes and constraints intact during the load. By design, using TRUNCATE is the quickest way to remove all rows. The “DELETE *” syntax also removes all rows; however, based on the database and table settings, this choice can incur overhead that will degrade performance. The database or database documentation should be consulted for a comparison of the two techniques. Caution: When using All rows using delete repeatedly to clear a SAS table, the size of that table should be monitors over time. All rows using delete only performs logical deletes for SAS tables; therefore, a table’s physical size will grow and the increased size can negatively affect performance.


Load Style – Append to Existing A second load style choice is Append to Existing. The new rows are added to the existing target table by using either PROC APPEND or PROC SQL, as previously mentioned.

18

Load Style – Update/Insert When the load style is set to Update/Insert, there are many additional choices available to affect the processing.

19

8-11

8-12


Match and Update Rows The Table Loader transformation provides four techniques for matching and updating rows in a table; all four techniques are associated with the Update/Insert load style.

The Skip Matching Rows choice ignores any input rows that match rows in the output table (by selected match-columns or index). This prevents any existing rows in the output table from being modified. 20

...


8-13

Four Merge/Update Techniques Setting

Description

Modify by Column(s) specifies that the DATA STEP MODIFY BY method will be used to update output table rows with all matching input data. Modify Using Index

specifies that the DATA STEP MODIFY with KEY= method will be used to update the output table rows with all matching input data.

Skip Matching Rows

ignores any input rows that match rows in the output table (by selected match-columns or index). This action prevents any existing rows in the output table from being modified.

SQL Set

specifies that PROC SQL will be used to update output table rows with all matching input data.

21

When Modify by Column(s) is selected, the Match by Column(s) group box, which enables you to select columns, is enabled.

When Modify Using Index is selected, the Modify Using Index group box, which enables you to select an index, is enabled. Also, the Modify Using Index group box has a check box enabled, Return to the top of the index for duplicate values coming from the input data.

The options Modify by Columns and Modify using Index have the added benefit of being able to take unmatched records and add them to the target table during the same single through the source table. Of these three choices, the DATA step MODIFY with KEY= method often out-performs the other update methods in tests conducted on loading SAS tables. The DATA step MODIFY with KEY= method can also perform adequately for database tables when indexes are used. When the SQL procedure with the WHERE or SET statements is used, performance varies. Neither of these statements in PROC SQL requires data to be indexed or sorted, but indexing on the key column(s) can greatly improve performance. Both of these statements use WHERE processing to match each row of the source table with a row in the target table. The update technique chosen should depend on the percentage of rows being updated. If the majority of target records are being updated, the DATA step with MERGE (or UPDATE) might perform better than the DATA step with MODIFY BY or MODIFY KEY= or PROC SQL because MERGE makes full use of record buffers. Performance results can be hardware and operating environment dependent, so you should consider testing more than one technique.

8-14


8.3 Table Properties and Load Techniques of the Table Loader Transformation

Objectives „

„

„

Discuss various types of keys and how to define in SAS Data Integration Studio. Discuss indexes and how to define in SAS Data Integration Studio. Discuss Table Loader options for keys and indexes.

23

Keys Several transformations available in SAS Data Integration Studio, including the Table Loader transformation, can take advantage of different types of keys that can be defined for tables. This section simply defines: „ Primary Keys „ Foreign Keys „ Unique Keys „ Surrogate Keys

24


Key Definitions – Primary Key, Foreign Key Primary Key

One or more columns that are used to uniquely identify a row in a table. A table can have only one primary key. A primary key cannot contain null values.

Foreign Key

One or more columns that are associated with a primary key or unique key in another table. A table can have one or more foreign keys. A foreign key is dependent upon its associated primary or unique key. In other words, a foreign key cannot exist without that primary or unique key.

25

Key Definitions – Unique Key, Surrogate Key Unique Key

One or more columns that can be used to uniquely identify a row in a table. A table can have one or more unique keys. Unlike a primary key, a unique key can contain null values.

Surrogate Key A numeric column in a dimension table that is the primary key of that table. The surrogate key column contains unique integer values that are generated sequentially when rows are added and updated. In the associated fact table, the surrogate key is included as a foreign key in order to connect to specific dimensions. 26

8-15

8-16


Asg Key Definitions Key definitions can be assigned in the Properties window for a table object, and in Properties for some transformations.

27


8-17

Table Loader and Primary Keys The Load Technique tab of the Table Loader enables the data integration developer to specify how to handle integrity constraints regardless of the load style.

28

Integrity constraints preserve the consistency and correctness of stored data. They restrict the data values that can be updated or inserted into a table. Integrity constraints can be specified at table creation time or after data already exists in the table. In the latter situation, all data are checked to that they satisfy the candidate constraint before the constraint is added to the table. Integrity constraints are enforced automatically by the SAS System for each add, update, and delete of data to the table containing the constraint(s). Specifying constraints is the 's responsibility. There are five basic types of integrity constraints: • Not Null (Required Data) • Check (Validity Checking) • Unique (Uniqueness) • Primary Key (Unique and Not Null) • Foreign Key (Referential) The first four types of constraints are referred to as "general constraints" in this document. Foreign keys and primary keys that are referenced by one or more foreign keys are referred to as "referential constraints". Note that a primary key alone is insufficient for referential integrity. Referential integrity requires a primary key and a foreign key.

8-18


Indexes An index is an optional file that you can create for a SAS data file that does the following: „ points to observations based on the values of one or more key variables „ provides direct access to specific observations An index locates an observation by value.

29

The Purpose of Indexes Indexes can provide direct access to observations in SAS data sets to accomplish the following: „ yield faster access to small subsets (WHERE) „ return observations in sorted order (BY) „ perform table lookup operations (SET with KEY=) „ observations (PROC SQL) „ modify observations (MODIFY with KEY=)

30


Business Scenario The SAS data set orion.sales_history is often queried with a WHERE statement. Partial Listing of orion.sales_history Customer _ID

. . .

Order_ ID

14958

. . .

1230016296

Order_ Type 1

Product_ID

. . .

Product_ Group

. . .

210200600078

. . .

N.D. Gear, Kids

. . .

. . .

14844

. . .

1230096476

1

220100100354

. . .

14864

. . .

1230028104

2

240600100115

. . .

Eclipse Clothing Bathing Suits

14909

. . .

1230044374

1

240100200001

. . .

Darts

. . .

14862

. . .

1230021668

1

240500200056

. . .

Running Clothes

. . .

14853

. . .

1230021653

1

220200200085

. . .

Shoes

. . .

14838

. . .

1230140184

1

220100300042

. . .

Knitwear

. . .

14842

. . .

1230025285

1

240200100053

. . .

Golf

. . .

14815

. . .

1230109468

3

230100700004

. . .

Tents

. . .

14797

. . .

1230168587

1

220101000004

. . .

Shorts

. . .

. . .

31

Business Scenario You need to create three indexes on the most frequently used subsetting columns. Index Name

Index Variables

Customer_ID

Customer_ID

Product_Group

Product_Group

SaleID

Order_ID Product_ID

Partial Listing of orion.sales_history Customer _ID

. . .

Order_ ID

14958

. . .

1230016296

14844

. . .

1230096476

32

Order_ Type

Product_ID

. . .

Product_ Group

. . .

1

210200600078

. . .

N.D. Gear, Kids

. . .

1

220100100354

. . .

Eclipse Clothing

. . .

8-19

8-20


Simplified Index File The index file consists of entries that are organized in a tree structure and connected by pointers. Partial Listing of

orion.sales_history Customer_ID Employee_ID 14958 121031 14844 121042 14864 99999999 14909 120436 14862 120481 14853 120454 14838 121039 14842 121051 14815 99999999 14797 120604 . . .

. . .

. . . . . . . . . . .

. . . . . . . . . . .

. . . . . . . . . . . . . .

33

Customer_ID

Simplified Index

Key Value

Record Identifier (RID) Page(obs, obs, ...)

4006

17(85)

4021

17(89)

4059

17(90)

4063

17(80, 86)

. . . 14958

1(1, 24)

14972

1(14)

. . .

Why Use an Index? How is data processed if the input data is not indexed? data customer14958; set orion.sales_history; where Customer_ID=14958; run;

34


Reading SAS Data Sets without an Index Input SAS Data

Buffers Data pages are loaded.

Output SAS Data

Buffers

The WHERE statement selects observations by reading data sequentially. PDV Customer_ID

Employee_ID

40

Why Use an Index? How is data processed if the input data is indexed? data customer14958; set orion.sales_history; where Customer_ID=14958; run;

41

…

8-21

8-22


Reading SAS Data Sets with an Index Index

Index

Input SAS Data

Buffers Only necessary pages are loaded.

Output SAS Data

The index file is checked.

The WHERE statement selects observations by using direct access. PDV Customer_ID

Employee_ID

Buffers

48

SAS Data Integration Studio and Indexes Indexes can be added to a table definition by using the Properties window:

49

PROC DATASETS and PROC SQL can also be used in -written code to create indexes.

…


Indexes on SAS Data Tables For large-scale ETL flows, indexes on SAS data sets might be useful to perform the following processes: „ resolve WHERE clauses „ allow an index as an optimization for two-table equis „ key uniqueness

50

WHERE Clauses Resolved in ETL Flows In the context of ETL flows, indexes on SAS data sets are beneficial for processing selective WHERE clauses that qualify for index use. When resolving a WHERE clause, if the number of rows that qualify for the subsetting is 15% or less than the total number of rows, then the index is used; otherwise, a sequential scan occurs. Keep this in mind when building indexes, because the overhead of building and maintaining an index is costly.

51

8-23

8-24


Index Usage in SAS MSGLEVEL=

controls the level of detail in messages that are written to the SAS log.

If MSGLEVEL=I, SAS writes informative messages to the SAS log about index processing. In general, when a WHERE expression is executed for a data set with indexes: „ if an index is used, a message displays that specifies the name of the index. „ if an index is not used but one exists that could optimize at least one condition in the WHERE expression, messages provide suggestions that describe what you can do to influence SAS to use the index. For example, a message could suggest to sort the data set into index order or to specify more buffers. 52

Controlling Indexes and Constraints The Load Technique tab of the Table Loader transformation can be used to control the timing of index and constraint removal. There are four settings that enable you to specify the desired conditions for the indexes and constraints before and after the load.

53


Condition Settings The Constraint Condition and Index Condition options that are available will depend on the load technique specified. The choices translate to three different tasks: „ Put on „ Take off „ Leave as is

54

Condition Settings Information

55

„

If leave off/off is specified for the Before Load option(s), then the generated code checks for and removes any indexes (or constraints) that are found and then loads the table.

„

If an index is required for an update, then that index is not removed or it will be added, as needed.

„

If leave on/on is specified for the After Load option(s), then the indexes will be added after the load.

8-25

8-26


Physical Table Out of Sync with Metadata If indexes are specified as part of a table’s metadata, and if load options are specified to drop indexes and/or constraints, then care needs to be taken not to allow the physical table not to remain out-of-sync with the table metadata.

56

Remove Indexes Before Load There are times when performance can be improved by removing non-essential indexes before a load and re-creating those indexes after the load. General Rule Consider removing and re-creating indexes if more than 10% of the data in the table will be re-loaded.

57


Remove Constraints Before Load Performance can also be improved if key constraints are temporarily removed. Example: If there are a significant number of transactions and the data that is being loaded conforms to the constraints, then removing the constraints from the target before a load removes the overhead that is associated with maintaining those constraints during the load.

58

8-27

8-28


Chapter 9 Working with Slowly Changing Dimensions 9.1

Defining Slowly Changing Dimensions ........................................................................ 9-3

9.2

Using the SCD Type 2 Loader and Lookup Transformations ................................... 9-15 Demonstration: Populate Star Schema Tables Using the SCD Type 2 Loader with the Surrogate Key Method................................................................................ 9-29

9.3

Introducing the Change Data Capture Transformations ........................................... 9-93

9-2

Chapter 9 Working with Slowly Changing Dimensions

9.1 Defining Slowly Changing Dimensions


Objectives „ „ „ „

Explain slowly changing dimensions. Define keys. List benefits of slowly changing dimensions. Define SCD types 1, 2, and 3.

3

Understanding Slowly Changing Dimensions Slowly changing dimensions refers to a process that updates dimension tables in a star schema in a way that preserves a history of changes.

4

9-3

9-4


Star Schema (Review) A star schema is a partly normalized data model that separates data into two categories: facts and dimensions. Facts

Event-based numeric information, such as number of items sold and invoice amount in sales transactions

Dimensions Descriptive information that or give context to the facts, such as customer information, product information, and calendar information for sales transactions A star schema uses keys to specify relationships between the tables. 5

Star Schema Keys The following keys are used in a star schema:

6

Business Key

identifies a business entity, such as Customer_ID and Product_ID.

Primary Key

uniquely identifies a row in a dimension table.

Foreign Key

associates a row in the fact table to the corresponding row in a dimension table.


Primary and Foreign Keys (Review) A data model that separates data into tables uses primary/foreign keys to maintain referential integrity. „ Each dimension table in a star schema has a primary key. „ The associated foreign keys are in the fact table.

7

Primary and Foreign Keys (Review) Referential integrity constraints include: „ primary keys are unique and not null (dimensions) „ foreign keys (in the fact table) can only have values that exist in the primary key „ a primary key value cannot be deleted if it exists in a foreign key

8

9-5

9-6


Primary and Foreign Keys (Review) The key values „ relate records in the fact table to records in the dimension tables „ can be used to the fact table with one or more dimension tables for analysis and reporting.

9

Business Keys Often the business key in a dimension table can function as a primary key in that table „

Customer_ID in Customer Dimension Product_ID in Product Dimension

„

etc…

„

10


Using Slowly Changing Dimensions With slowly changing dimensions: „

dimension tables are structured to keep historical records

„

dimension tables can accumulate multiple records per business key

„

an alternate column or a combination of columns must be used as a primary key

„

when a new record is loaded into the fact table, key values are included that link that record to the corresponding current records in the dimension tables

„

the historical facts maintain their correct business context

11

Examples of Slowly Changing Dimensions Examples of dimensional changes over time:

12

„

product name, product price

„

country name

„

customer name, customer address

„

organization name, organization address

„

employee job title, employee salary

9-7

9-8


Additional Benefits of Slowly Changing Dimensions The record of changes in the dimension tables also provide a basis for: „

historical reporting

„

trend analysis of demographic data

13

Value of Historical Records Historical reporting: Through a merger a customer has been absorbed into a new organization. In analyses of purchases by this customer, the transactions before the merger can be distinguished from the transactions after the merger. Trend analysis: A customer dimension table can be analyzed for changes in mailing addresses in order to establish a subset of customers who are most likely to remain in a given residence for a given amount of time. This information enables those customers to be targeted for special buying incentives.

14


Types of Slowly Changing Dimensions Slowly changing dimensions are categorized into three types: „

Type 1: no history

„

Type 2: history in rows

„

Type 3: history in columns

15

Example of Slowly Changing Dimensions Consider an Employee Salary table to explain the three types of slowly changing dimensions. In the Employee Salary table Emp_ID „

is the business key. Each value of Emp_ID uniquely identifies a business entity

„

can be used as a primary key Employee Salary Information Emp_ID

Emp_Name

Year

Salary

K Munch

2004

$42,000

(PK) 1649

16

9-9

9-10


Type 1 Slowly Changing Dimension A Type 1 SCD is updated by replacing old values with new values. In the year 2005 the salary of K Munch changes. The old year (2004) and salary ($42,000) are updated by replacing them with the new values. With Type 1 SCD, old values are not recoverable. The table only contains current information. Employee Salary Information Emp_ID

Emp_Name

Year

Salary

K Munch

2005

$45,000

(PK) 1649

17

Type 2 Slowly Changing Dimension A Type 2 SCD is updated by adding a record when a change occurs. In this Type 2 SCD a record is added with the new year and salary values.

Emp_ID can no longer be used as a primary key.

Employee Salary Information Emp_ID

Emp_Name

Year

Salary

1649

K Munch

2004

$42,000

1649

K Munch

2005

$45,000

(PK)

18


Type 2 Slowly Changing Dimension An effective datetime and an expiration datetime may be added to the table. These specify when the record is valid. The combination of Emp_ID with the effective datetime can be used as a primary key. Employee Salary Information

Emp_I D (PK)

Effective DateTime (PK)

Emp_Name Year Salary Expiration DateTime

1649

01Jan2004 K Munch 12:00PM

2004

$42,000

1649

01Jan2005 K Munch 12:00PM

2005

$45,000

31Dec2004 11:59PM

19

Type 3 Slowly Changing Dimension A Type 3 SCD is updated by moving the old value to a column holding old values and adding the new value into a column holding the most recent value. Employee Salary Information Old Old Salary Emp_ID Emp_Name Year Salary Year (PK) 1649

K Munch

2005

$45,000

2004

$42,000

The Type 3 SCD is limited to retaining only a fixed number of historical values. Employee Salary Information Old Old Emp_ID Salary Emp_Name Year Salary Year (PK) 1649 20

K Munch

2007

$58,000

2006

$50,000

9-11

9-12


Updating a Star Schema SAS Data Integration Studio provides transformations to implement type 1 and type 2 slowly changing dimensions. SCD Type 2 loads a dimension table, with Loader configurable type 2 primary key generation and configurable date/version/current record identification. It also provides type 1 loading. Lookup

loads a fact table while using the dimension tables as lookup tables for key values.

21

Loading a Dimension Table The SCD Type 2 transformation 1. loads new source records into a dimension table 2. uses the business key to detect new or historical records.

1 2

22


Loading a Dimension Table The SCD Type 2 transformation 3. generates surrogate key values or 4. generates retained key values.

3

4

23

Loading a Fact Table The Lookup transformation 1. loads new source records into a fact table 2. uses the business key for lookups into the dimension tables tables.

2

24

1

9-13

9-14


Loading a Fact Table The Lookup transformation 3. retrieves key values from the dimension tables and adds them to the fact table.

3

25

9.2 Using the SCD Type 2 Loader and Lookup Transformations


Objectives „ „ „ „ „

List the functions of the SCD Type 2 transformation. Define business keys. Define surrogate and retained keys. Detect and track changes. List the functions of the Lookup transformation.

28

SCD Type 2 Loader Transformation The SCD Type 2 Loader transformation implements type 2 and type 1 slowly changing dimensions. It is used to update dimension tables. It performs the following tasks: „ detects changes „ tracks changes „ generates key values Generated key values give the dimension table a primary key that is not dependent on the business key.

29

9-15

9-16


Business Key The business key consists of one or more columns that identify a business entity, like a customer, a product, or an employee. The Business Key tab is used to specify one or more columns in a target dimension table that represent the business key.

30

Change Detection The business key is used as the basis for change detection. The business keys in source rows are compared to the business keys in the target. The Detect Changes tab is used to specify one or more columns in a dimension table that are monitored for changes.

31


Change Detection By default all columns are included in change detection, except the columns that are specified in the Change Tracking, Business Key, and Generated Key tabs.

32

Change Tracking The SCD Type 2 Loader provides three methods for tracking historical records: „ Beginning and End Date (or datetime) values „ Version number „ Current record indicator

33

9-17

9-18


Change Tracking The Change Tracking tab is used to specify one or more methods and associated columns in a target dimension table to be used for tracking historical records. Multiple methods can be selected.

34

Generated Key The SCD Type 2 Loader generates values for a Generated Key column in the target dimension table. The Generated Key tab is used to specify a column for the generated key values as well as a method for generating the key values.

35


Generated Key The generated key: „ eliminates dependencies on the source data, as the business key may be subject to redefinition, reuse, or recycling „ can be used as a primary key or as part of a composite primary key „ is generated at run time for each new row that is added to the target

36

Generated Key The generated key can be specified as a surrogate key or a retained key.

37

Surrogate Key

A numeric column in the target table that is assigned a unique integer value for each row in the table. It can function as a primary key.

Retained Key

A numeric column in the target table that is assigned a unique integer value for each business key value. It is combined with a begin-date(time) column to make up a primary key.

9-19

9-20


Updating with the Surrogate Key Method An incoming source record for the Customer Dimension table has a changed Annual Income: Cust ID

Customer Name

Annual Income

0001

Sam Cook

$45,000

With the Surrogate Key method, a row is added as follows: Cust Cust Customer Annual ID GenKey Name Income

Begin Date

End Date

Current

0001 1

Sam Cook $40,000 01Jan05

31Dec05 0

0002 2

B Diddley

$42,000 01Jan05

.

1

0003 3

John Doe

$25,000 01Jan05

.

1

0001 4

Sam Cook $45,000 01Jan06

.

1

38

Cust Id is the business key in this example. The surrogate key method generates a new key value for each added row.


Updating with the Retained Key Method (1) An incoming source record for the Customer Dimension table has a changed Annual Income: Cust ID

Customer Name

Annual Income

0001

Sam Cook

$45,000

With the Retained Key method (new column), the added row is as follows: ---Primary Key---

Cust GenKey

Begin Date

Customer Name

Annual Income

End Date Current

1

01Jan05

Sam Cook

$40,000

31Dec05

0

2

01Jan05

B Diddley

$42,000

.

1

3

01Jan05

John Doe

$25,000

.

1

1

01Jan06

Sam Cook

$45,000

.

1

39

...

Cust Id is the business key in this example. The retained key method with new column generates a new retained key value if the added row represents a new business key. If the added row represents an existing business key: • the same retained key is assigned • the old row is closed out (end date assigned or current record turned off) • the new row is opened (begin date assigned or current record turned on)

9-21

9-22


Updating with the Retained Key Method (2) An incoming source record for the Customer Dimension table has a changed Annual Income: Cust ID

Customer Name

Annual Income

0001

Sam Cook

$45,000

With the Retained Key method (existing column), the added row is as follows (there is no generated key): ---PrimaryKey---

Cust ID

Begin Date

Customer Name

Annual Income

End Date Current

0001

01Jan05

Sam Cook

$40,000

31Dec05

0

0002

01Jan05

B Diddley

$42,000

.

1

0003

01Jan05

John Doe

$25,000

.

1

0001

01Jan06

Sam Cook

$45,000

.

1

40

...

Cust Id is the business key in this example. The retained key method with existing column does not generate key values. If the added row represents an existing business key: • the old row is closed out (end date assigned or current record turned off) • the new row is opened (begin date assigned or current record turned on)

Change Tracking: Retained Key Method Retained keys consist of a generated integer column and a begin date or datetime column. Change tracking by date or datetime is required with a retained key. Retained Key method (new column)

Retained Key method (existing column)

---Primary Key---

---PrimaryKey---

Cust GenKey

41

Begin Date

Cust ID

Begin Date

1

01Jan05

0001

01Jan05

2

01Jan05

0002

01Jan05

3

01Jan05

0003

01Jan05

1

01Jan06

0001

01Jan06


9-23

Updating Dimension Tables During the update of a dimensional target, incoming source rows are compared to the current row for each entry in the target tables. Depending on data content, source rows can do the following: „ „ „

begin new entries add new current rows to existing entries close out existing entries

42

An entry consists of all the rows (the current row and the historical rows) for a business entity represented by a business key value. If a source row has a business key that does not exist in the target, then that row represents a new entry. The new row is added to the target with appropriate change-tracking values. If a source row has the same business key as a current row in the target, and a value in a column identified as a Change Detection column differs, then that row represents an update to an existing entry. The source row is added to the target. The source row becomes the new current row for that entry; it receives appropriate change-tracking values. The superseded target row can also receive new change-tracking values (closed out). If a source row has the same business key and content as a current row in the target, it might indicate that the entry is being closed out. The entry is closed out if change tracking is implemented with begin and end datetime values, and if the end datetime value in the source is older than the same value in the target. When this is the case, the new end date is written into the target to close out the entry. If a source row has the same business key and the same content as a current row in the target, then that source row is ignored.

9-24


About Cross-Reference Tables For efficiency, cross-reference tables are used during updates of dimension tables or fact tables by the following transformations: „ „ „

SCD Type 2 Loader Key Effective Date Lookup

A cross-reference table contains only the current rows in the lookup or target table. Columns in the cross-reference table include the key columns and sometimes a digest column.

43

The digest column contains a concatenated encryption of values from selected columns other than the key columns. It is a character column with a length of 32 and named DIGEST_VALUE. The encrypted concatenation uses the MD5 algorithm.

Lookup Transformation The Lookup transformation can be used to load a target table with columns taken from a source and from a number of lookup tables. When a job containing a Lookup transformation is run, each source row is processed as follows:

44

„

The key columns in the source row are compared to the key columns in the specified lookup tables.

„

If matches are found, specified lookup values and source values are added to the target row in the transformation temporary output table.

„

The temporary output table rows are then loaded into the target.


Updating Fact Tables An example of using the Lookup transformation is to load source data into a fact table in a star schema. The dimension tables are used as lookup tables.

45

Lookups Tab The Lookups tab in the Lookup transformation is used to specify lookup properties: „

Source to Lookup columns

„

Lookup to Target columns

„

Where expression

„ Exceptions A set of lookup properties must be specified for each Lookup table.

46

9-25

9-26


Lookups: Source to Lookup Mapping The column(s) specified in the Source to Lookup Mapping tab are used to find the matching record in the lookup (dimension) table for each source record. Map the business key column(s) in the source table to the corresponding column(s) in the lookup table.

47

Lookups: Lookup to Target Mapping The column(s) specified on the Lookup to Target Mapping tab are copied from the lookup (dimension) table to the target (fact) table when a match is found. Map the generated keys such as the surrogate or retained key and possibly the begindate (time) in the lookup table to the corresponding columns in the target table.

48


Lookups: Selection Criteria The Where tab specifies selection criteria that refine the match between source row and the lookup table.

49

Lookups: Exception Handling The Exceptions tab is used to specify exception actions. Available Conditions

Available Actions

Lookup table missing

Abort

Lookup table contains no records

Skip the record

Lookup value not found

Add row to error table Add row to exception table Set target column value Set target column value to missing None

50

9-27

9-28


Error Tables The Errors tab in the properties of the Lookup transformation is used to specify: „

Errors Table: includes a generated column (source row number) and any column from the source data.

„

Exception Table: includes four generated columns (exception information) and any column from the source data.

51

Steps for Workshop 1. Define metadata for source (external file) 2. Define metadata for targets (2 dimensions,1 fact) 3. Create job flow diagram 4. Propagate Columns from Sources to Targets 5. Specify properties of the transformations (File Reader, Column Splitter, Sort Dedup, SCD Type 2, Lookup, Table Loader) 6. Specify the Processing order

52


9-29

Populate Star Schema Tables Using the SCD Type 2 Loader with the Surrogate Key Method to SAS Data Integration Studio as Barbara 1. If necessary, access SAS Data Integration Studio using Barbara’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. Select Barbara’s Work Repository as the connection profile. c. Click




9-30


Define Metadata for the Source Table (External CSV File) 1. the Source Data Template data set. The Source Data Template data set is a SAS data set that was prepared with 29 column definitions for use in this workshop. The data set has no rows, so it stores no data. It serves only as a repository for column definitions. a. Select the Folders tab. b. Expand Data Mart Development Ö Orion SCD. c. that the Orion SCD folder is selected. d. Select File Ö Tables…. The Tables wizard starts.

e. Select SAS as the type of table and click displayed.

. The Select a SAS Library window is


f.

Select the DIFT SAS Library from the SAS Library drop-down list and click

g. Select the Orion_Source_Data_Template table and click

.

9-31

.

9-32


h. Review the final information, and then click

.

i.

Rename Orion_Source_Data_Template. Right-click Orion_Source_Data_Template and select Properties.

j.

Type DIFT SCD Source Data Template in the Name field and click

.


9-33

2. Define metadata for the source data table. a. Select the Folders tab. b. Expand Data Mart Development Ö Orion SCD. c. that the Orion SCD folder is selected. d. Select File Ö New Ö External File Ö Delimited…. The New Delimited External File wizard starts.

e. Type DIFT SCD Source Data as the value for the Name field. f.

that the location is set to /Data Mart Development/Orion SCD.

9-34


g. Click

1) Click

. The External File Location window is displayed.


2) Navigate to S:\Workshop\dift\data. 3) Change the Type field to Delimited Files (*.csv). 4) Select OrionSourceDataM01.csv.

5) Click

to close the Select a file window.

h. that the correct file is selected

i.

To view the contents of the file, click

.


9-35

The file has 7 records. Note that the first record has column names and that the data fields are comma-delimited. j.

Click

k. Click

to close the Preview File window. . The Delimiters and Parameters window is displayed.

1) Clear Blank and select Comma.

2) Click

. The Advanced File Parameters window is displayed.

3) Set the Start record value to 2.

4) Click l.

to close the Advanced File Parameters window.

. The Column Definitions window is displayed. Increase the size of the window Click by dragging the corners.

m. Click

to see the source data.

9-36


to close the Warning

n. A dialog window indicates that there are only 7 records. Click dialog window. o. Import column definitions from a template data set. 1) Click

. The Import Column Definitions window opens.

2) Select Get the column definitions from other existing tables or external files. 3) Click

. The Import Columns window opens.

4) Select the Folders tab. 5) Expand Data Mart Development Ö Orion SCD. 6) Select the DIFT SCD Source Data Template table and click

.


9-37

7) 29 columns from the DIFT SCD Source Data Template table are selected. 8) Click

to close the Import Columns window.

9) Click

to close the Import Column Definitions window.

p. Definitions for 29 columns are imported. These column definitions will be forward propagated through the process flow to the target tables.

9-38


q. the data. Select the Data tab in the lower pane and click

. Scroll to the data.

.

r.

Click

s.

Review the final information, and then click

.


9-39

9-40


Define Metadata for the Target Tables 1. Define metadata for the Customer Dimension target table. a. Select the Folders tab. b. Expand Data Mart Development Ö Orion SCD. c. that the Orion SCD folder is selected. d. Select File Ö New Ö Table…. The New Table wizard starts.

e. Type DIFT SCD Customer Dimension as the value for the Name field. f.

that the location is set to /Data Mart Development/ Orion SCD.

g. Click

.

h. that the DBMS field is set to SAS.


i.

Define a new library. 1) Click

next to the Library field. The New Library Wizard opens.

2) Type DIFT SCD Target Tables Library as the value for the Name field. 3) that the location is set to /Data Mart Development/ Orion SCD. 4) Click

.

5) Select SASApp in the Available servers pane.

6) Click 7) Click

to move SASApp to the Selected servers pane. .

9-41

9-42


8) Type diftscd as the value for the Libref field.

9) Define a new path for this library. a) Click in the Path Specification area. A new path needs to be created for the SCD tables.

b) Click

next to Paths in the New Path Specification window.

c) In the Browse window, navigate to S:\Workshop\dift. d) Click


e) Type scdtarget as the new folder name and press ENTER.


9-43

f) Single-click on the new folder, scdtarget, to select it.

g) Click


h) Click


i)

that S:\Workshop\dift\scdtarget is now listed in the Selected items pane.

10) Click

.

11) the information is correct, and then click j.

.

Change the default name of the new table. Type SCDCustDim in the Name field.

9-44


k. Click l.

. The Select Columns window is displayed.

Do not select columns at this time. You propagate columns from sources to the targets in a later . step. Click

m. The Change Columns/Indexes window displays. No actions are needed. Click dialog box displays.

n. Click

.

o. Review the metadata listed in the New Table window. p. Click

.

2. Define metadata for the Product Dimension target table. a. Select the Folders tab. b. Expand Data Mart Development Ö Orion SCD. c. that the Orion SCD folder is selected. d. Select File Ö New Ö Table…. The New Table wizard starts. e. Type DIFT SCD Product Dimension as the value for the Name field.

.A


f.

9-45

that the location is set to /Data Mart Development/ Orion SCD. .

g. Click


Select DIFT SCD Target Tables Library as the value for the Library field.

j.

Change the value in the Name field to SCDProdDim.

k. Click l.

. The Select Columns window is displayed.

Do not select columns at this time. You propagate columns from the source to the targets in a later step.

m. Click

. The Change Columns/Indexes window displays.

n. Click

. A dialog saying that no columns are specified displays.

o. Click

.

p. Review the metadata listed in the New Table window. q. Click

.

3. Define metadata for the Order Fact target table. a. Select the Folders tab. b. Expand Data Mart Development Ö Orion SCD. c. that the Orion SCD folder is selected. d. Select File Ö New Ö Table…. e. Type DIFT SCD Order Fact as the value for the Name field. f.

that the location is set to /Data Mart Development/ Orion SCD.

g. Click

.

h. that the DBMS field is set to SAS.

9-46


i.

Select DIFT SCD Target Tables Library as the value for the Library field.

j.

Change the value in the Name field to SCDOrderFact. . The Select Columns window is displayed.

k. Click l.

Do not select columns at this time. You propagate columns from the source to the targets in a later step.

m. Click

. The Change Columns/Indexes window displays.

n. Click

. A dialog saying that no columns are specified displays.

o. Click

.

p. Review the metadata listed in the finish window. q. Click

.


Define Metadata for the Job 1. Orient the process flow from top to bottom and turn off automatic column propagation. a. Select Tools Ö Options. b. Select the Job Editor tab. c. Click Top To Bottom in the Layout area.

2. Turn off automatic column mapping and automatic column propagation. a. Locate the Automatic Settings area.

b. Deselect Automatically map columns. c. Deselect Automatically propagate columns. d. Click


3. Create the initial job metadata. a. Select the Folders tab. b. Expand Data Mart Development Ö Orion SCD. c. that the Orion SCD folder is selected. d. Select File Ö New Ö Job…. The New Job window opens.

9-47

9-48


e. Type Populate Star Schema with SCD-RK Processing as the value for the Name field.

f.

that the Location is set to /Data Mart Development/Orion SCD.

g. Click


4. Add the source table to the diagram for the process flow. a. Select the Checkouts tab. b. Drag the DIFT SCD Source Data external file to the Diagram tab of the Job Editor. 5. Add a File Reader transformation to the process flow. a. Select the Transformations tab. b.

Expand the Access group and locate the File Reader transformation template.

c. Drag the File Reader transformation to the Diagram tab of the Job Editor. Place it under the DIFT SCD Orion Detail Information external file object. d. Connect the DIFT SCD Orion Detail Information external file object to the File Reader transformation.

6. Add a Splitter transformation to the process flow. The Splitter transformation is used to split the columns from the source. Customer information columns go to the Customer dimension table, product information columns go to the Product dimension table, and fact columns go to the Fact table. a. Select the Transformations tab. b. Expand the Data group and locate the Splitter transformation template. c. Drag the Splitter transformation to the Diagram tab of the Job Editor. Place it under the File Reader transformation.


9-49

d. Connect the File Reader transformation to the Splitter transformation.

7. Add two Sort transformations to the process flow. Each source data file in this demonstration represents a monthly extract from a transactional database. All transactions in a monthly extract must be added to the Fact table. But only unique Customer records and unique Product records must be added to the Dimension tables. The Sort transformation with the Nodupkey option will be used to remove duplicates. a. Select the Transformations tab b. Expand the Data group and locate the Sort transformation template. c. Drag the Sort transformation to the Diagram tab of the Job Editor. d. Connect one of the temporary table outputs from the Splitter to one of the Sort transformations. e. Drag a second Sort transformation to the Diagram tab of the Job Editor. f.

Connect the second temporary table output from the Splitter to the other Sort transformation.

9-50


8. Select File Ö Save to save diagram and job metadata to this point. 9. Add two SCD Type 2 Loader transformations to the process flow. a. Select the Transformations tab. b. Expand the Data group locate the SCD Type 2 Loader transformation template. c. Drag the SCD Type 2 Loader transformation to the Diagram tab of the Job Editor. d. Connect the temporary table output from one Sort transformation to one of the SCD Type 2 Loader transformations. e. Drag a second SCD Type 2 Loader transformation to the Diagram tab of the Job Editor. f.

Connect the temporary table output from the second Sort transformation to the other SCD Type 2 Loader transformation.

10. Add the two target dimension tables. a. Select the Checkouts tab. b. Drag the DIFT SCD Customer Dimension table object to the Diagram tab of the Job Editor, placing it under one of the SCD Type 2 Loader transformations. c. Connect the SCD Type 2 Loader transformation to the DIFT SCD Customer Dimension table object. d. Drag the DIFT SCD Product Dimension table object (from the Checkouts tab) to the Diagram tab of the Job Editor, placing it under one of the SCD Type 2 Loader transformations.


9-51

e. Connect the SCD Type 2 Loader transformation to the DIFT SCD Product Dimension table object.

11. Select File Ö Save to save diagram and job metadata to this point. 12. Add a third output table to the Splitter by right-clicking on the Splitter transformation and selecting Add Work Table.

13. Add the Lookup transformation to the process flow. a. Select the Transformations tab. b. Expand the Data group and locate the Lookup transformation template. c. Drag the Lookup transformation to the Diagram tab of the Job Editor. d. Connect the third temporary output table from the Splitter transformation to the Lookup transformation.

This must be the first input to the Lookup transformation.

e. Next connect the DIFT SCD Customer Dimension table object to the Lookup transformation.

9-52


f.

Add a third input port to the Lookup transformation by right-clicking on the Lookup transformation and selecting Ports Ö Add Input Port.

g. Connect the DIFT SCD Product Dimension table object to the Lookup transformation.

14. Add a Table Loader transformation to the process flow. a. Select the Transformations tab. b. Expand the Access group and locate the Table Loader transformation template. c. Drag the Table Loader transformation to the Diagram tab of the Job Editor.


d. Connect the Lookup transformation to the Table Loader transformation.

15. Add the DIFT SCD Order Fact table as the final output for this process flow. a. Select the Checkouts tab. b. Drag the DIFT SCD Order Fact table object to the Diagram tab of the Job Editor, placing it under the Table Loader transformations.

9-53

9-54


c. Connect the Table Loader transformation to the DIFT SCD Order Fact table object.

16. Select File Ö Save to save diagram and job metadata.


9-55

Propagate Columns from Sources to Targets 1. There are currently no columns defined in the temporary output tables or the target tables. You manually propagate columns forward starting at the source table. 2. Define columns for the temporary output table from the File Reader. a. Right-click on the File Reader and select Properties. b. Select the Mappings tab. The 29 columns in the DIFT SCD Source Data table are listed in the Source table pane on the left. Propagate all 29 columns to the Target table pane. Click

to propagate all columns from sources to targets.

All 29 columns are propagated forward from the DIFT SCD Source Data table to the temporary output table of the File Reader transformation. c. Click

to close the File Reader properties window.

3. Rename the temporary output tables from the Splitter transformation. a. Right-click on the first temporary output table (leading to the DIFT SCD Customer Dimension table) from the Splitter and select Properties. b. Select the Physical Storage tab.

9-56


c. Enter TempForCustDim as the value for the Name field.

d. Click

to close the Splitter 0 Properties window.

e. Right-click on the second temporary output table (leading to the DIFT SCD Product Dimension table) from the Splitter and select Properties. f.

Select the Physical Storage tab.

g. Enter TempForProdDim as the value for the Name field.

Click


h. Right-click on the first temporary output table (leading to the Lookup transformation and the DIFT SCD Order Fact table) from the Splitter and select Properties. i.

Select the Physical Storage tab.

j.

Enter TempForLookup as the value for the Name field.

Click


4. Define columns for the temporary output tables from the Splitter. a. Right-click on the Splitter and select Properties. b. Select the General tab. c. Type Column Splitter in the Name field.


9-57

d. This is a reminder that the splitter is used to direct different columns from the data source to the three target tables. e. Select the Row Selection tab. f.

that All Rows are selected for each of the three output tables.

g. Select the Mappings tab. The 29 columns in the temporary output table from the File Reader are listed in the Source table pane on the left. Propagate only the necessary columns to each output table. h. Select the Splitter 0 (TempForCustDim) table from the Target table drop-down list.

i.

Select columns 1 through 11 in the Source table pane. Use the Shift key to make the selection. The selected columns should include only: Customer_ID Customer_Country Customer_Gender Customer_Name Customer_FirstName Customer_LastName Customer_Birth_Date Customer_Type Customer_Group Customer_Age Customer_Age_Group

9-58


j.

Click

and select Selected Source Columns Ö To Targets.

11 columns are propagated forward and mapped. k. Select the Splitter 1 (TempForProdDim) table from the Target table drop-down list.

l.

Select columns 12 through 19 in the Source table pane. Use the Shift key to make the selection. The following 8 columns should be selected. Product_ID Product_Name Supplier_ID Supplier_Name Supplier_Country Product_Group Product_Category Product_Line


m. Click

9-59


Now you have 8 columns propagated forward and mapped. n. Select the Splitter 2 (TempForLookup) table from the Target table drop-down list.

o. Select columns 20 through 29 in the Source table pane. Use the Shift key to make the selection. The 9 selected columns are these: Order_ID Order_Item_Num Quantity Total_Retail_Price CostPrice_Per_Unit Discount Order_Type Employee_ID Order_Date Delivery_Date p. Click


q. Select Customer_Id (column 1) and while holding the CTRL key, select Product_Id (column 12). The two selected columns are these: Customer_ID Product_ID

9-60


r.

Click


This gives you 12 columns propagated forward and mapped. s.

Click


5. Define columns for the temporary output tables from the Sort transformations. a. Right-click on the first Sort transformation and select Properties. b. Select the General tab.


c. Type Sort Dedup in the Name field.

This is a reminder that the sort transformation will be used to remove records with duplicate Customer Id values. d. Select the Mappings tab. e. Click

to propagate and map all 11 columns from sources to targets.

All 11 columns are propagated forward and mapped. f.

Click

to close the Sort properties window

g. Right-click on the second Sort transformation and select Properties. h. Select the General tab.

9-61

9-62


i.

Type Sort Dedup in the Name field.

This is a reminder that this sort transformation will be used to remove records with duplicate Customer Id values. j.

Select the Mappings tab.

k. Click


All 8 columns are propagated forward and mapped. l.

Click

to close the Sort properties window.

6. Propagate columns to the Dimension target tables. a. Right-click on the first SCD Type 2 transformation and select Properties. b. Select the Mappings tab. c. Click d. Click

to propagate and map all 11 columns from sources to targets. to close the SCD Type 2 Properties window.

e. Right-click on the DIFT SCD Customer Dimension table and select Properties.


f.

Select the Columns tab and that it has the 11 propagated columns.

g. Add 4 new columns. h. Select the Customer_ID column. 1) Click

4 times to insert 4 new columns.

2) Update the metadata for the new columns as follows:

to close the DIFT SCD Customer Dimension Properties window.

i.

Click

j.

Right-click on the second SCD Type 2 transformation and select Properties.

k. Select the Mappings tab. l.

Click

m. Click

to propagate and map all 8 columns from sources to targets. to close the SCD Type 2 properties window.

n. Right-click on the DIFT SCD Product Dimension table and select Properties. o. Select the Columns tab and that it has the 8 propagated columns. 7. Add metadata for 4 new columns. a. Select the Product_ID column. b. Click

4 times to insert 4 new columns.

c. Update the metadata for the new columns as follows:

d. Click

to close the DIFT SCD Product Dimension Properties window.

8. Define columns for the temporary output table from the Lookup transformation. a. Right-click on the Lookup transformation and select Properties. b. Select the Mappings tab.

9-63

9-64


c. Click


d. Add metadata for 4 new columns in the Target table pane. 1) Select the Customer_ID column in the Target table pane. 2) Click


3) Select the Folders tab. 4) Expand Data Mart Development Ö ORION SCD Ö DIFT SCD Customer Dimension. 5) Select the GenKeyCust and BeginDateTimeCust columns and click

6) Click

.

to close the Import Columns window.

7) Select the Product_ID column in the Target table pane. 8) Click


9) Select the Folders tab. 10) Expand Data Mart Development Ö Orion SCD Ö DIFT SCD Product Dimension.


11) Select the GenKeyProd and BeginDateTimeProd columns and click

.

16 columns are defined for the temporary output table from the Lookup transformation. e. Click

to close the Lookup Properties window.

9. Propagate columns to the DIFT SCD Order Fact target table. a. Right-click on the Table Loader transformation and select Properties. b. Select the Mappings tab. c. Click d. Click

to propagate all 16 columns from sources to targets. to close the Table Loader properties window.

e. Right-click on the DIFT SCD OrderFact table and select Properties.

9-65

9-66


f.

Select the Columns tab and that it has the 16 propagated columns.

g. Click

to close the DIFT SCD OrderFact Properties window.


Update the Properties of the Transformations 1. Accept the properties of the File Reader. 2. Accept the properties of the Splitter transformation. 3. Update the properties of the Sort transformations. a. Right-click on the first Sort Dedup transformation and select Properties. b. Select the Sort By Columns tab. c. Select Customer_ID from the Available columns and click

.

Select the Options tab. d. Select SAS Sort * in the left pane. e. In the Remove duplicate records area, select Remove rows with duplicate keys (NODUPKEY).

f.

Click

to close the Sort Dedup Properties window

g. Right-click on the second Sort Dedup transformation and select Properties. h. Select the Sort By Columns tab.

9-67

9-68


i.

Select Product_ID from the Available columns and click

j.

Select the Options tab.

.

k. Select SAS Sort * in the left pane. l.

In the Remove duplicate records area select Remove rows with duplicate keys (NODUPKEY).

m. Click

to close the Sort Dedup Properties window.

4. Update the properties of the first SCD Type 2 Loader transformation that will populate DIFT SCD Customer Dimension. Apply the Retained Key method. a. Right-click on the SCD Type 2 transformation (for DIFT SCD Customer Dimension table) and select Properties. b. Click the Change Tracking tab. Use beginning and end dates to track changes for each Customer ID and use a current record indicator to keep track of the current record for each Customer ID. 1) Check Use beginning and end dates. 2) Select BeginDateTimeCust as the Column Name for Beginning Date. 3) Select EndDateTimeCust as the Column Name for the End Date. 4) Check Use current indicator.


9-69

5) Select CurrRecCust as the Current indicator column.

SAS Data Integration Studio provides default expressions for the Change Tracking columns. The DATETIME function is used to generate the beginning datetime value. A datetime constant with a future date is used to specify an open ended value for the ending datetime. You can click in the Expression field to specify a custom expression.

Use Version Number and Use Current Indicator are provided as alternative methods to track the current record.

c. Click the Business Key tab. Specify Customer_ID as the business key in the Customer Dimension. 1) Click

. The DIFT SCD Customer Dimension Columns window is displayed.

2) Select Customer_ID.

9-70


3) Click

to close the DIFT SCD Customer Dimension Columns window.

d. Click the Generated Key tab. Specify GenKeyCust as the retained key column. 1) Select GenKeyCust as the column to contain the generated key values. 2) Check Generate retained key to implement the retained key method.

A default expression is provided to generate retained key values in the New record . field. To specify a custom expression, click


9-71

A macro variable “NewMaxKey” is automatically updated with the current maximum key value. To specify a custom method for obtaining the current maximum key value, click Define Max Key.

To implement the surrogate key method, uncheck Generate retained key. The surrogate key method provides default expressions for new and changed records.

e. Click the Detect Changes tab. Specify Customer_Name as the column on which changes are based. 1) Select Customer_Name and move it form Available columns to Selected columns.

If no columns are selected on the Detect Changes tab, then all columns are used to detect changes, except those used for Change Tracking, Business Key, Generated Key, and Type 1 columns.

9-72


f.

Click the Mappings tab. 1) that all target columns are mapped except GenKeyCust, BeginDateTimeCust, EndDateTimeCust, and CurrRecCust; these columns are populated by the transformation.

g. Click the Precode and Postcode tab. h. Check Postcode. i.

Enter the following PROC SORT step.

j.

Click

to close the SCD Type 2 Properties window.


5. Update the properties for the second SCD Type 2 Loader transformation that will populate DIFT SCD Product Dimension. a. Right-click on the SCD Type 2 Loader transformation (for DIFT SCD Product Dimension table) and select Properties. b. Click the Change Tracking tab. 1) that Use beginning and end dates is checked. 2) Select BeginDateTimeProd as the Column Name for Beginning Date. 3) Select EndDateTimeProd as the Column Name for the End Date. 4)

Check Use current indicator.

5) Select CurrRerod as the Current indicator column.

c. Click the Business Key tab. 1) Click

. The DIFT SCD Product Dimension Columns window is displayed.

2) Select Product_ID.

9-73

9-74


3) Click

to close the DIFT SCD Product Dimension Columns window.

d. Click the Generated Key tab. 1) Select GenKeyProd as the column to contain the generated key values. 2) Check Generate retained key to implement the retained key method. 3) Accept the default expression to generate key values in the New record field.


9-75

e. Click the Detect Changes tab. 1) Select Product_Name and move it form Available columns to Selected columns.

f.

Click the Mappings tab. 1) that all target columns are mapped except GenKeyProd, BeginDateTimeProd, EndDateTimeProd, and CurrRerod; these columns are populated by the transformation.

g. Click the Precode and Postcode tab.

9-76


h. Check Postcode. i.

Enter the following PROC SORT step:

j.

Click

to close the SCD Type 2 Properties window.

6. Select File Ö Save to save diagram and job metadata to this point. 7. Specify properties for the Lookup transformation that will populate DIFT SCD Order Fact. a. Right-click on the Lookup transformation and select Properties. b. Select the Lookups tab. Specify lookup mappings to the DIFT Customer Dimension table. 1) Select the row for DIFT SCD Customer Dimension and click Lookup Properties.

2) In the Lookup Properties - DIFT SCD Customer Dimension window, select the Source to Lookup Mapping tab. 3) Click on the Customer_ID column in the Source table pane to select it and click on the Customer_ID column in the Lookup table pane.


4) Click

9-77

to map the selected columns.

The Lookup transformation uses the Customer_Id value in an incoming transaction to do a lookup into the Customer Dimension table to find a matching customer id value. 5) In the Lookup Properties - DIFT SCD Customer Dimension window, select the Lookup to Target Mapping tab. 6) Click on the GenKeyCust column in the Lookup table pane to select it and click on the GenKeyCust column in Target table the pane. 7) Click


8) Click on the BeginDateTimeCust column in the Lookup table pane to select it, and click on the BeginDateTimeCust column in Target table the pane.

9-78


9) Click


The Lookup will retrieve the GenKeyCust and BeginDateTimeCust values from the Customer Dimension table and assign them to the GenKeyCust and BeginDateTimeCust columns in the target table. This links the transaction in the target table to the current record for the customer ID in the Customer Dimension table. 10) In the Lookup Properties - DIFT SCD Customer Dimension window, select the Where tab. 11) Click the Data Sources tab. 12) Expand the SCDCustDim table. 13) Select the CurrRecCust column and click Expression Text pane.

to insert the column reference in the


9-79

14) Add an = 1 to complete the expression.

15) In the Lookup Properties - DIFT SCD Customer Dimension window, select the Exceptions tab.

Note that several exceptions are defined by default. Two exceptions specify that the process will be aborted: 1. if the lookup table (Customer Dimension) is missing 2. if the lookup table does not have any records

9-80


Three exceptions take effect when a lookup value is not found: 1. a row is added to an error table 2. a row is added to an exception table 3. the target column will be assigned a missing value 16) Click window.

17) Click

to close the Lookup Properties - DIFT SCD Customer Dimension

to close the Warning.

c. Specify lookup mappings to the DIFT Product Dimension table. 1) In the Lookup Properties window, select the row for DIFT SCD Product Dimension and click Lookup Properties.

2) In the Lookup Properties - DIFT SCD Product Dimension window, select the Source to Lookup Mapping tab. 3) Click on the Product_ID column in the Source table pane to select it, and click on the Product_ID column in the Lookup table pane.


4) Click

9-81


The Lookup transformation will use the Product_Id value in an incoming transaction to do a lookup into the Product Dimension table to find a matching product id value. 5) In the Lookup Properties - DIFT SCD Product Dimension window, select the Lookup to Target Mapping tab. 6) Click on the GenKeyProd column in the Lookup table pane to select it, and click on the GenKeyProd column in Target table the pane. 7) Click


8) Click on the BeginDateTimeProd column in the Lookup table pane to select it, and click on the BeginDateTimeProd column in Target table the pane.

9-82


9) Click


The lookup will retrieve the GenKeyProd and BeginDateTimeProd values from the Customer Dimension table and assign them to the GenKeyProd and BeginDateTimeProd columns in the target table. This links the transaction in the target table to the current record for the customer id in the Product Dimension table. 10) In the Lookup Properties - DIFT SCD Product Dimension window, select the Where tab. 11) Click the Data Sources tab. 12) Expand the SCDProdDim table. 13) Select the CurrRerod column and click Expression Text pane.

to insert the column reference in the


9-83

14) Add an = 1 to complete the expression.

15) In the Lookup Properties - DIFT SCD Customer Dimension window, select the Exceptions tab.

Accept the default exceptions. The process is aborted if the lookup table (Customer Dimension) is missing or if it does not have any records. If a lookup value is not found a row is added to an error tables as well as to an exception table. Also a lookup value is not found the target column will be assigned a missing value.

9-84


16) Click window.

17) Click d.

to close the Lookup Properties - DIFT SCD Customer Dimension

to close the Warning.

In the Lookup Properties window, select the Errors tab. 1) Check Create error table and click

.


2)

9-85

All source columns and a generated column are selected by default. Remove all columns except Source Row Number, Order_ID, Customer_ID, and Product_ID from the Selected columns pane.

3) Click

to close the Choose Error Table Columns window

4) Check Create exception table and click

.

9-86


5) Four generated columns and no source columns are selected by default. Accept the default column selection.

6) Click

to close the Choose Exception Table Columns window


e. Select the mappings tab and review the mappings.

These mappings for the Lookup transformation were established in an earlier step. f.

Click

to close the Lookup Properties window.

8. Select File Ö Save to save diagram and job metadata to this point. 9. Specify properties for the Table Loader transformation that will populate DIFT SCD Order Fact. a. Right-click on the Table Loader transformation and select Properties. b. Select the Load Technique tab.

9-87

9-88


c. Select Append to Existing as the Load Style.

d. Click



Update the Processing Order, Run the Job, and view the Results 1. If necessary, select View Ö Detail to open the Details . It opens below the Diagram Editor. 2. In the Details , select the Control Flow tab. a. Use the

and

arrows to arrange the transformations in the following order:


b. the order of processing on the Diagram tab in the Job Editor window.

The number in the upper-left corner of the transformation indicates the order of processing. 3. Run the job. a. Select the Status tab in the Details pane to monitor the execution of the job. b. Click

to run the job.

c. that all the steps completed successfully. d. View the Log for the executed Job.

9-89

9-90


4. Enable row count. a. Select Tools – Options to open the Options window. b. Select the General tab c. Check Enable row count on basic properties for tables.

d. Select Ok to close the options window. 5. View the data. a. Select the Diagram tab in the Job Editor window. b. Right-click on the DIFT SCD Order Fact table and select Open. The data is displayed in the View Table window. Scroll to see the right-most columns:

c. Close the View Data window. 6. Run the job for month 2. a. On the Diagram tab in the Job Editor window, right-click on the DIFT SCD Source Data and select Properties. b. Select the File Location tab. c. Change the value of File Name to S:\workshop\dift\data\OrionSourceDataM02.csv. d. Click

to close the DIFT SCD Source Data Properties window.

e. Select the Status tab in the Details pane to monitor the execution of the job. f.

Click

to run the job.

g. that all the steps completed successfully. h. View the Log for the executed Job. i.

View the data.

7. Select the Diagram tab in the Job Editor window.


9-91

a. Right-click on the DIFT SCD Order Fact table and select Open. The data is displayed in the View Table window. Scroll to see the right-most columns:

Look for following updates to data: Customer_Name changes: Ines Deisser (Customer_ID = 65)

changed to

Ines Muller

b. Close the View Data window. 8. Run the job for month 3. a. On the Diagram tab in the Job Editor window, right-click on the DIFT SCD Source Data and select Properties. b. Select the File Location tab. c. Change the value of File Name to S:\workshop\dift\data\OrionSourceDataM03.csv. d. Click

to close the DIFT SCD Source Data Properties window.

e. Select the Status tab in the Details pane to monitor the execution of the job. f.

Click

to run the job.

g. that all the steps completed successfully. h. View the Log for the executed Job. 9. View the data. a. Select the Diagram tab in the Job Editor window.

9-92


b. Right-click on the DIFT SCD Order Fact table and select Open. The data is displayed in the View Table window. Scroll to see the right- most columns:

Look for following updates to data: Customer_Name changes: Ines Muller (Customer_ID = 65) c. Close the View Data window. 10. Save and close the job.

changed to

Ines Deisser

9.3 Introducing the Change Data Capture Transformations


Objectives „ „ „

Define Change data capture. List the types of CDC transformations. List functions of the CDC transformations.

56

Change Data Capture Defined Change data capture (CDC) is a process that shortens the time required to load data from a relational database. The process is efficient because the source is a changed data table, rather than the entire base table. The CDC transformations in SAS Data Integration Studio can be used to load dimension tables in a star schema, as part of an implementation of slowly changing dimensions.

57

9-93

9-94


CDC Transformations SAS Data Integration Studio provides four CDC transformations: Q Attunity CDC Q DB2 CDC Q Oracle CDC Q General CDC. The Attunity, DB2, and Oracle transformations work directly with changed data tables that are in native database format. The General transformation can be used to load change data from other vendors and custom applications.

58

The separately licensed Attunity software enables you to generate source change tables from a variety of relational databases running in a variety of operational environments.

CDC Transformation Requirements The database must „ generate the change data table. The rows in the change data table include the data involved in the changes The CDC transformation „ uses the change data table as a source table „ can optionally use a control table to prevent reprocessing of change data that was loaded in an earlier run. The control table is not needed if processed rows are discarded from the change data table. 59


Prerequisites for Change Data Capture The CDC transformations require the following software: Attunity CDC

Attunity is a data integration product in which the Attunity Stream software enables connectivity between databases and across operating environments. The Attunity CDC transformation has been validated on Attunity AIS 5.0 with Attunity Stream. To use the Attunity software you need to license SAS/ACCESS Interface to ODBC.

continued...

60

Prerequisites (continued)

61

Oracle CDC

The Oracle CDC transformation has been validated on Oracle 10G with asynchronous CDC. The transformation requires that you license SAS/ACCESS to Oracle.

DB2 CDC

The DB2 CDC transformation has been validated on DB2/UDB, release 8.1, fixpak 3. The transformation requires that you license SAS/ACCESS to DB2.

General CDC

The General CDC transformation has no prerequisites.

9-95

9-96


Using CDC Transformations The CDC transformation „ implements change data capture „ reads data from a source change table that contains a record of inserts, updates, and deletes since a specified date and time „ can optionally use a control table to direct the loading process „ can generate end date/time values in targets that are used to implement type 2 slowly changing dimensions. The end date/time values are used to close-out target records in response to delete transactions in the source change table.

62

CDC Control Tables A control table can be used to prevent the update of target rows that were processed in an earlier run. In a job with a control table, the CDC transformation first finds in the source the most recent insert, update, or delete action for a given unique identifier (business key). The most recent source row is then compared to the prior actions that appear in the control table. If the unique identifiers match, and if the rest of the rows are identical, then the source row is a duplicate and it is not added to the target.

63


CDC Control Tables – Additional Points Some additional points to note about the CDC control tables: Q Control tables are optional, so you need to use one only if the source changed data table contains information that was already loaded into the target. Q The control table can be in SAS format or in native database format. Q Column definitions in the control table are similar to those that are required in the source changed data tables. Q You can use the New Table Wizard to create control tables.

64

CDC Control Table Structure In control tables, the names and order of the following columns can vary, because you identify those columns in the properties window of the CDC transformation: Application identifies the application that compares the Name source change data records to the records in the target to test for previous updates. A typical value for this column is SAS Data Integration Studio. The column type is character and the length is 64. Table Name identifies the source changed data table. The column type is character and the length is 64.

65

continued...

9-97

9-98


Context

provides the unique identifiers in the target that are not to be overwritten. The context is a character value with length of 32 for DB2, Attunity, and General. Oracle context is numeric with a length of 8.

Rows Processed

records the number of source changed data records that were processed the last time that the job was run. This value is updated at the end of the job run, as an output from the CDC transformation. The type of this column is numeric and the length is 8.

Timestamp

identifies the time and date when the job was run, in DATETIME16.6 format. The type of this column is numeric and the length is 8.

66

Chapter 10 Defining Generated Transformations 10.1 SAS Code Transformations ......................................................................................... 10-3 10.2 Using the New Transformation Wizard ....................................................................... 10-8 Demonstration: Creating a Generated Transformation ...................................................... 10-11 Exercises ............................................................................................................................ 10-35

10.3 Solutions to Exercises ............................................................................................... 10-40

10-2

Chapter 10 Defining Generated Transformations

10.1 SAS Code Transformations



Define SAS code transformation templates. Explain the prompting framework. Describe the different types of prompts that make up the prompting framework.

3

Transformation Templates The Process Library tree contains two kinds of transformation templates:

4

Java Plug-In Transformation Templates

Created with the Java programming language

SAS Code Transformation Templates

Created with the Transformation Generator wizard

10-3

10-4


SAS Code Transformations Right-clicking on a SAS code transformation in the Transformations tree yields a menu with several unique options: Menu for SAS Code Transformation „ Properties „ Analyze „ Export Menu for Java Plug-In Transformation

5

SAS Code Transformations SAS code transformations in the Transformations tree can be identified by the icon associated with the transformation:

6


SAS Code Transformation Templates One of the easiest ways to customize SAS Data Integration Studio is to write your own SAS code transformation templates. SAS code transformation templates use the SAS language to accomplish the following: „ extract data „ transform data „ create reports „ load data into data stores Unlike Java-based plug-ins that require software development, SAS code transformation templates are created with a wizard. 7

New Transformation Wizard The New Transformation wizard has the following characteristics: „ guides you through the steps of specifying SAS code for a transformation template and saving the template to the current metadata repository „ is used to create custom SAS code transformation templates „ enables you to enter the SAS code that runs when the template is executed as part of a job After the template is saved, it is displayed in the Transformations tree, where the template is available for use in any job.

8

10-5

10-6


SAS Code and Macro Variables The SAS code for a transformation template typically includes macro variables. options &options; title "&title"; proc gchart data=&syslast; vbar &classvar1 / sumvar=&analysisvar group=&classvar2; run;quit;

9

Macro Variable Values When the transformation template is used in a job, the person specifying the properties of the transformation must specify the values of the macro variables.

&classvar1

&classvar2

&analysisvar &title 10


Generated SAS Code SAS Data Integration Studio generates a corresponding %LET statement(s) that creates the macro variables and assigns the values. %let %let %let %let %let %let

syslast=yy.xx; options=; classvar1=Customer_Age_Group; classvar2=Customer_Gender; analysisvar=Quantity; title=Sum of Quantity across Gender and Age Group;

11

Options Window The New Transformation wizard’s Options window is the facility for creating the options to be used for the new transformation.

12

10-7

10-8


10.2 Using the New Transformation Wizard

Objectives „

Create a custom transformation.

15

New Transformation – General Settings General settings such as name and description are specified in the initial window of the New Transformation wizard.

16


New Transformation – SAS Code The SAS Code window allows for direct entry of SAS code. Typically, the SAS code is developed elsewhere, copied, and then pasted in to this window.

17

New Transformation – Inputs/Outputs The Inputs and Outputs area of this window will determine if the new transformation is to accept inputs (and up to how many). Similar choices are made for outputs.

18

10-9

10-10


New Transformation - Finish The review window of the wizard is used for checking and ing various settings for the new transformation. Finishing this window creates the new transformation.

19


10-11

Creating a Generated Transformation

This demonstration creates a report on customer order information. The HTML report must have a textbased output with summary statistics as well as a bar chart graphic. A transformation with this type of output does not currently exist. Hence, a new SAS code transformation is created and then used in a job. 1. If necessary, access SAS Data Integration Studio using Barbara’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. Select Barbara’s Work Repository as the connection profile. c. Click




2. Specify metadata for the new transformation. a. Select the Folders tab. b. Expand Data Mart Development Ö Reports. c. that the Reports folder is selected. d. Select File Ö New Ö Transformation…. e. Type Summary Table and Vertical Bar Chart as the value for the Name field. f.

that the location is set to /Data Mart Development/Reports.

g. Type Defined as the value for the Transformation Category field.

10-12


The General information for the new transformation should resemble the following:

h. Click i.

.

Add the necessary SAS code. 1) Access Windows Explorer. 2) Navigate to S:\Workshop\dift\. 3) Right-click on TabulateGraph.sas and select Open With Ö Notepad…. 4) Right-click in the background of the Notepad window and select Select All. 5) Right-click in the background of the Notepad window and select Copy. 6) Access the New Transformation wizard. 7) Right-click in the background of the SAS Code pane and select Paste.

j.

Click

.


10-13

The SAS code used is the following: %let path=S:\Workshop\dift\reports; %macro TabulateGChart; options mprint; %if (%quote(&options) ne) %then %do; options &options; %end; %if (%quote(&path) ne) %then %do; ods html path="&path" gpath="&path" %end; %if (%quote(&file) ne) %then %do; file="&file..html" ; %end; %if (%quote(&tabulatetitle) ne) %then %do; title1 "&tabulatetitle"; %end; proc tabulate data=&syslast; class &classvar1 &classvar2; var &analysisvar; table &classvar1*&classvar2, &analysisvar*(min="Minimum"*f=comma7.2 mean="Average"*f=comma8.2 sum="Total"*f=comma14.2 max="Maximum"*f=comma10.2); run; %if (%quote(&gcharttitle) ne) %then %do; title "&gcharttitle"; %end; GOPTIONS DEV=PNG; PROC GCHART DATA=&syslast; VBAR &classvar1 / SUMVAR=&analysisvar GROUP=&classvar2 CLIPREF FRAME TYPE=SUM OUTSIDE=SUM COUTLINE=BLACK; RUN; QUIT; ods html close; %mend TabulateGChart; %TabulateGChart;

10-14


k. Click

.

The Options window is available to define the options to be used in this transformation.


l.

Define metadata for three groups. 1) Click

.

2) Type Data Items as the value for the Displayed text field.

3) Click

. .

4) Click

5) Type Titles as the value for the Displayed text field. 6) Click

. .

7) Click

8) Type Other Options as the value for the Displayed text field. 9) Click

.

10-15

10-16


The three groups should resemble the following:

m. Define metadata for the options in the Data Items group. 1) Define metadata for the first classification variable. a) Select the Data Items group. b) Click

.

c) Type classvar1 as the value for the Name field. d) Type Column to Chart for Vertical Bar Chart as the value for the Displayed text field. e) Type The column selected for this option will be the charting column for GCHART and a classification column in the row dimension for TABULATE. as the value for the Description field. f) Click Requires a non-blank value in the Options area.


10-17

g) Select the Prompt Type and Values tab. h) Select Data source column as the value for the Prompt type field. i)

that Select from source is selected in the Columns to select from area.

j)

that all data types are selected.

k) Click Limit number of selectable columns. l)

Type 1 as the value for the Minimum field.

m) Type 1 as the value for the Maximum field.

n) Click

.

2) Define metadata for the second classification variable. a) Select the Data Items group. b) Click

.

c) Type classvar1 as the value for the Name field. d) Type Column for Grouping Charting Variable as the value for the Displayed text field.

10-18


e) Type The column selected for this option will be the grouping column for GCHART and a classification column in the row dimension for TABULATE. as the value for the Description field. f) Click Requires a non-blank value in the Options area. g) Select the Prompt Type and Values tab. h) Select Data source column as the value for the Prompt type field. i)


j)

that all data types are selected.



m) Type 1 as the value for the Maximum field. n) Click

.

3) Define metadata for the analysis variable. a) Select the Data Items group. b) Click

.

c) Type analysisvar as the value for the Name field. d) Type Column to Analyze for Vertical Bar Chart as the value for the Displayed text field. e) Type The column selected for this option will determine the heights of the bars for GCHART and an analysis column for TABULATE. as the value for the Description field. f) Click Requires a non-blank value in the Options area. g) Select the Prompt Type and Values tab. h) Select Data source column as the value for the Prompt type field. i)


j)

Clear Character in the Data types area.



m) Type 1 as the value for the Maximum field. n) Click

.


10-19

The three options in the Data Items group should resemble the following:

n. Define metadata for the options in the Titles group. 1) Define metadata for the title to be used with TABULATE output. a) Select the Titles group. b) Click

.

c) Type tabulatetitle as the value for the Name field. d) Type Title for Table Report as the value for the Displayed text field. e) Type Specify some text that will be used as the title for the TABULATE output. as the value for the Description field. f) Select the Prompt Type and Values tab. g) that Text is specified as the value for the Prompt type field. h) Accept the defaults for the rest of the fields. i)

Click

.

2) Define metadata for the title to be used with GCHART output. a) Select the Titles group. b) Click

.

c) Type gcharttitle as the value for the Name field. d) Type Title for Graph Report as the value for the Displayed text field. e) Type Specify some text that will be used as the title for the GCHART output. as the value for the Description field. f) Select the Prompt Type and Values tab. g) that Text is specified as the value for the Prompt type field. h) Accept the defaults for the rest of the fields. i)

Click

.

10-20


The two options in the Titles group should resemble the following:

o. Define metadata for the options in the Other Options group. 1) Define metadata for SAS system options a) Select the Other Options group. b) Click

.

c) Type options as the value for the Name field. d) Type Specify SAS system options as the value for the Displayed text field. e) Type Specify a space separated list of global SAS system options. as the value for the Description field. f) Select the Prompt Type and Values tab. g) that Text is specified as the value for the Prompt type field. h) Accept the defaults for the rest of the fields. i)

Click

.

2) Define metadata for the name of the HTML file to be created. a) Select the Other Options group. b) Click

.

c) Type file as the value for the Name field. d) Type Name of HTML file to be created as the value for the Displayed text field. e) Type Enter the name of the HTML file that will contain the reports generated by this transformation. Do NOT enter the HTML file extension! as the value for the Description field.


f) Select the Prompt Type and Values tab. g) that Text is specified as the value for the Prompt type field. h) Accept the defaults for the rest of the fields. i)

Click

.

The two options in the Other Options group should resemble the following:

10-21

10-22


p. Test the prompts. 1) Click

. The Test the Prompts window opens as follows:

that the three items in the Data Items group are all required (note the *). The descriptions entered for each of the parameters are displayed. Clicking opens a dialog window to navigate the SAS Folders to a data source from which a column can be selected.


10-23

2) Click Titles in the selection pane. The two options in the Titles group are displayed:

3) Click Other Options in the selection pane. The two options in the Other Options group are displayed:

10-24


4) Click q. Click

to close the Test the Prompts window. .

r. Clear Transform s outputs in the Outputs area.

The Inputs group box values add a specified number of inputs to the transformation when it is used in a job. If you later update the transformation to increase this minimum number of inputs value, any jobs that have been submitted and saved use the original value. The increased minimum number of inputs is enforced only for subsequent jobs. Therefore, you can increase the minimum number of inputs without breaking existing jobs. The Maximum number of inputs field is used to allow you to connect additional inputs into the input port. For example, a setting of 3 allows you to have up to three inputs. The rules for inputs also apply to outputs.


s. Click

.

t. Click

.

10-25

2. Create a job to use the new transformation. a. Select the Folders tab. b. Expand Data Mart Development Ö Jobs. c. that the Jobs folder is selected. d. Select File Ö New Ö Job…. The New Job window opens. e. Type Report and Graphic for Customer Orders as the value for the Name field. f.

that the Location is set to Data Mart Development Ö Jobs.

g. Click


10-26


3. Add source table metadata to the diagram for the process flow. a. Select the Folders tab. b. Navigate to the Data Mart Development Ö Data folder. c. Drag the DIFT Customer Order Information table object to the Diagram tab of the Job Editor. 4. Add the Summary Table and Vertical Bar Chart transformation to the process flow. a. Select the Checkouts tab. b. Drag the Summary Table and Vertical Bar Chart transformation to the Diagram tab of the Job Editor. 5. Connect the DIFT Customer Order Information table object to the Summary Table and Vertical Bar Chart transformation.



10-27

7. Specify properties for the Summary Table and Vertical Bar Chart transformation. a. Right-click on the Summary Table and Vertical Bar Chart transformation and select Properties. b. Select the Options tab. c. that Data Items is selected in the selection pane.

for the Column to Chart for Vertical Bar Chart option.

1) Click

2) Select Customer Age Group in the Select a Data Source Item window.

3) Click

.

4) Click

for the Column for Grouping Chart Variable option.

5) Select Customer Gender in the Select a Data Source Item window. 6) Click

.

10-28


7) Click option.

for the Column to Analyze for Vertical Bar Chart

8) Select Quantity Ordered in the Select a Data Source Item window. 9) Click

.


10-29

d. that Titles is selected in the selection pane.

1) Type Analyzing &analysisvar across &classvar1 and &classvar2 in the Title for Table Report field. 2) Type Sum of &analysisvar for &classvar1 grouped by &classvar2 in the Title for Graph Report field.

10-30


e. that Other Options is selected in the selection pane.

1) Type NODATE NONUMBER LS=80 as the value for the Specify SAS system options field. 2) Type CustomerOrderReport as the value for the Name of HTML file to be created field.

f.

Click

.



9. Run the job. a. Right-click in background of the job and select Run. b. Select the Status tab in the Details area.

c. Double-click the Warning for the content of the Warning.

10-31

10-32


10. View the generated HTML file. a. Access the Windows Explorer. b. Navigate to the S:\Workshop\dift\reports folder. c. Double-click CustomerOrderReport.html. An information window opens like the following:

d. Click

.

e. In the Microsoft Internet Explorer window, click

The report is displayed as follows:

to close the information bar.


Scroll to view the graphic:

11. Select File Ö Close to close the browser window.

10-33

10-34


12. Check in the objects. a. Select the Checkouts tab.

b. Select Check Outs Ö Check In All. c. Type Adding transformation and job using the new transformation as the value for the Title field. d. Click


e. Click


f.


Click

g. Select the Transformations tab. h. Expand the Defined folder. Note the transformation.

After the transformation is checked in, it can be updated only by an .


10-35

Exercises

1. Creating a Forecast Transformation A transformation needs to be generated that uses PROC FORECAST to create a data set containing the forecast information. A PROC GPLOT step can then be used to plot this information. • The code for the transformation is found in S:\Workshop\dift\ForecastGraph.sas. • Name the transformation Forecast as Graphic and store it in the Defined folder in the Transformations tree. • Create three groups of options: Data Items Forecast Options Titles and Other Options • Add the following options for the Data Items group: Name

Idvariable

Displayed text:

ID Variable

Description:

The column used to identify obs in input and output data sets. Its values are interpreted and extrapolated according to the values of the INTERVAL= option.

Required

Yes

Prompt type:

Data Source Column

Other information:

Do not allow character columns; only allow 1 column as a selection.

Name

Fcastvariable

Displayed text:

Column to Forecast

Description:

The column from the input data set that is to be forecasted.

Required

Yes

Prompt type:

Data Source Column

Other information:

Do not allow character columns; only allow one column as a selection.

10-36


• Add the following options for the Forecast Options group: Name

Alpha

Displayed text:

Significance Level

Description:

Specify significance level for confidence intervals (default is .05).

Required

No

Prompt type:

Numeric

Other information:

Allow values other than integers; provide a default value of .05

Name

Lead

Displayed text:

Number of Periods to Forecast

Description:

Specify the number of periods ahead to forecast (default is 6).

Required

No

Prompt type:

Numeric

Other information:

Provide a default value of 6.

Name

Method

Displayed text:

Method to Model the Series

Description:

Specify the method to use to model the series and generate the forecasts. (Default is STEPAR)

Required

No

Prompt type:

Text

Other information:

Provide a list of values of STEPAR, EXPO, WINTERS, ADDWINTERS. Set STEPAR as the default.


10-37

• Add the following options for the Titles and Other Options group: Name

Title

Displayed text:

Title for Forecast Graphic

Description:

Specify text that will be used as the title for the FORECAST output.

Required

No

Prompt type:

Text

Name

Options

Displayed text:

Specify SAS system options

Description:

Specify a space separated list of global SAS system options.

Required

No

Prompt type:

Text

Name

File

Displayed text:

Name of HTML file to be created

Description:

Enter the name of the HTML file that will contain the report generated by this transformation. Do NOT enter the HTML file extension!

Required

No

Prompt type:

Text

• Be sure to test the prompts after they are built. • The transformation should not produce an output table.

10-38


2. Using the Forecast Transformation in a Job Create a job that reads the DIFT Profit Information external file object, subsets the data for Orion Australia, and generates an HTML file named OrionAustraliaProfitInfo.html using the Forecast as Graphic transformation. The process flow should resemble the following:

Use the YYMM column as the ID column and the Profit column as the column to forecast. View the HTML file. The output should resemble the following:

3. Check In Objects Check in the transformation and job objects.


10-39

4. Create Forecast for all Country Companies (Optional) Create a job that uses the Loop transformations to create a series of HTML files, one for each distinct company value. However, only include those company values that contain “Orion”. The control table for the loop transformations can be created as part of the job, or as a separate job. The following diagram shows the creation of the control table as part of the job:

The inner job should be a copy of the job from Exercise 3, but modified with parameters.

10-40


10.3 Solutions to Exercises 1. Creating a Forecast Transformation a. Select the Folders tab. b. Expand Data Mart Development Ö Reports. c. that the Reports folder is selected. d. Select File Ö New Ö Transformation…. e. Enter the General information for the new transformation. 1) Type Forecast as Graphic as the value for the Name field. 2) that the location is set to Data Mart Development Ö Reports. 3) Type Defined as the value for the Transformation Category field. 4) Click

.

f. Add the necessary SAS code. 1) Access Windows Explorer. 2) Navigate to S:\Workshop\dift\. 3) Right-click on ForecastGraph.sas and select Open With Ö Notepad…. 4) Right-click in the background of the Notepad window and select Select All. 5) Right-click in the background of the Notepad window and select Copy. 6) Access the New Transformation wizard. 7) Right-click in the background of the SAS Code pane and select Paste. 8) Click

.

The SAS code used is the following: %let path=S:\Workshop\dift\reports; %macro ForeCastGraph; options mprint; %if (%quote(&options) ne) %then %do; options &options; %end; proc forecast data=&syslast alpha=&alpha interval=month lead=&lead method=&method out=fcast outall;


%if (%quote(&idvariable) ne) %then %do; id &idvariable; %end; %if (%quote(&fcastvariable) ne) %then %do; var &fcastvariable; %end; run; symbol1 symbol2 symbol3 symbol4 symbol5 symbol6 legend1

I=spline c=green; I=spline c=blue l=3; I=spline c=black; I=spline c=red; I=spline c=red; I=spline c=black; down=2 across=3 label=('Legend:') position=(bottom center outside) ; goptions dev=png; %if (%quote(&file) ne) %then %do; ods html path="S:\Workshop\dift\reports" gpath="S:\Workshop\dift\reports" file="&file..html"; %end; %if (%quote(&title) ne) %then %do; Title “&title”; %end; proc gplot data=fcast; plot &fcastvariable * &idvariable = _type_ / legend=legend1 autovref href='01jan2003'd ; run; quit; ods html close; %mend ForeCastGraph; %ForeCastGraph; g. Click

.

h. Define metadata for three groups. .

1) Click

2) Type Data Items as the value for the Displayed text field. 3) Click 4) Click

. .

5) Type Forecast Options as the value for the Displayed text field.

10-41

10-42


6) Click

. .

7) Click

8) Type Titles and Other Options as the value for the Displayed text field. 9) Click

.

The three groups should resemble the following:

i. Define metadata for the options in the Data Items group. 1) Define metadata for the first variable. a) Select the Data Items group. b) Click

.

c) Type idvariable as the value for the Name field. d) Type ID Variable as the value for the Displayed text field. e) Type The column used to identify obs in input & output data sets. Its values are interpreted and extrapolated according to the values of the INTERVAL= option. as the value for the Description field. f) Click Requires a non-blank value in the Options area. g) Select the Prompt Type and Values tab. h) Select Data source column as the value for the Prompt type field. i) that Select from source is selected in the Columns to select from area. j) Clear Chacacter in the Data Types area. k) Click Limit number of selectable columns. l) Type 1 as the value for the Minimum field. m) Type 1 as the value for the Maximum field. n) Click

.

2) Define metadata for the second variable. a) Select the Data Items group.


b) Click

10-43

.

c) Type fcastvariable as the value for the Name field. d) Type Column to Forecast as the value for the Displayed text field. e) Type The column from the input data set that is to be forecasted. as the value for the Description field. f) Click Requires a non-blank value in the Options area. g) Select the Prompt Type and Values tab. h) Select Data source column as the value for the Prompt type field. i) that Select from source is selected in the Columns to select from area. j) Clear Chacacter in the Data Types area. k) Click Limit number of selectable columns. l) Type 1 as the value for the Minimum field. m) Type 1 as the value for the Maximum field. n) Click

.

The Data Items group has the following items defined:

j. Define metadata for the options in the Forecast Options group. 1) Define metadata for the significance level options. a) Select the Forecast Options group. b) Click

.

c) Type alpha as the value for the Name field. d) Type Significance Level as the value for the Displayed text field. e) Type Specify significance level for confidence intervals (default is .05). as the value for the Description field. f) Select the Prompt Type and Values tab. g) Select Numeric is specified as the value for the Prompt type field.

10-44


h) Clear Allow only integer values. i) Type .05 as the value for the Default value field. j) Click

.

2) Define metadata for the number of periods to forecast option. a) Select the Forecast Options group. b) Click

.

c) Type lead as the value for the Name field. d) Type Number of Periods to Forecast as the value for the Displayed text field. e) Type Specify the number of periods ahead to forecast (default is 6). as the value for the Description field. f) Select the Prompt Type and Values tab. g) Select Numeric is specified as the value for the Prompt type field. h) Type 6 as the value for the Default value field. i) Click

.

3) Define metadata for the method to model the series forecast option. a) Select the Forecast Options group. b) Click

.

c) Type method as the value for the Name field. d) Type Method to Model the Series as the value for the Displayed text field. e) Type Specify the method to use to model the series and generate the forecasts. (Default is STEPAR) as the value for the Description field. f) Select the Prompt Type and Values tab. g) Accept Text as the value for the Prompt type field. h) Select selects values from a static list as the value for the Method for populating prompt field. i) Click

in the List of values area.

j) Type STEPAR as the Unformatted Value. k) Click



10-45

l) Type EXPO as the Unformatted Value. m) Click


n) Type WINTERS as the Unformatted Value. in the List of values area.

o) Click

p) Type ADDWINTERS as the Unformatted Value. q) Click

under Default for STEPAR.

r) Type 6 as the value for the Default value field. s) Click

.

The Forecast Options group has the following items defined:

k. Define metadata for the options in the Titles and Other Options group. 1) Define metadata for the title to be used with forecast graphic. a) Select the Titles and Other Options group. b) Click

.

c) Type title as the value for the Name field. d) Type Title for Forecast Graphic as the value for the Displayed text field. e) Type Specify text that will be used as the title for the FORECAST output. as the value for the Description field. f) Select the Prompt Type and Values tab. g) that Text is specified as the value for the Prompt type field. h) Accept the defaults for the rest of the fields. i) Click

.

2) Define metadata for SAS system options

10-46


a) Select the Title and Other Options group. b) Click

.

c) Type options as the value for the Name field. d) Type Specify SAS system options as the value for the Displayed text field. e) Type Specify a space separated list of global SAS system options. as the value for the Description field. f) Select the Prompt Type and Values tab. g) that Text is specified as the value for the Prompt type field. h) Accept the defaults for the rest of the fields. i) Click

.

3) Define metadata for the name of the HTML file to be created. a) Select the Title and Other Options group. b) Click

.

c) Type file as the value for the Name field. d) Type Name of HTML file to be created as the value for the Displayed text field. e) Type Enter the name of the HTML file that will contain the report generated by this transformation. Do NOT enter the HTML file extension! as the value for the Description field. f) Click Requires a non-blank value in the Options area. g) Select the Prompt Type and Values tab. h) that Text is specified as the value for the Prompt type field. i) Accept the defaults for the rest of the fields. j) Click

.

The two options in the Other Options group should resemble the following:


l. Test the prompts. 1) Click

. The Test the Prompts window opens as follows:

10-47

10-48


2) Click Forecast Options in the selection pane. The three options in the Forecast Options group are displayed. 3) that all fields have default values and that four values are available for selection for the Method to Model the Series.


10-49

4) Click Titles and Other Options in the selection pane. The three options in the Titles and Other Options group are displayed. 5) that Name of HTML file to be created is a required field.

6) Click

to close the Test the Prompts window. .

m. Click

n. Clear Transform s outputs in the Outputs area. o. Click

.

p. Click

.

2. Using the Forecast Transformation in a Job a. Define the initial job metadata. 1) Select the Folders tab. 2) Expand Data Mart Development Ö Jobs. 3) that the Jobs folder is selected. 4) Select File Ö New Ö Job…. The New Job window opens. 5) Type Create Profit Forecast for Orion Australia as the value for the Name field.

10-50


6) that the Location is set to Data Mart Development Ö Jobs. 7) Click


b. Add source metadata to the diagram for the process flow. 1) Select the Folders tab. 2) Navigate to the Data Mart Development Ö Data folder. 3) Drag the DIFT Profit Information table object to the Diagram tab of the Job Editor. c. Add the File Reader transformation to the process flow. 1) Select the Transformation tab. 2) Expand the Access folder and locate the File Reader transformation template. 3) Drag the File Reader transformation to the Diagram tab of the Job Editor. d. Connect the DIFT Profit Information external file object to the File Reader transformation. e. Add the Extract transformation to the process flow. 1) Select the Transformation tab. 2) Expand the Data folder and locate the Extract transformation template. 3) Drag the Extract transformation to the Diagram tab of the Job Editor. f. Add the Forecast as Graphic transformation to the process flow. 1) Select the Checkouts tab. 2) Drag the Forecast as Graphic transformation to the Diagram tab of the Job Editor. g. Connect the Extract transformation to the Forecast as Graphic transformation.

h. Select File Ö Save to save diagram and job metadata to this point. i. Specify properties for the Extract transformation. 1) Right-click on the Extract transformation and select Properties. 2) Select the Where tab. 3) Type Company = “Orion Australia” as the value for the Expression Text area. 4) Click


j. Select File Ö Save to save diagram and job metadata to this point.


10-51

k. Specify properties for the Forecast as Graphic transformation. 1) Right-click on Forecast as Graphic transformation and select Properties. 2) Select the Options tab. 3) Select the Data Items in the selection pane. 4) Select YYMM as the value for the ID Variable. 5) Select Profit as the value for the Column to Forecast. 6) Select Titles and Other Options in the selection pane. 7) Type Profit Forecast for Orion Australia as the value for Title for Forecast Graphic. 8) Type OrionAustraliaProfitInfo as the value for Name of HTML file to be created. 9) Click


l. Run the job. 1) Right-click in background of the job and select Run. 2) Select the Status tab in the Details area. Note that all processes completed successfully. 3) Click


4) View the Log for the executed Job. 5) Access the Windows Explorer. 6) Navigate to S:\Workshop\dift\reports. 7) Double-click OrionAustraliaProfitInfo.html to open the file in a browser. 8) Select File Ö Close to close the browser. 9) Select File Ö Close to close the job editor. 3. Check In Objects a. Select the Checkouts tab. b. Select Check Outs Ö Check In All. c. Type Adding objects that use SORT, RANK, TRANSPOSE and LIST DATA as the value for the Title field. d. Click


e. Click


f. Click


4. Create Forecast for all Country Companies (Optional)

10-52


Chapter 11 Implementing Data Quality Techniques (Self-Study) 11.1 SAS and Data Quality ................................................................................................... 11-3 11.2 Working with the DataFlux IS Transformations ....................................................... 11-10 Demonstration: Confirming DataFlux Integration Server Is Running ................................. 11-15 Demonstration: Configuring the DataFlux Integration Server Manager ............................. 11-18 Demonstration: Creating Jobs for Execution on DataFlux Integration Server ................... 11-20 Demonstration: Creating a Job to be Used as a Service ................................................... 11-63 Demonstration: ing Jobs and Services to DataFlux Integration Server.................. 11-69 Demonstration: ing DataFlux Integration Server in SAS Management Console .................................................................................................... 11-76 Demonstration: Using the DataFlux IS Job Transformation............................................... 11-87 Demonstration: Using the DataFlux IS Service Transformation ...................................... 11-101 Exercises .......................................................................................................................... 11-120

11-2

Chapter 11 Implementing Data Quality Techniques (Self-Study)

11.1 SAS and Data Quality


Objectives „ „

Define data quality. Discuss data quality offerings from SAS.

3

What Is Data Quality? Poor quality information is detrimental to any organization. One of the critical foundations for the effective warehousing and mining of volumes of data is data quality. If data is of insufficient quality, then the knowledge workers who query the data warehouse and the decision makers who receive the information cannot trust the results.

4

11-3

11-4


Cleansing the Data Data can be cleansed and/or validated to reduce the volume and increase the accuracy of the data sent through an ETL flow.

Cleansing data will result in accurate reports

5

When to Cleanse the Data The least expensive time to deal with data quality issues is at the point of data capture. If rules and techniques can be employed to keep the data from being stored incorrectly, the ETL developer or business does not have to deal with data quality issues downstream. If data quality cannot be applied at the point of data capture, it is best to correct the data issues as soon as possible, and in the ETL flow is a valid option.

6


SAS Data Integration Studio and Data Quality SAS Data Integration Studio has a number of features that can help improve the quality of data in a data warehouse: „ Data Validation transformation „ Apply Lookup Standardization transformation „ Create Match Code transformation „ DataFlux IS Job transformation „ DataFlux IS Service transformation „ SAS Data Quality Server functions in the Expression Builder With the exception of the Data Validation transformation, these features require the licensing of SAS Data Quality Server software. ...

7

The Data Validation transformation is covered in a previous chapter.

SAS Data Quality Server The SAS Data Quality Server product enables s to analyze, transform, standardize, and de-duplicate data from within the SAS programming environment. The procedures are surfaced in SAS Data Integration Studio through the Data Quality transformations.

8

11-5

11-6


Functional Overview of the SAS Data Quality Server Software SAS Data Quality Server Local Process PROC DQSCHEME PROC DQMATCH

DataFlux Products

read

18 functions

Quality Knowledge Base

dfPower Studio

read

Server Process PROC DQSRV PROC DQSRVSVC 8 functions

run

Integration Server

Profile jobs Architect jobs

9

The language elements in the SAS Data Quality Server software can be separated into two functional groups. As shown in the previous diagram, one group cleanses data in SAS, and the other group runs data cleansing jobs and services on Integration Servers from DataFlux (a SAS company). The language elements in the Local Process group read data definitions out of the Quality Knowledge Base to, for example, create match codes, apply schemes, or parse text. The language elements in the Server Process group start and stop jobs and services and manage log entries on DataFlux Integration Servers. The DataFlux Integration Servers and the related dfPower Profile and dfPower Architect applications are made available with the SAS Data Quality Server software in various software bundles.


Apply Lookup Standardization Transformation The Apply Lookup Standardization transformation applies one or more schemes to one or more columns in a source table. Applying schemes modifies your source data according to rules that are defined in the schemes. Example City Scheme: Data Standard Fort Knox Fort Knox Ft Knox Fort Knox Ft Knocks Fort Knox Danbury Danbury DANBURY Danbury Dan Bury Danbury N Yarmouth North Yarmouth North Yarmouth North Yarmouth . . .

. . .

Original Data: Name City Glenn Gray Ft Knox Ric Rogers DANBURY . . .

. . .

Updated Data: Name City Glenn Gray Fort Knox Ric Rogers Danbury . . .

. . .

State ME CT . . .

State ME CT . . .

10

Create Match Code Transformation The Create Match Code transformation can be used to establish relationships between source rows. Name

Address

Jen Barker

PO Box 15

Bath

Jenny Barker

Box 15

Bathe

Jennifer Barker P.O. Box 15

City

Bath

Are these three records referring to the same individual?

With the generated match codes based on name, address and city, the three records are shown to be the same – two records can be removed from the data source. Name

Address

Jen Barker

PO Box 15

Bath

MY3Y$$Z5$$M~2

Jenny Barker

Box 15

Bathe

MY3Y$$Z5$$M~2

Bath

MY3Y$$Z5$$M~2

Jennifer Barker P.O. Box 15 11

City

MatchCode

11-7

11-8


DataFlux IS Job Transformation Jobs can be created with DataFlux dfPower Architect and dfPower Profile. Q dfPower Architect jobs are typically used to cleanse larger amounts of source data. Q dfPower Profile jobs are typically used to analyze data. SAS Data Integration Studio enables you to execute the dfPower Architect or dfPower Profile jobs within a SAS Data Integration Studio job using the DataFlux IS Job transformation.

12

DataFlux IS Service Transformation DataFlux real-time services can be created using dfPower Architect. Q Real-time services are used to synchronously process smaller amounts of data, when a client application is waiting for a response. SAS Data Integration Studio enables you to execute DataFlux real-time services within a SAS Data Integration Studio job using the DataFlux IS Service transformation.

13

All DataFlux jobs and real-time services run on DataFlux Integration Servers. To execute DataFlux jobs and services from SAS Data Integration Studio jobs, you must first install a DataFlux Integration Server and that server in SAS metadata.


Data Quality Terminology The data quality transformations and functions work with a special database called the Quality Knowledge Base (QKB). Quality A collection of locales and other Knowledge information that is referenced during data Base (QKB) analysis and data cleansing Locale

A collection of definitions in a QKB that is specific to a national language and geographic region

Example: The locale ENUSA is referenced when the specified source column contains character data in the English language that applies to the region of the United States. 14

Settings for Data Quality Transformations Settings for the data quality transformations can be set from the Data Quality tab in the SAS Data Integration Studio Options window. The following can be established: ¾ Default Locale ¾ DQ Setup File Location ¾ Scheme Repository Type

15

¾ Scheme Repository

11-9

11-10


11.2 Working with the DataFlux IS Transformations

Objectives „

19

Discuss the DataFlux Integration Server.


11-11

DataFlux Integration Server In the past, data cleansing was a batch, reactive process used to clean data being fed into data warehouses and data marts. The DataFlux Integration Server provides key benefits that include the following: Q batch process scalability Q real-time data cleansing Q full DataFlux functionality from within SAS

20

Batch jobs can be run on a server-grade machine, meaning the process is more scalable to larger data sources. Server-class machines ed by DataFlux Integration Server include: • Windows • Unix – AIX/HP-UX/Solaris/Linux The data cleansing processes, available as real-time services via Service Oriented Architecture (SOA), are available to any Web-based application that can consume services (Web applications, ERP systems, operational systems, SAS, and more). The data cleansing jobs and services ed to the DataFlux Integration Server are available (via procedures and functions) from within SAS. This gives the the full power of dfPower Architect and dfPower Profile functionality from within SAS.

11-12


What Is an Integration Server? An integration server is a service-oriented architecture application server that enables you to execute dfPower Architect or dfPower Profile jobs on a server-based platform. By processing these jobs in Windows or UNIX, where the data resides, you can avoid network bottlenecks and can take advantage of performance features available with higher-performance computers.

21

Standard versus Enterprise DataFlux Integration Server is available in two editions: Q Standard Q Enterprise In addition, a SAS restricted DataFlux Enterprise Integration Server is also provided with SAS Enterprise Data Integration Server bundle. This server is only callable from within SAS.

22


DataFlux Standard Integration Server The DataFlux Standard Integration Server s the ability to run batch dfPower Studio jobs in a client/server environment, as well as the ability to call discrete DataFlux data quality algorithms from numerous native programmatic interfaces (including C, COM, JAVA, Perl, and more). The DataFlux Standard Integration Server enables any dfPower Studio client to offload batch dfPower Profile and dfPower Architect jobs into more powerful server environments. This capability frees up the 's local desktop, while enabling higher performance processing on larger, more scalable servers.

23

DataFlux Enterprise Integration Server The DataFlux Enterprise Integration Server offers an approach to data quality that drastically reduces the time required to develop and deploy real-time data quality and data integration services.

24

11-13

11-14


Real-Time Services In addition, existing batch jobs can be converted to realtime services that can be invoked by any application that is Web service enabled. This provides s with the ability to reuse the business logic developed when building batch jobs for data migration or loading a data warehouse, and apply it at the point of data entry to ensure consistent, accurate, and reliable data across the enterprise.

25


Confirming DataFlux Integration Server Is Running

1. Select Start Ö Control . 2. Double-click istrative Tools.

11-15

11-16


The istrative Tools window opens. 3. Double-click Services.


The Services window opens. 4. Locate the DataFlux Integration Server service and that it is Started.

5. Select File Ö Exit to close the Services window. 6. Select File Ö Close to close the istrative Tools window.

11-17

11-18


Configuring the DataFlux Integration Server Manager

1. Select Start Ö All Programs Ö DataFlux Integration Server 8.1 Ö Integration Server Manager.

2. Click

to close the warning window.

3. Select Tools Ö Options.

4. Type localhost as the value for the Server name field. 5. Type 21036 as the value for the Server port field.

6. Click

To configure a remote DataFlux Integration Server, specify a valid Server name and Server port for the remote server. to save the changes and close the Options window.


The DataFlux Integration Server Manager window now displays the machine/port specified.

7. Select File Ö Exit to close DataFlux Integration Server Manager.

11-19

11-20


Creating Jobs for Execution on DataFlux Integration Server

This demonstration illustrates the creation of a dfPower Profile job, how to run the job, and how to review the generated metrics.

Create a Profile Job 1. From within SAS Data Integration Studio, select Tools Ö dfPower Tool Ö dfPower Profile(Configurator).

Alternatively, invoke dfPower Profile(Configurator) by selecting Start Ö All Programs Ö DataFlux dfPower Studio 8.1 Ö dfPower Studio, and then select Tools Ö Profile Ö Configurator.


2. In the data sources area, click on

11-21

to expand the DataFlux Sample database.

3. Click the s table. The right side of the window populates with a listing of columns found in the s table.

11-22


4. Click to the left of the s table. This selects all the fields of the table to be part of the Profile job.

5. Clear the selections for DELETE_FLG and MATCH_CD fields.


6. Specify which metrics to calculate for each of the selected columns. a. Select Job Ö Select Metrics to open the Metrics window.

11-23

11-24


b. Click

to the left of Select/unselect all to select all the Column profiling metrics.

c. Click

to save the metric selections and close the Metrics window.


7. Specify additional metrics for the Address field. a. Click the ADDRESS field.

11-25

11-26


b. Select Edit Ö Field Ö Override Metrics. c. Click

to the left of Frequency distribution.

d. Click

to the left of Pattern frequency distribution.

e. Click



The ADDRESS field has a specified.

under the M field to identify that this column has metric overrides

8. Specify additional metrics for the City field. a. Click the CITY field. b. Select Edit Ö Field Ö Override Metrics. c. Click


d. Click


e. Click


9. Specify additional metrics for the Company field. a. Click the COMPANY field. b. Select Edit Ö Field Ö Override Metrics. c. Click


d. Click


e. Click


11-27

11-28


10. Specify additional metrics for the Phone field. a. Click the PHONE field. b. Select Edit Ö Field Ö Override Metrics. c. Click


d. Click


e. Click


11. Specify additional metrics for the State field. a. Click the STATE field. b. Select Edit Ö Field Ö Override Metrics. c. Click


d. Click


e. Click


Once the desired metrics are specified, the profile job is ready to run and produce the profile report.


11-29

12. Save the Profile job. a. Select File Ö Save As. The Save As window is displayed. b. Type s Profile as the value for the Name field. c. Type Profile fields for s table as the value for the Description field.

d. Click

.

The name is now displayed on the title bar for dfPower Profile (Configurator).

11-30


Create an Architect Job (Optional) 1. From within SAS Data Integration Studio, select Tools Ö dfPower Tool Ö dfPower Architect.

Alternatively, invoke dfPower Profile(Configurator) by selecting Start Ö All Programs Ö DataFlux dfPower Studio 8.1 Ö dfPower Studio, and then select Tools Ö Base Ö Architect.


2. Add a Data Source node to the job flow. a. Locate the Nodes tab in the Toolbox .

b. Click

next to Data Inputs to expand the Data Inputs category of nodes.

11-31

11-32


c. Click the Data Source node, and then select Insert Ö Node Ö On Page. (Alternatively, you can double-click on node and it will be automatically appended to the job flow, or you can drag and drop the node onto the job flow and perform a manual connection.) The node is added to the job flow and a Data Source Properties window is opened.


d. In the Data Source Properties window, click Table window opens. e. Click f.

g. Click

next to the Input table field. The Select

to expand the DataFlux Sample database.

Click the s table.

.

11-33

11-34


The s table is now listed as the value for the Input table field. The fields found in the s table are now listed as Available.

h. Click

i.

Click

to move all columns from Available to Selected.

.


11-35

3. Add a Standardization node to the job flow, and specify appropriate properties for it. a. From the Nodes tab in the Toolbox , click b. Click

to collapse the Data Inputs category of nodes.

next to Quality to expand the Quality category of nodes.

11-36


c. Click the Standardization node, and then select Insert Ö Node Ö Auto Append. (Alternatively, you can double-click on node and it will be automatically appended to the job flow, or you can drag and drop the node onto the job flow and perform a manual connection.)

d. In the Standardization Properties window, move the ADDRESS and STATE fields from Available to Selected. 1) In the Standardization fields area, click ADDRESS field in the Available list. 2) Click

to move the ADDRESS field to the Selected list.

3) Click STATE field in the Available list. 4) Click

to move the STATE field to the Selected list.


11-37

e. Specify the appropriate Definition and/or Scheme for the two selected columns. in the Definition field for the ADDRESS field to allow selection of a valid 1) Click standardization definition.

2) Select Address as the value for the Definition field.

The selected definition appears as displayed:

11-38


3) Click in the Definition field for the STATE field to allow selection of a valid standardization definition. 4) Select State/Province (Abbreviation) as the value for the Scheme field.

5) that the default names given to the output fields are ADDRESS_Stnd and STATE_Stnd,


f.

Select the desired additional outputs fields. 1) Click 2) Click

. to move all columns from Available fields to Output fields.

11-39

11-40


3) Remove DATE, MATCH_CD, and DELETE_FLG from the Output fields list. a) Click DATE field. b) Hold down the CTRL key and click MATCH_CD field. c) Hold down the CTRL key and click DELETE_FLG field.


d) Click

4) Click g. Click

to move the selected fields from the Output fields list.

to close the Additional Outputs window. to close the Standardization Properties window.

11-41

11-42


4. Preview the results of the Standardization node. a. Click Standardization 1 node in the job flow diagram. b. Click Preview tab in the Details area.

c. that the ADDRESS and STATE field values are standardized by checking for the following: • The state values for the first two records originally were state names spelled out – the STATE_Stnd field now has these values as abbreviations. • The address value for the first record has the word Street – the ADDRESS_Stnd field has St. • The address value for the second record has the word Road – the ADDRESS_Stnd field has Rd. • Some of the original address values are all uppercased – the ADDRESS_Stnd field has these values proper-cased.


11-43

5. Add an Identification Analysis node to the job flow, and specify appropriate properties for it. a. From the Nodes tab in the Toolbox , click b. Click



c. Click the Identification Analysis node, and then select Insert Ö Node Ö Auto Append. (Alternatively, you can double-click on node and it will be automatically appended to the job flow, or you can drag and drop the node onto the job flow and perform a manual connection.)

d. In the Identification Analysis Properties window, move the field from Available to Selected. 1) In the Identification analysis fields area, click field in the Available list. 2) Click

- this moves the field to the Selected list.

11-44


e. Click

f.

in the Definition area to allow selection of a valid identification definition.

Select Individual/Organization as the value for the Definition field.

g. that the default name given to the output column is _Identity,


h. Click i.

Click

j.

Click

k. Click


to close the Additional Outputs window. to close the Identification Analysis Properties window.

6. Preview the results of the Identification Analysis node. a. Click Identification Analysis 1 node in the job flow diagram. b. Click Preview tab in the Details area.

For the first set of records in the s table, the field values are identified as INDIVIDUAL.

11-45

11-46


7. Add a Branch node to the job flow, and specify appropriate properties for it. a. From the Nodes tab in the Toolbox , click b. Click

to collapse the Quality category of nodes.

next to Utilities to expand the Utilities category of nodes.

c. Click the Branch node, and then select Insert Ö Node Ö Auto Append. The Branch Properties window is displayed.

d. Click

to accept the default settings and close the Branch Properties window.


11-47

8. Add a Data Validation node to the job flow, and specify appropriate properties for it. a. From the Nodes tab in the Toolbox , click b. Click

to collapse the Utilities category of nodes.

next to Profiling to expand the Profiling category of nodes.

c. Click the Data Validation node, and then select Insert Ö Node Ö Auto Append.

11-48


The Data Validation Properties window is displayed.

d. Click

next to the Field field and select _Identity.


e. Click

f.

next to the Operation field and select equal to.

Type INDIVIDUAL as the value for the Single field.

g. Click

h. Click

to view full expression in the Expression area.


11-49

11-50


The job flow now should resemble the following:

9. Add a Gender Analysis node to the job flow, and specify appropriate properties for it. a. From the Nodes tab in the Toolbox , click b. Click

to collapse the Profiling category of nodes.


c. Click the Gender Analysis node, and then select Insert Ö Node Ö Auto Append. d. In the Gender Analysis Properties window, move the field from Available to Selected. 1) In the Gender analysis fields area, click field in the Available list. 2) Click e. Click f.

to move the field to the Selected list. in the Definition area to allow selection of a valid gender definition.

Select Name as the value for the Definition field.

g. that the default name given to the output column is _Gender.


h. Click i.

Click

j.

Click

k. Click


to close the Additional Outputs window. to close the Gender Analysis Properties window.

11-51

11-52




11-53

10. Preview the results of the Gender Analysis node. a. Click Gender Analysis 1 node in the job flow diagram. b. Click Preview tab in the Details area.

There is a mixture of males and females in the data. 11. Add a Frequency Distribution node to the job flow, and specify appropriate properties for it. a. From the Nodes tab in the Toolbox , click b. Click

to collapse the Quality category of nodes.


c. Click the Frequency Distribution node, and then select Insert Ö Node Ö Auto Append. d. In the Frequency Distribution Properties window, move the _Gender field from Available to Selected. 1) In the Frequency distribution fields area, click _Gender field in the Available list. 2) Click e. Click

to move the _Gender field to the Selected list. to close the Frequency Distribution Properties window.

11-54



12. Preview the results of the Frequency Distribution node. a. Click Frequency Distribution 1 node in the job flow diagram. b. Click Preview tab in the Details area.


11-55

13. Add a second Data Validation node to the job flow, and specify appropriate properties for it. a. From the Nodes tab in the Toolbox , that the Profiling category of nodes is expanded. b. Click the Data Validation node, and then select Insert Ö Node Ö Insert On Page. c. Click


d. Click on the new Data Validation node and drag it so that it is next to the first Data Validation node.

11-56


e. Click on the Branch node and (without releasing the mouse button) drag cursor to the second Data Validation node. Release the mouse button.

f.

Right-click on the second Data Validation node and select Properties.

g. Click

next to the Field field and select _Identity.

h. Click

next to the Operation field and select equal to.

i.

Type UNKNOWN as the value for the Single field.

j.

Click

to view full expression in the Expression area.


k. Click



11-57

11-58


14. Preview the results of the Data Validation node. a. Click Data Validation 2 node in the job flow diagram. b. Click Preview tab in the Details area.

Seven records were identified as UNKNOWN.


11-59

15. Add a Text File Output node to the job flow, and specify appropriate properties for it. a. From the Nodes tab in the Toolbox , click b. Click

to collapse the Profiling category of nodes.

next to Data Outputs to expand the Data Outputs category of nodes.

c. Click the Data Validation 2 node in the job flow. d. Click the Text File Output node, and then select Insert Ö Node Ö Auto Append. e. In the Text File Output Properties window, specify the attributes of the file that will be created. 1) Click

next to the Output file field. The Save As window opens.

2) Navigate to S:\Workshop\lwdiwn (where X is a number provided by the instructor). 3) Type Unknown_Identities.txt as the value for the File name field. 4) Click

. The Output file field should resemble the following:

11-60

f.


Select Tab as the value for the Field delimiter field.

g. Click Include header row.

h. Click

to close the Text File Output Properties window.



11-61

16. Add an HTML Report following the Frequency Distribution node. a. From the Nodes tab in the Toolbox , that the Data Outputs category of nodes is expanded. b. Click

next to Data Outputs to expand the Data Outputs category of nodes.

c. Click the Frequency Distribution 1 node in the job flow. d. Click the HTML Report node, and then select Insert Ö Node Ö Auto Append. e. In the HTML Report Properties window, specify the attributes of the file that will be created. 1) Type Frequency Counts from Gender Analysis as the value for the Report title field. 2) Click

next to Report name field.

a) Type s Gender Frequencies as the value for the Name field. b) Click

.

3) Accept the defaults for the Output fields area. The final settings for the HTML Report Properties window should resemble the following:

f.

Click

to close the HTML Report Properties window.

11-62



17. Save the Architect job. a. Select File Ö Save. b. Type s Table Analysis as the value for the Name field. c. Click

.


11-63

Creating a Job to be Used as a Service

This demonstration illustrates the creation of a dfPower Architect job using the External Data Provider. This job will be ed to the DataFlux Integration Server and then processed with the DataFlux IS Service transformation in SAS Data Integration Studio. 1. From within SAS Data Integration Studio, select Tools Ö dfPower Tool Ö dfPower Architect. 2. Add an External Data Provider node to the job flow. a. Locate the Nodes tab in the Toolbox . b. Click

next to Data Inputs to expand the Data Inputs category of nodes.

c. Click the External Data Provider node, and then select Insert Ö Node Ö On Page. (Alternatively, you can double-click on node and it will be automatically added to the job flow.)

11-64


The node is added to the job flow and an External Data Provider window is opened.


3. Specify four generic input fields. a. Click

to add the first of four generic fields.

b. Click

to add the second of four generic fields.

c. Click

to add the third of four generic fields.

d. Click

to add the fourth of four generic fields.

11-65

11-66


e. Select the first generic field (Field) and change the value of the Field Name field to Field_1. f.

Type 20 as the value for the Field Length field for Field_3.

g. Type 25 as the value for the Field Length field for Field_4.

h. Click

.


11-67

4. Add a Basic Statistics node to the job flow, and specify appropriate properties for it. a. From the Nodes tab in the Toolbox , click b. Click



c. Click the Basic Statistics node, and then select Insert Ö Node Ö Auto Append.

11-68


d. Specify the Basic Statistics properties. 1) Click Select All. All fields become selected. 2) Type 20 as the value for the Percentile interval.

e. Click

.

5. Save the job. a. Select File Ö Save As. b. Type LWDIWN EDP Basic Stats as the value for the Name field. c. Click

.

6. Select File Ö Exit to close dfPower Architect.


11-69

ing Jobs and Services to DataFlux Integration Server

1. Select Start Ö All Programs Ö DataFlux Integration Server 8.1 Ö Integration Server Manager.

11-70


2. Click the Profile Jobs tab.

3. Select Actions Ö . The Profile Jobs window is displayed.


4. Click s Profile in the Available list. 5. Click

to move the s Profile job to the Selected list.

6. Click

to close the Profile Jobs window.

The Profile Jobs tab now displays the ed job.

11-71

11-72


7. (Optional) Click the Architect Jobs tab.

8. (Optional) Select Actions Ö . The Architect Jobs window is displayed.


9. (Optional) Click s Table Analysis in the Available list. 10. (Optional) Click

11. (Optional) Click

to move the s Table Analysis job to the Selected list.

to close the Architect Jobs window.

The Architect Jobs tab now displays the ed job.

11-73

11-74


12. Click the Real-Time Services tab.

13. Select Actions Ö . The Real-time Services window is displayed.


14. Click LWDIWN EDP Basic Stats in the Available list. 15. Click

16. Click

to move the LWDIWN EDP Basic Stats job to the Selected list.

to close the Real-time Services window.

The Real-Time Services tab now displays the ed service.

17. Select File Ö Exit to close DataFlux Integration Server Manager.

11-75

11-76


ing DataFlux Integration Server in SAS Management Console 1. Select Start Ö All Programs Ö SAS Ö SAS Management Console 9.2. 2. Log on using Ahmed’s credentials. a. that the connection profile is My Server.

b. Click

Do not click Set this connection profile as the default. to close the Connection Profile window and access the Log On window.

c. Type Ahmed as the value for the ID field and Student1 as the value for the field.

d. Click

Do not click Save ID and in this profile. to close the Log On window.


SAS Management Console is displayed.

11-77

11-78


3. that the Plug-ins tab is selected in the Navigation pane. 4. Locate the Server Manager plug-in.


5. Right-click on the Server Manager plug-in and select New Server….

The New Server Wizard is displayed.

11-79

11-80


6. Select the type of server. a. If necessary, expand Content Server grouping. b. Click Http Server.

7. Click

.


11-81

8. Specify the name and description for the new server. a. Type DataFlux Enterprise Integration Server as the value for the Name field. b. Type Integration Server running on localhost as the value for the Description field.

9. Click

.

11-82


10. Specify server properties. a. Type DataFlux 8.1.1 as the value for the Software version field. b. Type DataFlux as the value for the Vendor field. c. Click

in the Base Path(s) area.

1) that / is the value for the Base Path field. 2) Type Default Base Path as the value for the Description field.

3) Click

.

The Selected items area displays the new base path.

d. Click

next to the Application Server Type field.

e. Select DataFlux Integration Server.


The final server properties should resemble the following:

11. Click

.

11-83

11-84


12. Specify connection properties. a. Click

next to the Authentication domain field.

b. Select DefaultAuth.

c. Type localhost as the value for the Host name field.

d. the value for the Port number field is set to 21036.

The final settings for connection properties should resemble the following:


13. Click

.

14. Review the final settings in the Finish window.

15. Click

.

11-85

11-86


The Server Manager plug-in now displays the new server.

16. Select File Ö Exit to close SAS Management Console.


11-87

Using the DataFlux IS Job Transformation





2. Create a new folder. a. Click the Folders tab in the tree view area. b. Right-click on Data Mart Development folder and select New Ö Folder. c. Type DataFlux IS Examples as the name for the new folder.

11-88


3. Create initial job metadata that will use DataFlux IS Job transformation. a. Right-click on DataFlux IS Examples folder and select New Ö Job. b. Type LWDIWN - Run Profile and Architect Jobs as the value for the Name field. c. that /Data Mart Development/DataFlux IS Examples is the value for the Location field.

d. Click

.


4. Add the DataFlux IS Job transformation to the job flow. a. Click the Transformation tab. b. Click

to expand the Data Quality grouping of transformations.

c. Drag the DataFlux IS Job transformation to the Diagram tab of the Job Editor.

d. Right-click on the DataFlux IS Job transformation and select Properties. e. Type – s Profile at the end of the default value for the Name field.

11-89

11-90

f.


Click the Job tab.

g. Click

under Job type and select Profile.

The Profile.pfi job file should appear in the Job field. h. Click

next to Output destination and select File.

i.

Type S:\Workshop\lwdiwn\s Profile.pfo as the value for the File/repository name field.

j.

Type s Report as the value for the Description field.

k. Click

.

The diagram tab of the Job Editor window now displays the following:


11-91

5. (Optional) Add a second DataFlux IS Job transformation to the job flow. a. (Optional) Click the Transformation tab. b. (Optional) If necessary, click


c. (Optional) Drag a second DataFlux IS Job transformation to the Diagram tab of the Job Editor and drop it next to the first DataFlux IS Job transformation.

d. (Optional) Right-click on the second DataFlux IS Job transformation and select Properties. e. (Optional) Type – Architect Reports at the end of the default value for the Name field.

11-92

f.


(Optional) Click the Job tab.

g. (Optional) that Architect is the value for the Job type field. h. (Optional) that s Table Analysis.dmc is the value for the Job field.

i.

(Optional) Click

.

The diagram tab of the Job Editor window now displays the following:


6. Select File Ö Save to save the updated job. 7. Run the job by clicking

in the tool set from the Job Editor window.

The Details area shows that the job ran successfully.

For DataFlux IS Job transformations, completed successfully simply means that the process was ed off successfully to the DataFlux Integration Server. 8. Select File Ö Close to close the Job Editor window.

11-93

11-94


9. Access DataFlux Integration Server Manager and the jobs ran successfully. a. Select Start Ö All Programs Ö DataFlux Integration Server 8.1 Ö Integration Server Manager.

b. that both the Profile job and the Architect job completed. The bottom portion of the DataFlux Integration Server Manager displays this information on the Status of All Jobs tab.

c. Select File Ö Exit to close DataFlux Integration Server Manager. 10. View the generated profile report. a. From SAS Data Integration Studio, select Tools Ö dfPower Tool Ö dfPower Studio. b. In the left , click

to expand Management Resources.


c. Double-click the DataFlux Default management resources location.

The main area updates with a DataFlux Default tab.

d. Click

to expand the reports folder under DataFlux Default.

11-95

11-96


e. Right-click the profile folder and select Import.

The Import File window opens as displayed.

f.

Click

next to Source field.

g. Navigate to S:\Workshop\lwdiwn. h. Select All Files (*.*) as the value for the Files of type field. i.

Select s Profile.pfo.


j.

Click

k. Click

11-97

. The path and filename update in the Import File window.

.

The s Profile report is imported to the DataFlux Default management resources location.

11-98

l.


Double-click the s Profile report and dfPower Profile(Viewer) is invoked with the profile report.

m. When done viewing, select File Ö Exit to close dfPower Profile(Viewer). 11. (Optional) View the generated HTML report (from dfPower Architect job). a. (Optional) Expand the DataFlux Default reports architect folders.


b. (Optional) Double-click the s Gender Frequencies report. A browser opens to display the generated HTML frequency report.

c. (Optional) When done viewing, select File Ö Close to close the browser. 12. (Optional) View the generated text file output. a. (Optional) Open a Windows Explorer by selecting Start Ö All Programs Ö Accessories Ö Windows Explorer. b. (Optional) Navigate to S:\Workshop\lwdiwn.

11-99

11-100 Chapter 11 Implementing Data Quality Techniques (Self-Study)

c. (Optional) Double-click the Unknown_Identities.txt file. The file opens in a Notepad window as displayed.

d. (Optional) When done viewing, select File Ö Exit to close the Notepad window.

11.2 Working with the DataFlux IS Transformations 11-101

Using the DataFlux IS Service Transformation

The job being created will use the previously ed DataFlux Integration Server service. Before taking advantage of the DataFlux IS Service transformation, a table needs to be defined in metadata (this table will be the source data for the DataFlux IS Service transformation.

the s Table from the DataFlux Sample Database 1. If necessary, access SAS Data Integration Studio using Bruno’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Data Integration Studio 4.2. b. that the connection profile is My Server. c. Click





2. the s table in metadata. a. Click the Folders tab. b. Right-click on the DataFlux IS Examples folder and select Tables. c. Expand the ODBC Sources folder and select ODBC – Microsoft Access as the type of table that needs to be ed.

d. Click

.


e. The needed SAS Library does not exist in metadata. Click wizard.

to invoke the New Library

1) Type DIWN DataFlux Sample Database as the value for the Name field.

2) Click

.


3) Double-click SASApp to move it from the Available servers list to the Selected servers list.

4) Click

.


5) Type diwndfdb as the value for the Libref field.

6) Click

.


7) The needed Database Server does not exist in metadata. Click Server wizard.

to invoke the New

a) Type DIWN DataFlux Sample Database Server as the value for the Name field.

b) Click

.


c) Select ODBC – Microsoft Access as the value for the Data Source Type field.

d) Click

.


e) Click Datasrc. f) Type "DataFlux Sample" as the value for the Datasrc field.

g) Click

.


h) Review the final settings for the New Server Wizard.

i)

Click

.

11-110


8) that the newly defined database server (DIWN DataFlux Sample Database Server) appears in the Database Server field.

9) Click

.


10) Review the final settings for the New Library Wizard.

11) Click

.

11-111

11-112

f.


that the newly defined SAS library (DIWN DataFlux Sample Database) appears in the SAS Library field. Also, the value specified for the Datasrc (“DataFlux Sample”) information for the library server should appear in the Data Source field.

g. Click

.


h. Click the s table.

i.

Click

.

11-113

11-114

j.


Review the final settings for the Tables wizard.

k. Click

.

3. Update the properties for the s table. a. Right-click on the s table (in the Data Mart Development Ö DataFlux IS Examples folder on the Folders tab) and select Properties. b. Type DIWN s as the value for the Name field on the General tab.

c. Click the Columns tab. d. Click the DATE column, hold down the CTRL key and click the MATCH_CD column, hold down the CTRL key and click the DELETE_FLG column (all three columns should be selected). e. Click f.

Click

on the Columns tab tool set. to close the Properties window.


11-115

4. Create initial job metadata to use the DataFlux IS Service transformation. a. Right-click on DataFlux IS Examples folder and select New Ö Job. b. Type LWDIWN - Run Architect Service as the value for the Name field. c. that /Data Mart Development/DataFlux IS Examples is the value for the Location field.

d. Click

.

5. Create the needed job flow. a. From the Folders tab, drag the DIWN s table to the Diagram tab of the Job Editor window. b. Click the Transformations tab. c. Click

to expand the Data grouping of transformations.

d. Drag the Extract transformation to the Diagram tab of the Job Editor window, and place it next to the DIWN s table. e. Connect the DIWN s table object to the Extract transformation.

11-116

f.


Right-click on the Extract transformation and select Properties.

g. Click the Mappings tab. h. On the target table side, change the name of the STATE column to Field_1. i.

On the target table side, change the name of the PHONE column to Field_2.

j.

On the target table side, change the name of the OS column to Field_3.

k. On the target table side, change the name of the DATABASE column to Field_4.


l.

11-117

Remove the remaining target columns from the target table side. 1) Click ID column, hold down the SHIFT key and click the CITY column. 2) Click

m. Click

Ö Delete Target Columns from the tool set of the Mappings tab.


n. Click the Transformation tab. o. Click


p. Drag the DataFlux IS Service transformation to the Diagram tab of the Job Editor, placing this transformation next to the Extract transformation. q. Connect the temporary table from the Extract transformation to the DataFlux IS Service transformation.

r.

Right-click on the DataFlux IS Service transformation and select Properties.

s.

Click the Service tab.

t.

that the correct server and service information is specified.

u. Click

to close the DataFlux IS Service Properties window.

11-118


6. Run the job to this point. a. Click the DataFlux IS Service transformation on the Diagram tab of the Job Editor. b. Click

from the tool set of the Job Editor window.

c. Right-click on the temporary table object associated with the DataFlux IS Service transformation and select Open. A Warning window opens:

d. Click

.


The profiling metrics requested in the Architect job are displayed in the View Data window.

e. Select File Ö Close to close the View Data window. f.

Select File Ö Save to save the job metadata.

g. Select File Ö Close to close the Job Editor window.

11-119


Exercises

1. Working with DataFlux IS Service Transformation Create a dfPower Architect job. • Use the External Data Provider as a data source. Define two generic fields, Field_1 and Field_2, both character of length 15. • Add a Basic Pattern Analysis node to the job flow using both fields for the analysis. • Save the job, naming it LWDIWN EDP Basic Pattern Analysis. the new dfPower Architect job to the DataFlux Integration Server as a service. • Create a job in SAS Data Integration Studio. • Name the job LWDIWN – Architect Service Exercise. • Place the job in \Data Mart Development\DataFlux IS Examples folders. • Use the DIWN s table as a source table. • Use the Extract transformation following the source table, renaming the target columns for STATE and PHONE to Field_1 and Field_2. Remove the remaining target columns. • Add a DataFlux IS Service transformation to the job flow, connecting the output of the Extract transformation to this new transformation. Be sure to specify the new service in the Properties window. • Save and then run the job. • View the output to that the dfPower Architect job did produce basic pattern analysis results.

Chapter 12 Deploying Jobs 12.1 Overview of Deploying Jobs ....................................................................................... 12-3 12.2 Deploying Jobs for Scheduling ................................................................................... 12-5 Demonstration: Scheduling Orion Jobs ............................................................................. 12-13

12.3 Deploying Jobs as Stored Processes ...................................................................... 12-38 Demonstration: Creating SAS Stored Processes from Report Jobs.................................. 12-43

12-2

Chapter 12 Deploying Jobs

12.1 Overview of Deploying Jobs

12.1 Overview of Deploying Jobs

Objectives „

Discuss the types of job deployment available for SAS Data Integration Studio jobs.

3

About Deploying Jobs In a production environment, SAS Data Integration Studio jobs must often be executed outside of SAS Data Integration Studio. Jobs can be deployed: „ for scheduling „ as a stored process „ as a Web service. Under change management, only s can deploy jobs.

4

12-3

12-4


Deployment Techniques You can also deploy a job in order to accomplish the following tasks: „ Divide a complex process flow into a set of smaller flows that are ed together and can be executed in a particular sequence. „ Execute a job on a remote host.

5

12.2 Deploying Jobs for Scheduling



„

Provide an overview of the scheduling process. Discuss the types of scheduling servers. Discuss the Schedule Manager in SAS Management Console. Discuss batch servers.

8

Scheduling Requirements The SAS scheduling tools enable you to automate the scheduling and execution of SAS jobs across your enterprise computing environment. Scheduling requires four main components: „ SAS Application „ Schedule Manager „ Scheduling Server „ Batch Server

9

12-5

12-6


Steps for Scheduling Step 1

Step 2

Step 3

Step 4

SAS Data Integration Studio

Scheduling Manager plug-in in SAS Management Console

Scheduling server

Batch servers

Flow_ABC

Flow_ABC

Batch server 1

Deployment directory

event

Job_A

event

Job_A

event

Job_B

event

Job_B

event

Job_C

event

Job_C

Command line 1 Job A Command line 2 Job B

Batch server 2 Command line 3 Job C

10

Step 1:

A SAS application (such as SAS Data Integration Studio) creates a job that needs to be scheduled. If the job was created by SAS Data Integration Studio, the job is placed in a deployment directory.

Step 2:

A set up to ister scheduling can use the Schedule Manager plug-in in SAS Management Console to prepare the job for scheduling, or s can schedule jobs directly from other SAS applications. The job is added to a flow, which can include other jobs and events that must be met (such as the age of a specific amount of time or the creation of a specified file). The Schedule Manager also specifies which scheduling server should be used to evaluate the conditions in the flow and which batch server should provide the command to run each job. The type of events you can define depends on the type of scheduling server you choose. When the Schedule Manager has defined all the conditions for the flow, the flow is sent to the scheduling server, which retrieves the command that is needed to run each job from the designated batch server.

Step 3:

The scheduling server evaluates the conditions that are specified in the flow to determine when to run a job. When the events specified in the flow for a job are met, the scheduling server uses the command obtained from the appropriate batch server to run the job. If you have set up a recurring scheduled flow, the flow remains on the scheduling server and the events continue to be evaluated.

Step 4:

The scheduling server uses the specified command to run the job in the batch server, and then the results are sent back to the scheduling server.


12-7

Scheduling Servers SAS s scheduling through three types of scheduling servers: „ Platform Process Manager server „ Operating system scheduling server „ In-Process scheduling server The type you choose depends on the level of scheduling functions your organization requires, the budget and resources available to a scheduling server, and the applications that will be creating jobs to be scheduled. When you create a flow in SAS Management Console’s Schedule Manager, you specify which scheduling server the flow is to be associated with. Schedule Manager then es the flow information to the appropriate scheduling server. 11

You can create a definition for a scheduling server by using the Server Manager plug-in in SAS Management Console or an application that directly schedules jobs.

12-8


Platform Process Manager Server The Platform Process Manager server, which is part of Platform Suite for SAS, provides full-featured enterprise scheduling capabilities, including features such as workload prioritization and policy-based scheduling. The server enables you to schedule jobs using a variety of recurrence criteria and dependencies on other jobs, time events, or file events. You can use the Flow Manager application (also part of Platform Suite for SAS) to manage scheduled jobs, including deleting and stopping previously scheduled jobs. Because Platform Suite for SAS is a separate application, it requires an additional license fee. It also requires you to perform additional tasks to install, configure, and maintain all components of the application. However, the components included with the application also provide functions such as load balancing and submission of jobs to a grid. The metadata for a Process Manager Server includes the following information: • the network address or host name of a machine • the port number for the server

Operating System Scheduling Server Operating system scheduling provides the ability to schedule jobs through the services provided through a server’s operating system. Using operating system scheduling provides a basic level of scheduling at no additional cost, because the service is provided by software you already own. However, this type of scheduling does not advanced scheduling capabilities, such as the use of many types of dependencies. The specific scheduling functions that are ed vary according to the operating system used, which can make it more difficult to set up consistent scheduling criteria on several servers. Managing scheduled jobs requires you to issue operating system commands, rather than using a graphical interface. The metadata for an operating system scheduling server includes the following: • the network address of a machine • the port number for the server • the directory on the server where scheduled flows should be stored (control directory) • the command to start a SAS session on the server

In-Process Scheduling Server In-process scheduling provides the ability to schedule jobs from certain Web-based SAS applications without using a separate scheduling server. With in-process scheduling, the scheduling functions run as a process within the application. Although in-process scheduling is ed only for certain applications (such as SAS Web Report Studio), it offers basic scheduling capabilities without incurring any additional cost or requiring many installation or configuration tasks. Because an in-process scheduling server runs as part of the application, this type of scheduling also eliminates the need for the application to authenticate scheduled jobs to a separate server. However, the application must be running at the time the scheduled job attempts to run.


Schedule Manager The Schedule Manager plug-in for SAS Management Console is a interface that enables you to create flows, which consist of one or more SAS jobs. Each job within a flow can be triggered to run based on criteria such as a date and time, the state of a file on the file system, or the status of another job within the flow. The available scheduling criteria depend on the type of scheduling server used.

12

Schedule Manager is designed as a scheduler-neutral interface. When you create a flow, you specify which scheduling server that the flow is to be associated with. Schedule Manager converts the flow information to the appropriate format and submits it to the scheduling server (the Platform Computing server, an operating system scheduling server, or an in-process scheduling server).

12-9

12-10


Batch Servers Batch servers provide the command needed to run the programs that have been submitted for scheduling. „ Several batch server types are ed, each of which provides the command to run a scheduled SAS job from a specific application in a specific environment. „ The command is included in the metadata definition for each server. „ The batch server’s commands are independent of the type of scheduling server used. Batch server metadata objects are components of the SAS Application Server (for example, SASApp), and can be created by using the Server Manager plug-in in SAS Management Console. 13

The following types of batch servers are ed: SAS Java Batch server

SAS DATA Step Batch server

SAS Generic Batch server

Used for scheduling Java applications that are supplied by SAS and that are invoked and launched by the standard java.exe shell. This server definition is used for reports that are created by applications such as SAS Web Report Studio and campaigns created by SAS Marketing Automation. Used for scheduling SAS programs that contain DATA step and procedure statements. These programs include jobs that are created in and deployed from SAS Data Integration Studio. The metadata definition for a SAS DATA Step Batch server must include a SAS start command that will run in batch mode. Used for scheduling jobs from Java-based SAS applications that are not specifically ed by the SAS Java Batch server types. If a SAS application adds for scheduling jobs before a new type of SAS Java Batch server has been created, you can use the SAS Generic Batch server to provide a command to run jobs from the application.


Job Metadata Job metadata becomes available to the Schedule Manager when you use a SAS application such as SAS Data Integration Studio to schedule a job. The job metadata includes the following information: „ the command that is to be used to execute the job „ the name of the SAS batch server that is associated with the job „ the deployment directory that is associated with the job (required only for SAS DATA Step Batch servers) „ the filename of the executable program (required only for SAS DATA Step batch servers)

14

Flow Metadata Flow metadata is created when you use Schedule Manager to create a flow. The flow metadata includes the following information: „ the name of the scheduling server that is to execute the jobs in the flow „ the triggers and dependencies that are associated with the jobs in the flow Depending on the scheduling server that the specifies, Schedule Manager converts the flow metadata to the appropriate format and submits it to the scheduling server.

15

12-11

12-12


Platform Suite for SAS Platform Suite for SAS is an integrated enterprise job scheduler from Platform Computing Corporation. Platform Suite for SAS is specifically designed to manage job flows that are generated by SAS software, and includes the following components: „ Process Manager Server „ Platform Flow Manager „ Platform Calendar Editor „ Platform Load Sharing Facility (LSF) „ Platform Grid Management Services

16

Process Manager Server

Controls the submission of jobs to Platform Load Sharing Facility (LSF) and manages all dependencies among jobs.

Platform Flow Manager

Provides a visual representation of flows that have been created for a Process Manager Server. These include flows that were created and scheduled in the SAS Management Console Schedule Manager, as well as reports that have been scheduled through SAS Web Report Studio. Platform Flow Manager provides information about each flow’s status and associated dependencies. You can view or update the status of jobs within a flow, and you can run or rerun a single job regardless of whether the job failed or completed successfully.

Platform Calendar Editor

Is a scheduling client for a Process Manager Server. This client enables you to create new calendar entries. You can use it to create custom versions of calendars that are used to create time dependencies for jobs that are scheduled to run on the server.

Platform Load Sharing Facility (LSF)

Controls the submission of jobs to Platform Load Sharing Facility (LSF) and manages all dependencies among jobs.

Platform Grid Management Services

Manages jobs that are scheduled in a grid environment. This software collects information on the jobs that are running on the grid and the nodes to which work has been distributed. It makes the information available to the Grid Manager plug-in for SAS Management Console. You can use Grid Manager to view and manage the grid workload information.


Scheduling Orion Jobs





2. Locate the Orion Jobs folder. a. Click the Folders tab. b. Expand Data Mart Development. c. Expand Orion Jobs.

12-13

12-14


3. Deploy the DIFT Populate Order Fact Table for scheduling. a. Right-click DIFT Populate Order Fact Table and select Scheduling Ö Deploy. b. that SASApp – SAS DATA Step Batch Server is selected as the value for the Batch Server field.

c. Click

next to the Deployment Directory field.

1) Type Orion Star Jobs as the value for the Name field. next to the Directory field.

2) Click

3) Navigate to S:\Workshop\dift in the Select a directory window. 4) Click


5) Type OrionStarJobs and press ENTER. 6) Click the new folder, OrionStarJobs.

7) Click

.


The New directory window should resemble the following:

8) Click

.

The Deployment Directory field should resemble the following:

d. Accept the default value for the Deployed Job Name field.

e. that /Data Mart Development/Orion Jobs is the value for the Location field.

The final settings should resemble the following:

f.

Click

to save the information and close the Deploy a job for scheduling window.

12-15

12-16


An information message appears:

g. Click

.

The Orion Jobs folders shows that the DIFT Populate Order Fact Table job icon has been decorated to signify scheduling. Also, a new objects appears in the same folder, DIFT_Populate_Order_Fact_Table.

h. Single-click on the new object, DIFT_Populate_Order_Fact_Table. The status bar identifies this object as a deployed job.


i.

Right-click on the new object, DIFT_Populate_Order_Fact_Table, and select Properties.

12-17

12-18

j.


Click the Scheduling Details tab.

k. Click



l.

Right-click on the job object, DIFT Populate Order Fact Table. There are more options available:

12-19

12-20


4. Deploy DIFT Populate Old and Recent Orders Tables for scheduling. a. Right-click DIFT Populate Old and Recent Orders Tables and select Scheduling Ö Deploy. b. that SASApp – SAS DATA Step Batch Server is selected as the value for the Batch Server field. c. Click

next to the Deployment Directory field and select Orion Star Jobs.

d. Accept the default value for the Deployed Job Name field. e. /Data Mart Development/Orion Jobs is the value for the Location field. The final settings should resemble the following:

f.

Click to save the information and close the Deploy a job for scheduling window. An information message appears.

g. Click

to close the information window.

5. Deploy DIFT Populate Customer Dimension Table for scheduling. a. Right-click DIFT Populate Customer Dimension Table and select Scheduling Ö Deploy. b. that SASApp – SAS DATA Step Batch Server is selected as the value for the Batch Server field. c. Click


d. Accept the default value for the Deployed Job Name field. e. /Data Mart Development/Orion Jobs is the value for the Location field. f.

to save the information and close the Deploy a job for scheduling window. An Click information message appears.

g. Click



12-21

6. Deploy DIFT Populate Organization Dimension Table for scheduling. a. Right-click DIFT Populate Organization Dimension Table and select Scheduling Ö Deploy. b. that SASApp – SAS DATA Step Batch Server is selected as the value for the Batch Server field. c. Click




g. Click


7. Deploy DIFT Populate Product Dimension Table for scheduling. a. Right-click DIFT Populate Product Dimension Table and select Scheduling Ö Deploy. b. that SASApp – SAS DATA Step Batch Server is selected as the value for the Batch Server field. c. Click




g. Click


8. Deploy DIFT Populate Time Dimension Table for scheduling. a. Right-click DIFT Populate Time Dimension Table and select Scheduling Ö Deploy. b. that SASApp – SAS DATA Step Batch Server is selected as the value for the Batch Server field. c. Click




g. Click


12-22


The Orion Jobs folder should now resemble the following:

9. Access Windows Explorer and the creation of the code files. a. Select Start Ö All Programs Ö Windows Explorer. b. Navigate to S:\Workshop\dift\OrionStarJobs. c. that .sas files were created for each of the deployed jobs.


d. Select File Ö Close to close the Windows Explorer. 10. Access SAS Management Console using Ahmed’s credentials. a. Select Start Ö All Programs Ö SAS Ö SAS Management Console 9.2. b. that the connection profile is My Server. c. Click


d. Type Ahmed as the value for the ID field and Student1 as the value for the field. e. Click


11. Create a new flow using the Schedule Manager plug-in. a. If necessary, click the Plug-ins tab. b. Locate the Schedule Manager plug-in.

12-23

12-24


c. Right-click Schedule Manager and select New Flow.


The deployed jobs are displayed and available to be part of the new flow.

d. Type OrionStarJobs as the value for the Name field. e. Click

next to the Scheduling Server field and select Platform Process Manager.

f.

to move all items from the available list to the selected list.

Click

12-25

12-26


g. Click

.

The new flow appears under the Schedule Manager plug-in.


12-27

12. Establish dependencies within the job flow. a. Right-click on OrionStarJobs flow and select Edit Flow. The visual flow editor is available to create dependencies. b. Click

to that all six deployed jobs are found in the visual flow editor.

12-28


c. Click on the DIFT_Populate_Order_Fact_Table connection selector and drag to “connect” it to DIFT_Populate_Old_and_Recent_Orders_Tables. The New Job Event window is displayed. d. Accept Completes successfully as the value for the Event type field.

e. Click

.


f.

Click

g. Click

from the toolbar. .

h. Select Actions Ö Add Gate. i.

Click Run the deployed job only when all of the conditions occur.

j.

Click

.

12-29

12-30


A gate node is displayed in the upper-left corner.

k. Drag the gate node so that it is to the right of and in between the two jobs DIFT_Populate_Organization_Dimension_Table and DIFT_Populate_Time_Dimension_Table.

l.

Connect DIFT_Populate_Organization_Dimension_Table to the gate node.

m. Accept Completes successfully as the value for the Event type field.

n. Click

.

o. Connect DIFT_Populate_Time_Dimension_Table to the gate node.


p. Accept Completes successfully as the value for the Event type field.

q. Click r.

.

Connect the gate node to the DIFT_Populate_Product_Dimension_Table job.

12-31

12-32


s.

Click

from the toolbar.

t.

Click

to close the flow view.


13. the dependencies established using visual flow editor with the standard interface. a. Locate the deployed job DIFT_Populate_Time_Dimension_Table. b. Right-click and select Manage Dependencies.

c. Click

.

12-33

12-34


d. Locate the deployed job DIFT_Populate_Old_and_Recent_Orders_Tables. e. Right-click and select Manage Dependencies.

f.

Click

.


14. Schedule the flow. a. Right-click on OrionStarJobs job flow and select Schedule Flow. b. Type .\lsf as the value for the ID.\field. c. Type Student1 as the value for the field.

d. Click e. Click f.

. next to the Trigger field and select Manually in Scheduling Server. . A message is displayed confirming the successful scheduling.

Click


g. Click

15. Access and trigger the flow using Flow Manager. a. Select Start Ö All Programs Ö Platform Process Manager Ö Flow Manager. b. Type .\lsf as the value for the name field. c. Type Student1 as the value for the field.

d. Click

.

12-35

12-36


e. On the By tab, expand .\lsf and click OrionStarJobs.

f.

Select Flow Ö Open.

g. Select Flow Ö Trigger Ö Now.


h. Select View Ö Refresh (after a period of time).

i.

Select File Ö Close to close the flow view.

j.

Select File Ö Exit to close Flow Manager.

16. Select File Ö Exit to close SAS Management Console.

12-37

12-38


12.3 Deploying Jobs as Stored Processes

Objectives „ „

„

Describe SAS Stored Processes. List the applications that can be used to create and execute stored processes. Describe deployment of SAS Data Integration Studio jobs as a SAS Stored Process.

20

What Is a SAS Stored Process? A SAS Stored Process has the following characteristics: „ is a SAS program that is hosted on a server and described by metadata „ can be executed by many of the platform for SAS Business Analytics applications „ is similar in concept to programs run by SAS/IntrNet software, but more versatile because of the underlying metadata and security „ consists of a .sas file along with a metadata definition that describes how the stored process should execute

21


Comparing a Program to a Stored Process Consider the difference between these two methods of executing SAS programs: SAS program on local machine

One programmer or group is generally responsible for developing, testing, and executing the program. The programmer or group also manages distribution of the results.

SAS program as a stored process

One programmer or group might be responsible for developing and testing the stored process. However, many different applications can request that the stored process be executed and the results returned directly to the application.

22

Advantages of Stored Processes „

„

„

„

23

SAS Stored Processes accept input through parameters. This allows for code that is not static and can be easily run with different parameter values. SAS Stored Process code is not embedded into client applications. SAS Stored Process programs can be invoked from a variety of applications, including SAS Web applications, SAS Enterprise Guide, Microsoft Office, and SAS Visual BI (JMP software). Every application that invokes a stored process always gets the latest version available. Stored process programs ensure security and application integrity because the code is contained on a server and can be centrally maintained and managed from the server.

12-39

12-40


Where Is a Stored Process Defined and Stored? A stored process is basically a SAS program defined by metadata and stored in a source code repository. .SAS file proc report data=sashelp.class nowd; column name age height; define name / order 'Name'; define age / display; define height / display; run;

stored process stored process

Metadata Server

stored process stored process

Metadata Repository 24

Source Code Repository

Stored Processes Metadata When a stored process is created and ed, there are many different pieces of information that are stored in the metadata, including the following: ƒ attributes that describe the stored process ƒ information on how and where the stored process is executed ƒ prompts and attributes that can be surfaced to the ƒ instructions for how to return results to the application

25


What Can a Stored Process Access? A stored process can access almost any SAS data source or external file.

SAS Stored Process

External Files

SAS Data Sources

26

What Can a Stored Process Create? A stored process can create new data sets, files, and report output in a variety of formats.

SAS ODS Output

SAS Data Sources 27

SAS Stored Process

SAS Catalog Entry

Results Package

E-mail

External Files

12-41

12-42


Where Can a Stored Process Be Used? Stored processes can be invoked from several of the SAS platform applications.

SAS Stored Process

SAS Add-In for Microsoft Office

SAS Information Delivery Portal

SAS Enterprise Guide

28

SAS Web Report Studio

SAS Information Map Studio

SAS Visual BI (JMP Software)

Deploying Jobs as SAS Stored Processes You can select a job in the Inventory tree or the Folders tree and deploy it as a SAS stored process. Code is generated for the stored process and the code is saved to a file in a source repository. Metadata about the stored process is saved to the current metadata server.

29


Creating SAS Stored Processes from Report Jobs





2. Locate the Orion Report’s Extract and Summary folder. a. Click the Folders tab. b. Expand Data Mart Development. c. Expand Orion Reports. d. Expand Extract and Summary. 3. Open the job for editing. a. Double-click DIFT Create Report for US Customer Order Information. b. Right-click on the Summary Statistics transformation and select Properties. c. Click the Options tab. d. Click ODS options in the left selection pane. e. For ODS result, set the value to a blank.

f.

Click


g. Select File Ö Save to save the changes to the job. h. Select File Ö Close to close the job editor window. 4. Deploy DIFT Create Report for US Customer Order Information as a stored process. a. Right-click DIFT Create Report for US Customer Order Information and select Stored Process Ö New.

12-43

12-44


b. Accept the defaults for the General Tab information.

c. Click

.


d. Click

12-45

next to the SAS server field and select SASApp – Logical Stored Process Server.

e. Click next the Source code repository field. The Manage Source Code Repositories window is displayed. 1) Click

to add a new directory.

2) Type S:\Workshop\dift\SPSourceCode as the value for the Path field.

3) Click

to close the Add Source Code Repository window.

a) Open a Windows Explorer. b) Navigate to S:\Workshop\dift. c) Select File Ö New Ö Folder. d) Type SPSourceCode and press ENTER. e) Select File Ö Close. 4) Click f.

to close the Manage Source Code Repositories window.

Type USCustomerOrderInfo.sas as the value for the Source file field.

g. Click Package.

12-46


The Execution Tab window should resemble the following:

h. Click

. .

i.

There are no parameters to declare, so click

j.

There are no data sources or data targets to declare, so click

.


12-47

A new metadata object, a stored process, now should appear in the Extract and Summary folder.

12-48


5. Execute the SAS Stored Process using SAS Add-In for Microsoft Office. a. Select Start Ö All Programs Ö Microsoft Office Ö Microsoft Office Excel 2007. b. Click the SAS tab.

c. Click

.

d. In the Reports window, click

in the left selection pane.

e. Navigate to Data Mart Development Ö Orion Reports Ö Extract and Summary. f.

Click DIFT Create Report for US Customer Order Information.

g. Click

.


h. Accept the default location.

i.

Click

. The SAS Stored Process executes.

12-49

12-50


The requested report is displayed:

6. When finished viewing, select the Office button and then Exit Excel (do not save any changes).

Chapter 13 Learning More 13.1 SAS Resources ............................................................................................................. 13-3

13-2

Chapter 13 Learning More

13.1 SAS Resources

13.1 SAS Resources

Objectives „ „

Identify areas of that SAS offers. List additional resources.

2

Education Comprehensive training to deliver greater value to your organization „ „ „

„

More than 200 course offerings World-class instructors Multiple delivery methods: instructor-led and self-paced Training centers around the world

http://.sas.com/training/

3

13-3

13-4


SAS Publishing SAS offers a complete selection of publications to help customers use SAS software to its fullest potential: „

„ „

Multiple delivery methods: e-books, CD-ROM, and hard-copy books Wide spectrum of topics Partnerships with outside authors, other publishers, and distributors

http://.sas.com/publishing/

4

SAS Global Certification Program SAS offers several globally recognized certifications. „

„

„

Computer-based certification exams – typically 60-70 questions and 2-3 hours in length Preparation materials and practice exams available Worldwide directory of SAS Certified Professionals http://.sas.com/certify/

5

13.1 SAS Resources

SAS provides a variety of self-help and assisted-help resources. „ „ „ „ „

SAS Knowledge Base s and hot fixes License assistance SAS discussion forums SAS Technical

http://.sas.com/techsup/

6

Groups SAS s many local, regional, international, and special-interest SAS groups. „ „

SAS Global Forum Online SAS Community: www.sasCommunity.org

.sas.com/groups/

7

13-5

13-6


Selected Additional Resources Search Papers Presented at SAS Global Forum (previously known as SUGI) • .sas.com/events/sasglobalforum/previous/online.html SAS Code Samples on .sas.com • .sas.com/ctx/samples/index.jsp SAS Code Samples from Specific Books • .sas.com/documentation/onlinedoc/code.samples.html List of all SAS Products and Solutions • www.sas.com/products/index.html List of Papers • .sas.com/resources/papers/

Sas_di_developer_-_fast_track.pdf 53181f

Overview 26281t

More details 6y5l6z

More Documents from "ananth" 351e4s

Sas_di_developer_-_fast_track.pdf 53181f

Fact And Opinion Climbing Blindly 5o5s60

57-online Rejection Note,material Return 5v2q69

Artificial Intelligence With Python Tutorial 316f61

Smps 6w4u2q

Renderoc Tg 581s3h