Configuring Pia 1x3d1i

Configuring & Tuning the PeopleSoft Application Server

1

Agenda 







 

Inside the PeopleSoft Internet Architecture (PIA) In Depth Coverage of the Application Server's Role in PIA Installation Issues, Architecture Considerations Monitoring, Tuning, and Automating Tasks High Availability and Clustering Questions/Answers

2

Part I: Inside the PeopleSoft Internet Architecture (PIA)

3

Architecture History Windo Application ws Code

Window Rendering s Code

Brows er

DB

SQL

RPC

server

Application

SQL

server

HTTP/HTTPS

Web server

Jolt/Tuxedo

Application server

SQL

DB server

DB

server

4

PeopleSoft Internet Architecture 

PeopleSoft Application Server 

 

Controls all application logic/HTML/XML generation Jolt/Tuxedo interface to servlet/web server LDAP authentication optional (Active Directory) Brows er

HTTP/HTTPS

Web server

Jolt/Tuxedo

Application

SQL

server

LDAP

DB server

5

PeopleTools Development Environment   



PeopleSoft Designer (IDE) IDE talks to DB Server, or Application Server Runs in Windows-only environment Most objects are stored in the database Wi n

SQL Jolt/Tuxedo

DB Brows er

server HTTP/HTTPS

Web server

Jolt/Tuxedo

Application

SQL

server

6

Runtime 

HTML/XML Generation  





 

Occurs on Application Server Page definitions written in a proprietary code is stored in database Developer does not code HTML; does not need to know web programming HTML automatically generated from page definition Page design is somewhat limited Pages have a standard look and feel 7

PIA Web Applications/Servlets   



 



Page: end ‟s connection to PeopleSoft apps Portal: gathers content from multiple sources Report Distribution: ‟s interface to batch reports Application Messaging Gateway: transmits publish/subscribe (ie XML) messages between nodes XML Link: used for Business Interlinks Integration Broker: next generation application messaging/XML link Sync Gateway: used for mobile connectivity Web

8

PIA Web Applications/Servlets Integration Broker 



Messaging (sync/async) across the enterprise Web services based hub that handles system to system interactions XML

External Systems

Web Brows er

server HTTP/HTTPS

Jolt/Tuxedo

Application

server

SQL

DB server

9

In Depth Coverage of the Application Server's Role in PIA

10

Application Server Domains    

Built around Tuxedo „domains‟ Tuxedo developed by Bell Labs in 1983 BEA buys Tuxedo from Novell in 1995 Provides many benefits 





Scalability (multiple instances of app server processes, fewer connections to database) Restartability/fault tolerance of app server processes Load balancing of app server processes

11

Application Server Domains 

Domains consist of two items  



Sub-directory within $PS_HOME/appserv Set of server processes

Relationships   

Domains to Machines: Many-to-1 Domains to Databases: Many-to-1 Databases to Domains: 1-to-1 12

Application Server Domains 

Simple domain on one application server machine APPLICATION SERVER

BROWSER

WEB SERVER

HR Domain

HR database

13

Application Server Domains 

Multiple domains per application server each domain associated with a different APPLICATION URL SERVER BROWSER

WEB SERVER

FS Domain

HR Domain

FS database

HR database

14

Application Server Domains 

Fault resilient – multiple machines WEB SERVER

APPLICATION SERVER FS Domain

HR Domain

BROWSER

WEB SERVER

FS database

APPLICATION SERVER FS Domain HR Domain

HR database

15

Application Server Domains APPLICATION SERVER

BBL PSAPPSRV 1st Request

Workstation Listener

APPSRV Q

PSAPPSRV

PSAPPSRV

Request Request

Request

Workstation Handler Workstation Handler

QRYSRV Q

PSQRYSRV PSQRYSRV

16

BEA Tuxedo Communications 

Queue 



Transaction 



A process used when an application needs to communicate in a time-independent fashion Groups logical actions together and allows the operation to be undone

Event 

A server is notified when the information it wants is available or when something has happened 17

TUXEDO Communication Models 



 

 

Synchronous Request/Response: tpcall(), TxRPC

Asynchronous Request/Response: tpacall(), tpgetrply() Conversation: tpconnect(), tpsend(), tprecv() Unsolicited Notification: tpnotify(), tpbroadcast()

Event: tppost(), tpsubscribe() Queued Request/Response: tpenqueue(), tpdequeue()

18

PeopleSoft Servers 

PSAPPSRV  



PSQCKSRV (optional) 



Performs transactional SQL requests

PSQRYSRV 



Performs a subset of fast, non-transactional, read-only requests

PSSAMSRV 



Performs builds and save Caches object definitions & business logic in memory to speed processing

Handles SQL request from PS Query

Messenger Services   

Publishing Broker: PSBRKDSP, PSBRKHND Publishing Contractor: PSPUBDSP, PSPUBHND Subscription Contractor: PSSUBDSP, PSSUBHND

19

Application Server Domains 

PS 

 



psappsrv.cfg  

 

Command line utility to create and ister domains Domain status For use by the Application Server Contains domain specific settings For use by the Application Server

Many other files behind the scenes Process Scheduler domains use same procedure20

Installation Issues and Architecture Considerations

21

Two Main Discussion Points  

AppServer Sizing Simple Install

22

AppServer Sizing 

  

 



Measure against a known workload TPC/C Ratings (http://www.tpc.org) Compare the TPC/C ratings Compare the different models Physical or Logical Servers?… Are the Servers, OS, etc. ed by PeopleSoft? Upgrade options?

23

AppServer Sizing  

 

PeopleSoft Benchmarks Rule-of-Thumb: Application Server needs 6 times more processing power than a web server Estimate the concurrent usage patterns Categorize the s into light, medium and heavy

24

AppServer Sizing 



Estimate the number of Light, Medium & Power s Estimated load by type   



Light TPC/C = Medium TPC/C = Power TPC/C =

60 100 30

For Example Calculation, Assume   

500 light s 100 medium s 10 power s

25

App Server Sizing 

Calculating the TPC figures   

 

Thus Total Appserv TPC units = 43,000 You have to find a server that would deliver at least this many TPC units 



For 500 Light s, 60 X 500 = 30,000 For 100 Med s, 100 X 100 = 10,000 For 10 Power s, 300 X 10 = 3,000

Add any additional work load

Memory???? PS recommends 512MB per 25 s 

Add 1.5MB per each additional

26

Appserver Sizing: !   



Highly U bound Large memory foot print in psappsrv Memory growth may result in fragmentation of main memory Watch for the recycle count to recycle the appserv at periodic intervals. 

“Caching is good; Queuing is bad” 27

Simple Install 





Comes right off the Tuxedo CD  Install.sh or setup.exe  UNIX/AIX ftp  Tuxedo 6.5/Jolt 1.2 certified Tools 7.05 – 8.4x Max install approx. 160MB  Software  Docs  WebGUI Upgrading Tuxedo?  Win - Must uninstall old version!  UNIX – Tuxedo 6.3 vs 6.5?

28

Simple Install 





AppServer  Existing vs. New Environment Variables  TUXDIR  PATH  NLSPATH Validate Install  $TUXDIR/udataobj/lic.txt  “PeopleSoft Edition – Built on BEA – March 21, 2000” 29

Simple Install 

Problems with the Install?  Right CD?  Disk Space?  Windows - uninstall complete?  UNIX/AIX /var/tmp sizing?

30

Simple Install 

Configuration  PS  Command line utility to create and ister domains  Domain status  For use by the application server  New Install  Domains?  Sub-directory within $PS_HOME/appserv  Set of server processes  You can have more than one domain per machine  Each domain only connects to one database  Multiple domains can connect to the same database

31

Simple Install 

Configuration  Upgrades 





Recreate Domain!!!!!!!!! Tuxedo may not change but PIA does!

Creating the Domain (PS) 

  

Updates psappsrv.cfg, generates psappsrv.ubb, creates pstuxcfg Appserver Size Templates IDs Determine ports in advance (tracking is a good idea)  WSL (7000)  WSH Random assignment  JSL (9000)  JSH = JSL + Max Handler!  JRAD (9100)

32

Simple Install 

Problems?  Bad ID  Wrong Database  No 2-Tier access  Unavailable Port  No BBL  Add to Path? 33

Monitoring, Tuning, and Automating Tasks

34

Most Common Performance Problems - Web & App. Server S/W

http://www.newport-group-inc.com/ 35

Typical PeopleSoft Daily Monitoring

      

 

Database components System logs Process Scheduler Application Server(s) Web Server(s) Report Server File Server Network operation Customer Connection for patches and code updates 36

Monitor Application Server 



The PeopleSoft can check the status of each application server domain from the PS program Log files 





For each domain should be checked at least daily File size should also be checked; they can grow quite large, especially if the trace options are enabled

Check for queuing at peak and normal times 37

Automated Monitoring echo pq | tm -r 2> /dev/null > domainlog while test 1 -eq 1 do echo pq | tm -r 2> /dev/null >> domainlog sleep 5 done 38

Automated Monitoring   

Quest - Foglight Tivoli BMC – Patrol

39

Troubleshooting 



 

Validate that all servers and services are still running Check whether or not some additional servers need to be loaded (according to the workload) Check the ULOG files for error messages Check the status of the database the application is working with 40

Troubleshooting (cont‟d) 



 

Check whether the limit of IPC resources has been reached Check the free space status of each involved directory Check for network problems Check whether there are enough BEA TUXEDO licenses 41

Tuning  







 

How do you know you have a problem? Record “normal” baseline performance metrics Change a configuration value and then test Use a load tester (i.e. Mercury LoadRunner) Monitor transactions per min, U, memory, and disk utilization Compare results to baseline metrics Readjust performance metric if necessary 42

PSAPPSEVR Memory

CRM

100-150 MB

HRMS

100-150 MB

Supply Chain

300-500 MB

s Per Process 50 s 30 s 10-20 s

Financials

150-300 MB

20-30 s

Application

Memory Use

* PeopleSoft recommends using these values for estimating memory usage

43

PSAPPSERVR Memory 

Windows 2000 





 



Add up the “Memory Usage” and “Virtual Memory Usage” columns This is not an absolute value because some processes are shared Should not exceed 70% of real memory Check page/sec for page swapping Should be below 100 hard page swapping / sec

UNIX 



Use ps_chk_domain (search for “Chistopher Che”) Use vmstat and check po and pi columns for paging

44

Recycle Count 



  

The AppServer caches objects to memory If memory usage gets too high swapping occurs Set Recycle Count = 2000 Check for memory swapping To minimize the cost of re-caching, set EnableServerCaching=2 45

Timeouts 

Timeout during a PIA transaction  



Timeout between PIA transaction 



Transaction fails and must be resubmitted Values should be staged has been idle too long

Timeout that reserves a resource until expiration time 46

Application Server Tips 

Adjust timeout parameters in order 

Apache → Jserv → Jolt → Servlet → PIA → App service httpd.conf zone.properties psappsrv.cfg configuration.properties pstools.properties psappsrv.cfg

47

Application Server Tips 

tm 

PrintQueue (pq)



PrintServer (psr)

48

Application Server Tips 

Tuxedo settings 

Message size, queue, segment, semaphore Win2k: Control -> Bea Tux-> IPC Resources UNIX: kernel prefix with msg*,sem*,shm*

49

Application Server Tips 

OS tuning - NT/Win2K 



OS tuning - UNIX 





TTimedWaitDelay, MaxPort UNIX: kernel prefix with t* Process priority, increase load page size

Jolt handler 

Max handlers * Max Clients per Handler >= # of active s

50

High Availability (HA) and Clustering

51

HA and Clustering Topics 





PeopleSoft Internet Architecture (PIA) and Clustering Infrastructure Webserver Cluster  

  

WebLogic WebSphere

Application Server Cluster Database Server Cluster Example Clustered Environment

52



High Availability 



Continuous Availability 



A system designed to avoid loss of service Non-stop service

Fault Tolerance 

Method for achieving very high levels of availability 53

Downtime 

Is 99.9% good enough?   





20,000 incorrect drug prescriptions per year 500 incorrect surgical operations each week 3,500 US babies dropped at delivery each year 22,000 checks deducted from the wrong bank s each hour Your heart fails to beat 32,000 times each year 54

Clustering and HA 

Set target for Quality of Service (QOS) 





Clustering method  





Scalability (i.e hundreds of thousands of s) 99.9% availability Load balance Replication OS clustering

OS clustering 

Used only in DB Tier

55

PIA Clustering 

Also consider  



Scalability Availability Security Brows er

HTTP/HTTPS

Web

Jolt/Tuxedo

SQL

server

server

Sticky Sessions Load Balance

Application

Load Balance

DB server

OS Cluster or Replication

56

Redundant Architecture Internet

Router

Router

DMZ VLAN2

Corporate Net

VLAN1 Load Balancer

Router

Firewall

App Server1

hp proliant DL560

UID

Clustered VIP Address

Int

hp proliant DL360g3

Ext Lan1 Lan2

1

0

1

0

UID Int

hp proliant DL360g3

Ext

VLAN3

hp proliant DL560

App Server2

Lan1 Lan2

Web Servers hp proliant DL740

1

2

3

4

5

Database Server

Load Balancer Web Servers

OPEN

VLAN4

UID

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

OPEN

Int

hp proliant DL360g3

Clustered VIP Address

Ext Lan1 Lan2

1

0

UID Int

hp proliant DL360g3

Ext Lan1 Lan2

1

0

RAID Storage

57

Infrastructure 

Internet Access  



Routers 





Remove routing and forwarding if firewall is multi-homed

Hardware Load Balancer for scalability Reverse Proxy Server 



Enable Packet Filtering, Port Blocking, etc for security

Firewalls - Load balanced 



BGP 4 (Border Gateway Protocol) FatPipe

Microsoft ISA

Storage   

RAID 0+1 (striping and mirror) RAID 5 AutoRAID

58

Webserver Cluster 

Use hardware load balancer   

 

  

Cisco LocalDirector Enable sticky generic Sticky timeout greater than PeopleSoft session Create a http probe Monitor syslog for failure

Instances can use different ports Maximum scalability Minimum upgrade issues

59

Application Server Cluster    

Multiple Processes per domain Load balance to domain – random Load balance to process – round robin Edit configuration.properties  

Comma separated list of JOLT hosts:port psserver=AppHost1:9000,AppHost1:9100,App Host2:9000,AppHost2:9100 60

Database Cluster 

Replication 



Database clustering 





Lots of choices

Very complicated and expensive Requires O/S Lock Manager

OS Clustering     

AIX – HACMP Compaq – TruCluster HP – Service Guard Solaris – Sun Cluster Windows – MSCS 61

HA – Key Points 

 

Plan redundant infrastructure on paper and then test in a lab Justify each redundant item Keep it simple  

 

Some equipment may not require redundancy Avoid O/S clusters

Continue to make backups Learn how troubleshoot the system 

Use monitoring tools

62

Questions / Discussion

63