RE: Limiting CPUs on SUN
- From: Wolfson Larry - lwolfs <lawrence.wolfson@xxxxxxxxxx>
- To: "'oracle-l@xxxxxxxxxxxxx'" <oracle-l@xxxxxxxxxxxxx>
- Date: Thu, 13 May 2004 10:44:59 -0500
Still hoping to find someone with SUN experience on this.
Below is part of the doc we had for the TRU64 change with used the
"pset" command
Originally this was done by another DBA team on a server with 16
CPUs and 8.1.7.2.1
Thanks
Larry Wolfson
Processor Sets
Compaq provides the capability with a number of their servers to define
groups of processors called processor sets or psets. Each pset can consist
of 1 to many processors. If no sets are defined, then all processes will
run in pset 0, which consists of all processors on the server. Through
commands such as "runon" and "pset_assign_pid", processes can be run on
specific psets.
Recommendation: This strategy was the only viable option and was chosen for
use in production. The remainder of this article discusses how this
strategy was implemented.
Implementation of Processor Sets
To properly implement processor sets with Oracle, the server and Oracle
first needed to be configured to work with them. Next, methods needed to be
determined on how to monitor the configuration. Lastly, there are different
strategies to implement processor sets, so the different strategies needed
to be tested.
Pset Configuration
The GS160 server has 4 separate QBBs (Quad Building Blocks), each holding 4
processors and 4GB of memory. The fastest memory access by a process occurs
when the processor it's running on and the memory the process is using are
in the same QBB. This is what is called "local access".
To support this type of access with Oracle, the kernel parameters
rad_gh_regions[*] should be set, one per QBB. These kernel parameters
define a local region of memory within the associated QBB to hold a slice of
the SGA. The ideal configuration with Oracle is to divide the maximum SGA
size by the number of QBBs, then set this value on each rad_gh_region[*]
parameter. For the database involved, the SGA should never grow past 1GB,
but to be safe a size of 1GB was chosen for each rad_gh_region[*] parameter.
Oracle, SAS, and Orchestrate all need to run on the server and are licensed
for 8, 4, and 4 processors respectively. Therefore 3 psets were created to
support this licensed software. To keep access local within each QBB
whenever possible, the processors were assigned to psets relative to their
QBB. Processors 0-3 were on QBB 0, processors 4-7 on QBB 1, processors 8-11
on QBB 2, and processors 12-15 on QBB 3 (see Appendix A for methods to map
processors to QBBs). Therefore, pset 0 was created using processors 0-3,
pset 1 held processors 4-7, and pset 3 held processors 8-15.
Since Oracle was relegated to processors 8-15, QBBs 2 and 3, the kernel
parameters rad_gh_regions[2] and rad_gh_regions[3] were defined to 1024
(1GB). ssm_threshold was also set to value of 0, which appears to keep each
SGA slice local within the rad_gh_regions slice.
Implementation
To force a process to run on a specific pset, you have two options, either
1), precede the execution of all Unix-level commands with "runon" (to run on
a pset or processor), or 2), assign Unix PIDs to a pset using
"pset_assign_pid". While both methods achieve the same goal, it was found
that assigning Unix PIDs to a pset was a better route. The main reason for
this conclusion was that the "runon" command causes the given command to run
in a separate shell, doubling the number of Unix processes. As the number
of concurrent processes grows, the "runon" method will use more and more
memory because of the extra Unix processes.
The assignment of Unix processes by PID to a pset was accomplished through
the use of a script called "assign_pset.ksh" (see Appendix A for the code).
This script calls "pset_assign_pid" to assign a Unix PID to a pset, based on
the Unix username. By default the script will only reassign the parent PID
(or PID of the process that called this script), with the assumption the
code is sourced (run in the same shell by using ". assign_pset.ksh"). The
script can also be run in (A)ll mode, which causes the script to check the
pset of all processes running on the server.
"assign_pset.ksh" was added to the end of "/usr/local/XXX.profile", so that
most Unix processes will get properly assigned to a pset during process
creation. "assign_pset.ksh" was also added to "cron" under root and run in
(A)ll mode every hour, to catch any Unix processes that did not run the
standard process creation scripts, such as through "cron" or "su" without
the dash.
Fortunately in Tru64, by default child processes run on the same pset as the
parent. In this way all Oracle processes, background and server, run on the
pset of the parent, which gets properly assigned during process creation.
-----Original Message-----
From: oracle-l-bounce@xxxxxxxxxxxxx
[mailto:oracle-l-bounce@xxxxxxxxxxxxx]On Behalf Of Wolfson Larry -
lwolfs
Sent: Wednesday, May 12, 2004 6:37 PM
To: 'oracle-l@xxxxxxxxxxxxx'
Subject: RE: Limiting CPUs on SUN
I found these on GOOGLE re psrset:
http://www.sas.com/partners/directory/sun/sunfire.pdf
http://www.sun.com/sun-on-net/itworld/UIR980501perf.html
Anybody using this with Oracle?
Thanks
Larry
-----Original Message-----
From: oracle-l-bounce@xxxxxxxxxxxxx
[mailto:oracle-l-bounce@xxxxxxxxxxxxx]On Behalf Of Wolfson Larry -
lwolfs
Sent: Wednesday, May 12, 2004 12:37 PM
To: oracle-l@xxxxxxxxxxxxx
Subject: Limiting CPUs on SUN
Wanted to check on this.
Client (Currently 8174) discovered they're only licensed for X CPUs where
they have X+14. They want to limit this to be in compliance and still run
associated apps.
Someone's looking at CPU_COUNT which I don't think really works at limiting
anything. I've seen it set higher than actual by some DBAs, but didn't hear
what benefit that really was.
Sun does have a psrset command I just heard of. Will that do the job for
them?
Thanks
Larry Wolfson
We did this on TRU64 but we had to modify the OS ourselves. Not a
popular decision.
NAME
psrset - creation and management of processor sets
SYNOPSIS
psrset -a processor_set_id processor_id...
psrset -b processor_set_id pid...
psrset -c [processor_id]...
psrset -d processor_set_id
psrset -e processor_set_id command [argument(s)]
psrset -f processor_set_id
psrset [-i] [processor_set_id]...
psrset -n processor_set_id
psrset -p [processor_id]...
psrset -q [pid]...
psrset -r processor_id...
psrset -u pid...
DESCRIPTION
The psrset utility controls the management of processor sets. Processor sets
allow the binding of processes to groups of processors, rather than just a
single processor. There are two types of processor sets, those created by
the user using the psrset command or the pset_create(2)
<@FPI2SOI;targetdocent=-%2F%2FSun%3A%3ASunSoft%2F%2FDOCUMENT+REFMAN2+Version
+2.0%2F%2FEN;localinfo=pset-create-2;type=> system call, and those
automatically created by the system. Processors assigned to user-created
processor sets will run only LWPs that have been bound to that processor
set, but system processor sets may run other LWPs as well.
System-created processor sets will not always exist on a given machine. When
they exist, they will generally represent particular characteristics of the
underlying machine, such as groups of processors that can communicate more
quickly with each other than with other processors in the system. These
processor sets cannot be modified or removed, but processes may be bound to
them.
l
**********************************************************************
The information contained in this communication is
confidential, is intended only for the use of the recipient
named above, and may be legally privileged.
If the reader of this message is not the intended
recipient, you are hereby notified that any dissemination,
distribution, or copying of this communication is strictly
prohibited.
If you have received this communication in error,
please re-send this communication to the sender and
delete the original message or any copy of it from your
computer system. Thank You.
----------------------------------------------------------------
Please see the official ORACLE-L FAQ: http://www.orafaq.com
----------------------------------------------------------------
To unsubscribe send email to: oracle-l-request@xxxxxxxxxxxxx
put 'unsubscribe' in the subject line.
--
Archives are at //www.freelists.org/archives/oracle-l/
FAQ is at //www.freelists.org/help/fom-serve/cache/1.html
-----------------------------------------------------------------
----------------------------------------------------------------
Please see the official ORACLE-L FAQ: http://www.orafaq.com
----------------------------------------------------------------
To unsubscribe send email to: oracle-l-request@xxxxxxxxxxxxx
put 'unsubscribe' in the subject line.
--
Archives are at //www.freelists.org/archives/oracle-l/
FAQ is at //www.freelists.org/help/fom-serve/cache/1.html
-----------------------------------------------------------------
----------------------------------------------------------------
Please see the official ORACLE-L FAQ: http://www.orafaq.com
----------------------------------------------------------------
To unsubscribe send email to: oracle-l-request@xxxxxxxxxxxxx
put 'unsubscribe' in the subject line.
--
Archives are at //www.freelists.org/archives/oracle-l/
FAQ is at //www.freelists.org/help/fom-serve/cache/1.html
-----------------------------------------------------------------
Other related posts:
