In a RAC Configuration, OPatch supports 3 different patch methods:

1. ALL-Node Patch: The patch is applied to the local node first.  Then the patch is propagated to all the other nodes, and ultimately updates the OraInventory. For this patching process, all instances in the RAC configuration must be shutdown during the entire patching process.

2. Minimum Downtime Strategy Mode Patch:  With this strategy, OPatch first applies the patch on the local node then prompts the user for a sub-set of nodes, which will become the first subset of nodes to be patched. After the initial subset of nodes are patched, Opatch propagates the patch to the other nodes and finally updates the inventory. The downtime would happen between the shutdown of the second subset of nodes and the startup of the initial subset of nodes patched.  Here’s an example of how the minimum downtime process flow would look like:

. Shutdown all the Oracle instances on node 1 
. Apply the patch to the RAC home on node 1 
. Shutdown all the Oracle instances on node 2 
. Apply the patch to the RAC home on node 2 
. Shutdown all the Oracle instances on node 3 
. At this point, instances on nodes 1 and 2 can be brought up
. Apply the patch to the RAC home on node 3 
. Startup all the Oracle instances on node 3

3. No down time (Rolling Patch): With this method, we do not incur a downtime. Each node would be patched and brought up while all the other nodes are up and running, resulting in no disruption of the system. Some rolling patches may incur downtime due to post-installation steps, i.e. typically by running SQL scripts to patch the actual database. You must read the patch README.txt file to find out whether the post-installation steps requires downtime or not. Here’s how the Rolling Patch process will look like:

. Shutdown all the Oracle instances on node 1 
. Apply the patch to the RAC home on node 1 
. Start all the Oracle instances on node 1 
. Shutdown all the Oracle instances on node 2 
. Apply the patch to the RAC home on node 2 
. Start all the Oracle instances on node 2 
. Shutdown all the Oracle instances on node 3 
. Apply the patch to the RAC home on node 3 
. Start all the Oracle instances on node 3

 In order for a patch to be be applied in a “rolling fashion”, the patch must be designated as a “rolling updatable patch” or simply “rolling patch” in the README.txt file. When patches are released, they are tagged as either a “rolling” or “not rolling” patch. In general, patches that could be tagged as a rolling fashion are patches that do not affect the contents of the database, patches that are not related to the RAC internode communication infrastructure, and patches that change procedural logic and do not modify common header definitions of kernel modules. This includes client side patches that only affect utilities like export, import, sql*plus, sql*loader, etc. 

Only individual patches — not patch sets — will be “rollable”. It should also be noted that a merge patch of a “rolling patch” and an ordinary patch will not be a “rolling patch”. 

From onwards, all patches released will be marked as a “rolling” or “not rolling patch”, based on defined set of rules. Patches previously released are packaged as “not rolling”.

Because the set of rules currently defined are very conservative, patches released as “not rolling patches”, either before and after, may be eligible to be re-released as “rolling patches”, after analysis from Oracle Development.
If you plan to apply a patch that is marked as “not rolling” and want to check if is possible to take advantage of the rolling patch strategy, please contact Oracle Support. You can determine if a patch is a “rolling patch” or not by executing one of the following commands:

    – 9i or 10gR1: opatch query -is_rolling
    – 10gR2: opatch query -all  [unzipped patch location] | grep rolling
    – 10gR2 on Windows: opatch query -all [unzipped patch location] | findstr rolling
    – Later 10gR2 or 11g: opatch query -is_rolling_patch [unzipped patch location]

Please refer to patch readme to find out whether the patch is rolling patch or not.  For additional details, you can view Rolling Patch – OPatch Support for RAC (Doc ID 244241.1)


Posted in RAC

I am proud of our technical management team at Viscosity North America. We spent some time at Oracle OpenWorld providing our input on Oracle Linux. Surprisingly, each of us provided a different perspective and ended up being a rather comprehensive review. Please listen to our reviews of Oracle Linux and why it is the differentiator in the industry: (Rich) (Nitin) (Charles) (Jerry – Docker Containers)

Posted by Charles Kim, Oracle ACE Director

I was recently in LA (Santa Monica in particular). I saw a food truck and had to take a picture because this food truck was a little different in their advertising technique. I was more curious about how do they get away with this kind of public advertising?

IMG 6CAFC1E6022A 1

So I thought about how can I market my company, Viscosity North America. So here’s the new slogan for Viscosity:

“Oracle Experts, Cloud Experts, Apex Experts, Exadata Experts, and Experts of Other Shit !!!”

I wander if I can put a big sign up in our next building when we move offices.

Everyone knows me as an Oracle, Linux and VMware Expert. Few know me as a Certified Microsoft SQL Server expert from days of old. I am venturing into the SQL Server world again and plan on leveraging my expertise from the Oracle database world. I love the fact that Microsoft ported their SQL Server database to Linux. Stay tuned as I write future articles on how to deploy SQL Server on Linux and expose best practices to scale a SQL Server database on Linux.

For the first part of many series on virtualizing Microsoft SQL Server on VMware, let’s focus on the storage aspect of the virtalized infrastructure.

Improper storage configuration is often the culprit with performance issues. Majority of the SQL Server performance issues can be correlated back to storage configuration. Typically, relational databases, especially in the production workloads, produce heavy I/O workloads. When storage is misconfigured, performance degradations and additional latency can be introduced especially during heavy I/O workloads.

Storage is always about understanding throughput (IOPS) and disk latency. Understand your workload I/O usage patterns, thresholds, and times of high activity; benchmark and confirm you are achieving the true throughput of your hardware. Bad settings and incorrect configurations will keep the true throughput of the system from being achieved. It is important to understand the total IOPS your disk system can handle from the following formulas.

  • Total Raw IOPS = disk IOPS x number of disks
  • Functional IOPS = (disk IOPS x write%)/ (RAID overhead) + (Raw IOPS x Read%)

You need to find a balance between performance and capacity. Larger drives typically correlates to less performance. The more spindles you have, the more IOPS you can generate. Keep in mind that ESXi host demand is an aggregate demand of all VM’s residing on that host at that time. Low latency high I/O SQL Server databases are very sensitive to the latency of the I/O operations. Storage configurations are very important in achieving an optimal database configuration.

Recommendation for the best performance is always Eager Zeroes thick vmdk’s created in an Independent Persistent mode, to avoid any performance issues. Thick provisioned Lazy Zeroed vmdk’s or Thin provisioned vmdk’s can be used, as long as the Storage array is VAAI capable, improving the performance for first-time-write performance for these two types. Vmdk’s created in an Independent Persistent mode, i.e. Persistent refers to changes persistently written to disks. Independent refers to the vmdk being independent of VM based snapshots.

vAdmins can thinly provision a virtual disk. Thinly provisioned disks equate to storage on demand. Thin provisioning at the storage level and at the virtualization layer is commonly practiced in many companies, as it is a technique used to save space and to over-commit space on the storage array. Make sure how your storage is layed out for your SQL Server environments.

Development databases an be provisioned on thinly provisioned disks and can grow on-demand; however, for production workloads, make sure that you are always leveraging Eager Zeroed Thick VMDK.

This blog post touches on one of the key elements of virtualization to successfully deploy a highly performant SQL Server environment. For more details, sign up for one of my upcoming webinars on “Ten Surprising Performance Killers on Microsoft SQL Server” on Oct 12 at 1:00 PM CST.

Join Oracle’s Andy Rivenes along with three Oracle ACE Directors next week for the IOUG Master Class: Oracle Database 12c Release 2!

Register today for this on-site jammed packed technology day of learning where we will focus on two of the most compelling features – Multitenancy and Database In-Memory (DBIM) – with ton of discussions on all the new features. Come see:

Jim Czuprynski, Oracle Enterprise Architect, ViON Corporation
Rich Niemiec, Chief Innovation Officer, Viscosity North America
Charles Kim, Founder and President, Viscosity North America

You’ll also have the opportunity to win some great prizes from our event sponsors, ViON Corporation and Viscosity North America. And the best part – there is no cost to you! We have our sponsors to thank for covering all attendee costs for the day. All attendees need to do is register and attend! After registering, you will receive a confirmation of your registration. Location details and the full agenda can be found here.

Here’s the detailed agenda:
8:00 a.m – 9:00 a.m.: Breakfast, registration, networking
9:00 a.m. – 9:15 a.m.: Welcome
9:15 a.m. – 10:00 a.m.: Keynote on Oracle Future: 12cR2, Multitenant, Database In-Memory, Cloud
10:00 a.m. – 12:30 p.m.: Track 1 – PDB Me, ASAP! Oracle 12cR2 Multitenant HOL, Pt 1

Track 2 – Oracle Cloud On the Horizon and Oracle Database In-Memory Deep Dive
12:30 p.m. – 1:30 p.m.: Lunch and networking
1:30 p.m. – 4:00 p.m.: Track 1 – PDB Me, ASAP! Oracle 12cR2 Multitenant HOL, Pt 2

Track 2 – Oracle Database In-Memory By Example and Analytic Views
4:00 p.m. – 4:45 p.m.: 12cR2 Experts Panel Discussion
4:45 p.m. – 5:00 p.m.: Closing Remarks and Prize Giveaways

Click here for details and to register for this incredible event

I feel like I barely made it alive this week. My crazy life started out on Sunday evening as I delivered a webinar for the New Zealand and Australia Oracle User Group on Backing Up Your Oracle Databases to the Cloud. Because the target audience was in Australia and New Zealand, my webinar was scheduled for 7PM on Sunday. Immediately next day afternoon, I delivered another presentation for Database Trends and Applications on Database Replication for Zero downtime Migrations, High Availability and Disaster Recovery presentation leveraging SharePlex. I haven’t delivered a speech on SharePlex and Data Replication since 2000 at IOUG in Orlando. The turnout for this event was great!

Next day, I flew to San Francisco to deliver the full-day Hands-On-Lab workshop on Oracle Cloud on Wednesday. This is our featured workshop that we deliver for the Oracle Cloud SIG on behalf of the IOUG – Independent Oracle Users Group sponsored by the Northern California Oracle User Group. There was lot of content that had to be updated and lot of new content on Oracle Ravello Cloud and Bare Metal Cloud Services that had to be added.

On Thursday, I delivered the keynote at the Northern California Oracle Users Group (NoCOUG) on Why You Need to Consider Upgrading to Oracle Database 12c Release 2 (Oracle 12.2) to Securely and Easily Migrate to Oracle Cloud. This was my first time presenting for NoCOUG and the turnout was great. After my keynote, I also sat on the Expert Panel on Oracle Cloud fielding questions from the audience.

I do not think I want to repeat a week like this again. On a positive note, I did get to meet a lot of experts from our industry. If you would like to download a copy of any or all of the presentations, please feel free to visit our Viscosity North America website. I feel like this crazy presentation life style is becoming more and more of a norm for me these days. I do enjoy doing the research and geeking out with fellow Oracle technologists.

Posted by Charles Kim, Oracle ACE Director

Oracle Database 12c Release 2, packs a multitude of new features for Data Guard with high availability, data protection, and disaster recovery. Through the new functionality shared in this paper, DBAs can provide better protection for mission critical production databases from human errors, data corruptions, failures, and disasters. With the new features in Oracle 12.2, DBAs can deliver a robust reporting environment while addressing corporate disaster recovery goals.

Download the Data Guard 12.2 white paper here.

Posted by Charles Kim
Oracle ACE Director

I have been proud for the past 4 years to be the only person in the world who held the highest designation from both Oracle and VMware as the Oracle ACE Director and VMware vExpert at the same time.  This week, I am proud to announce that Nitin Vengurlekar (@dbcloudshifu) joins me in this designation.  We are known in both industries for our technical aptitude, authoring books, blogging and speaking at national/international conferences.  

I am also excited to announce that I was accepted into the VMware vExpert program for the 5th consecutive year for the 2017 calendar year.  This year will be Nitin’s first year as a VMware vExpert.  We, at Viscosity North America, are very excited and honored to be the only company in the world who house 2 individuals that hold titles of Oracle ACE Directors and VMware vExperts.

Posted by Charles Kim, Oracle ACE Director and VMware vExpert

Twitter: @racdba



I am happy to announce that I will be presenting Data Guard Best Practices and Oracle Database 12c Release 2 New Features at the next Rocky Mountain Oracle User Group Technology Day in Denver on February 9, 2017. Come learn how to bullet proof your Data Guard configurations and what’s new in Oracle 12.2. My sessions agenda will be:

Session 8 Thursday 11:15 am to 12:15 pm
What’s New in 12.2: Oracle Database 12.2 New Features

Session 10 Thursday 2:45 pm to 3:45 pm
Bulletproof Your Data Guard Environment

The Oracle Database 12.2 New Features will be a continuation of Viscosity’s 12 Days of Oracle 12.2 but will more in-depth content and code examples.

Charles Kim, Oracle ACE Director
President, Viscosity North America