One of the key benefits of virtualization is the ability to achieve a high consolidation ratio, thereby getting higher utilization of the hardware. This is especially true of the CPUs, since software licenses are usually tied to the number of CPUs in the hardware. During times of heavy utilization, the environment needs to be configured to make sure VMs with SLAs have their resources protected.

Remember best practices are recommendations which are dependent on their context. One often looks at two sources, stating a best practice that conflicts with each other. The difference can be the context. Here is a perfect example; one best practice is to never overcommit VMs running Oracle applications with stringent performance SLAs. Instead, protect the VMs with features like memory reservations, right size virtual CPUs, resource pools, allocation management mechanisms; such as Storage I/O Control (SIOC) and Network I/O Control (NIOC), etc. This is strongly recommended when virtualizing high workload intensive critical Oracle applications. The virtualized environment should be able to guarantee the resource and the Quality of Service (QoS), needed to meet the business requirements.

Another best practice in conflict with the previous one is to allow some level of overcommitment from Oracle environments. This is a great way to leverage all the features of virtualization by squeezing every ounce of utilization from your hardware that you can.  vSphere can manage resource sharing with its algorithms for fair share CPU scheduling, memory entitlement, NIOC, SIOC, and resource pools. However, this approach requires that the virtualization team have a lot of expertise and experience managing the overcommitting of resources and at the same time, ensuring the business SLA requirements are met. Latency-sensitive environments need to perform operations at the millisecond level. In order to achieve both goals, not having the required skill set will severely affect applications, especially when the environment grows or the application encounters increased utilization.

The best way to approach this issue is to start conservative and grow into aggressive, as and when you attain the required level of confidence with the workloads. The recommended way to go with over commitment would be:

  • Overcommit your development and test environments as much as you can, staying within common sense and meeting defined requirements.
  • Try not to overcommit production high profile environments, unless your expertise is ready for it. Initially, be conservative and do not overcommit production environments with SLAs. Then you can begin overcommitment of databases that do not have high utilization or strict performance SLAs. Use this strategy to build success and confidence with your users that the Oracle software will perform well in a VM.
  • Once you develop the right level of expertise, you can overcommit some production environments, if you have guidelines that ensure SLAs are always met. Your team must be good with resource pools, setting DRSpriorities and rules, I/O controls (Storage and Network), and SR-IOV, etc. To say you absolutely do not overcommit production environments is a simple answer, but it is not always the correct one.    Over committing allows much higher utilization of your hardware, but requires you to be smart as to when and how you overcommit.


Goals of Best Practices for Virtualizing Oracle

A goal for best practices is to reduce the possibility of errors and minimize variables when trouble shooting.

  • Develop virtualization best practices and make sure they are consistently followed.
  • Build analytical skills and metricknowledge around the four areas you are virtualizing: Memory, CPU, Storage, and Networking.
  • Understand dependencies and inter-dependencies of the various layers of the stack.
  • Educate the DBAs about the key metrics they need to understand about the virtual infrastructure, so they can determine if it is a virtualization issue or an Oracle issue.
  • Building custom widgets, using vCOPSfor DBAs, to be able to look at the virtual infrastructure the same way they would look at storage and networking in physical server environments.
  • Your bench marking should allow you to create consistent and reproducible results that you can compare against. Metrics should always be quantitative.
  • With VMware, develop best practices around vCenterand vCenter Operations, or the management and monitoring software you are using. Understand this is going to take time as well as the development of skill and expertise.
  • The approach that since the Oracle software has no knowledge if the underlying platform is physical or virtualized, the DBAs do not need to know about the virtual infrastructure will not help solve problems. DBAs and vAdmins need to work together as a team to effectively troubleshoot issues.

What are key metrics for virtualization? With any infrastructure it comes down to people, processes, and technology; learning to understand key metrics with tools like esxtop will be helpful. KISS (Keep It Simple Stupid) applies, because complex systems fail in complex ways. Good design is critical. It’s important to develop internal best practices, management processes, and guidelines for managing and monitoring a virtualization environment. It is vital to ensure your infrastructure management is ready to handle tier one workloads and the dynamics they can create.


Posted by Charles Kim, VMware vExpert, Oracle ACE Director

I was recently in LA (Santa Monica in particular). I saw a food truck and had to take a picture because this food truck was a little different in their advertising technique. I was more curious about how do they get away with this kind of public advertising?

IMG 6CAFC1E6022A 1

So I thought about how can I market my company, Viscosity North America. So here’s the new slogan for Viscosity:

“Oracle Experts, Cloud Experts, Apex Experts, Exadata Experts, and Experts of Other Shit !!!”

I wander if I can put a big sign up in our next building when we move offices.

Lot of my customers migrate databases from Solaris or AIX to Red Hat or Oracle Linux. I see more AIX databases being migrated to Linux than Solaris but this is probably just a reflection of the customers that I am involved with. Here’s a simple diagram that I created for a customer in the financial sector (of course, all confidential information is removed) who migrated from AIX to Red Hat Linux.

Shareplex Zero Downtime Database Migration Strategy

This same strategy can be leveraged to migrate customers from AIX/Solaris to Linux on a virtualized infrastructure or even AIX/Solaris to Exadata depending on the target platform. We do see more VMware customers than Oracle VM customers who want to migrate from a big endianness platform to a little endianness platform. I’ve got this entire transportable tablespace (TTS) migration almost automated. It is definitely scripted all the way through and have thoroughly tested the scripts in several customers. I guess I need to “put that lipstick on the pig” and GUI-ize it and productize the scripts to provide an additional value to my customers.

In this blog, everything starts with Shareplex. We need to plan for Shareplex installation on the production database servers (both source and target) couple of weeks prior to the final production cut-over date. We ask for couple of weeks as we are likely to encounter firewall ports that need to be opened between the AIX/Solaris database server to the new Linux servers. We will install Shareplex on both AIX and Linux and start Shareplex on both environments. On the Linux side, the skeleton database should also be pre-created and all the Oracle software installed and patched. Also on the Linux side, we will need to stop the post process (we will define what the post process is later).

On the source system (in our example AIX database), we will define the Sharplex configuration which identifies all the schemas or schema.tables that need to be replicated from the source database to the target database (in our example Linux database). I have a script that I can share which will generate the configuration file depending on which approach you choose. Once we define and activate the configuration, the capture process will start reading the redo logs or archive logs on the source system for changes to objects listed in the configuration. The Export process runs on the source system and reads data from the export queue and sends it across the network to the target system. The import process works directly with the export process. The import process runs on the target system to receive data and build a post queue. We may have more than one export and import process; they are always paired so if we have 2 export processes, we will have 2 import processes. By default, we have one of each. The post process also runs on the target system and reads the post queue, constructs SQL statements, and applies the SQL statements to replicated objects. We may have one or more post processes depending on performance design and considerations.

Depending on the size of the database and the approach that we take (RMAN image copy, datapump, export/import, CTAS over network, etc), the database cloning process can take 1 hours, 1/2 day, 1 day, 1 week or longer. We need to architect our zero downtime migration so that with any of these cloning options, the business perceives a zero downtime or a near zero downtime database migration. So how do we do that? We defined all the processes involved with Shareplex at a high-level. Let’s see how we can leverage our knowledge to start the zero downtime migration efforts. Earlier we discussed that we have a configuration file which defines the objects that need to be replicated. We need to activate our configuration so that the capture process will start reading redo logs/archivelogs and generating Shareplex queues. Once we activate our configuration, changes on the source system will be captures, exported and imported to the target system. Remember earlier, we stopped our post process as part of our high-level installation overview. All the changes from the source system will be sent to the target system (as we stopped the post process) and will accumulate for the entire duration of the migration window until we start the post process. We will need to size the target Shareplex file system with proper design considerations so that the file system can house all the Shareplex transaction queue files.

If you look at the top left corner of the diagram, we start with the RMAN image copy of the database to a file system. If you are on AIX, this can be a Veritas file system. If you cannot afford Veritas, you can perform a RMAN backup to a NFS file system. For VLDB databases, you can perceive the performance differences between a locally mounted file system versus a NFS file system. If you happen to have 10GigE available, you may not notice much performance differences.

The RMAN image copy strategy involves performing incremental update. We will perform an initial level 0 image copy backup of the database and take a incremental level 1 backup numerous times with the intention of updating the image copy with the incremental updates (aka Forever Incremental or Incrementally Updated Backups). Make sure to have block change tracking enabled before you start this process.

In this diagram, we also introduce an AIX staging server near the production database server. If we look at the transportable tables architecture, we must put the tablespaces in read-only mode to perform the TTS metadata datapump export. If you introduce the staging server, you simplify your approach and can eliminate any of the migration activity (such as putting the database in read-only mode) on the production database.

We need to go through the steps to synchronize the production database and the image copy database on the staging server. We can perform the final incremental level 1 backup update and/or even apply archivelogs to the database on the staging server as necessary depending on your approach.

  • This is where we need to decide if we want to work with SCNs and perform a zero downtime migration or take a little outage and have some flexibility. Some of our customers can afford the little downtime and some of our customers have told us that it must be zero downtime.
  • The staging server is needed so that you do not have to put the production database in read only mode for the duration that the TTS export is running

Next, we open the copied database with the resetlog option. Once the database is open, we issue the commands to put the entire database in read-only mode and copy the database files (in the absence of NFS or Veritas) to the Linux server. If we have Veritas in the equation, we can simply swing the file system to the Linux server and mount the volume. If we are using NFS, we simply present the NFS share to the Linux OS and mount the NFS share. For Solaris folks, we can mount a Solaris file system on Linux in read only mode and Veritas is not needed.

For the next step, this is where your datapump expertise starts to pay off. We need to perform a TTS export of the tablespaces that we are migrating over from AIX to Linux. The TTS datapump export is relatively simple for those who have done this before but can be a little intimidating to some who are new to this process. Once we are complete with the TTS metadata export, we need to SFTP the metadata export and log to the Linux server. After this step, we no longer need the staging server and can be shutdown. We want to the TTS export log so that we can parse the log to generate our RMAN endian conversion script. In our example, we are going to ASM so the RMAN endianness conversion will place the datafilee inside of ASM. The amount of time to migrate the database from file system to ASM will vary on the source and target storage array and wether we are talking 10gigE, bonded 1gigE, 4gig HBAs, 8gig HBAs or IB. Even for the slower HBA on older storage arrays, we can effectively drive 1 TB of endianness conversion per hour.


Honored to make vExpert for the second consecutive year. Congratulations to all vExpert 2014. I am proud to be part of this group.

We have 754 vExperts this year, which is impressive! Each of these vExperts have demonstrated significant contributions to the community and a willingness to share their expertise with others.

It’s time for the the annual IOUG Collaborate Conference again, April 7-11 in Las Vegas at the Venetian and Sands Expo Center.

We have a line up of great tracks and speakers focused on Cloud Computing, and this is a mini-compilation of the sessions focused on Cloud Tracks.

Enjoy and look forward to meeting everyone at Collaborate (#C14LV).

Best Wishes,

Charles Kim and the Cloud SIG Team (George, Bert, Kai, Ron, Steve).

Let’s look at the step-by-step procedures to create a VMware template from an existing golden image VM. The following URL to the PDF will demonstrate the the steps to templatize a VM and to provision a new VM from the newly created template:

VMware Clone to Template and Deploy Virtual Machine from this Template

Our concept of templatization does not stop at the VM. We have to create a template of the Grid Infrastructure and Database Home binaries. After we install Oracle Database or, we will apply the latest (N-1) PSU to both the Grid Infrastructure and Database Homes and required one-off patches as needed. For example, customers who have implemented GoldenGate may have to apply patches for the ACFS and/or for Integrated Extracts. Once we establish what we conceive to be the golden image for the Grid Infrastructure and Database software stack, we will create a Tar archives of both homes.

VMware vCenter Server is the centralized management tool that allows for the management of multiple ESXi servers and virtual machines (VMs) through a single console application. vCenter is a critical component in the VMware deployment. Lot of the feature/functionality are not available without vCenter. All of the well-known features in vSphere such as VMotion, Storage VMotion, Distributed Resource Scheduler (DRS), VMware High Availability (HA) and Fault Tolerance (FT) require vCenter Server.

In simplistic terms, VMware vCenter can be compared to Oracle’s Enterprise Manager 12c Cloud Control in the absence of Oracle VM in a physical server environments. For the DBAs who happen to be working on Oracle VM, vCenter is equivalent to Oracle VM Manager. At the time of writing this blog, the latest release of vCenter that was available was vCenter 5.1. News from the recent VMWorld conference, the next release of vCenter 5.5 will be available to the general public sometime in September 2013.

vCenter Server 5.1 provides a vCenter Server Simple Install option that installs vCenter Single Sign-On, Inventory Service, and vCenter Server for small VMware deployments on the virtual machine. If you prefer to customize the location and setup of each component, you can install the components separately by selecting the individual installation options in the following order:

  • vCenter Single Sign-On
  • Inventory Services
  • vCenter Server

Each of the components listed above can be installed in a different virtual machine.

Vcenter 1
Click on the option to perform the VMware vCenter simple install which will install vCenter Server, Single Sign On Server, and Inventory Service on the Windows server.

Vcenter 2

Click on OK because the vCenter server is not connected to an Active Directory domain

Then click on Next from the Single Sign On screen

Vcenter 3

When the Welcome screen appears, click Next to continue.

Vcenter 4

Select Next to accept the End-User Patent Agreement.

Vcenter 5

Please read the License Agreement; If you agree to the terms, select “I accept to the terms in the license agreement” and click on Next

Vcenter 6

Enter the password for the vCenter Single-Sign-On Admin user account

Click on Next

Vcenter 7

For the test environment, you can accept to Install a local Microsoft SQL Server 2008 R2 Express instance
For a production environment, you should install an Oracle Database with Standard or Enterprise edition

Click on Next

Vcenter 8

Enter the password for the RSA_DBA and RSA_USER database account

Click on Next

Vcenter 9

Click on OK

Click on Next

Vcenter 10

Accept the default directory path to install VMware vCenter

You can change the location if your corporate standards dictate an alternate location

Click on Next

Vcenter 11

Accept the default HTTPS Port

Click on Next

Vcenter 12

Click on the Install button to start the installation of vCenter

Vcenter 13

If you are performing just an evaluation term, click on the next button

Enter the License key for vCenter and click on Next

Vcenter 14

For the ODBC data source for vCenter Server, we will accept the default of 5 hosts and 50 virtual machines in our test environment

Click on Next

Vcenter 15

Either enter the fully qualified host.domain name or enter the IP address

Click on Next

Vcenter 16

Accept the acknowledgement for using an IP address instead of a fully qualified host name.

Click on OK

Vcenter 17

Either accept the default port assignments or modify the ports as needed and defined in the firewall rules

Click on Next

Vcenter 18

Since this is a test vCenter deployment, we will accept the default JVM Memory allocation of 1GB and click on Next

Vcenter 19

Click on the Install button to continue

Vcenter 20

Vcenter 21

Click on the Finish button

Vcenter 22

Click on OK

Login as windows administrator: administrator / password to access vCenter from the vSphere client

Book Title:
Successfully Virtualize Business Critical Oracle Databases

VMware iBook Cover

Here’s the book Description:
Written by VMware vExperts (Charles Kim (VCP), Viscosity North America, and George Trujillo (VCI), HortonWorks) and leading experts within VMware VCI and Learning Specialist (Steve Jones) and Chief Database Architect in EMC’s Global IT organization (Darryl Smith), this book will provide critical instructions for deploying Oracle Standalone and Real Application Cluster (RAC) on VMware enterprise virtualized platforms. You will learn how to setup an Oracle ecosystem within a virtualized infrastructure for enterprise deployments. We will share industry best practices to install and configure Linux, and to deploy Oracle Grid Infrastructure and Databases in a matter of hours. Whether you are deploying a 4 node RAC cluster or deploying a single standalone database, we will lead you to the foundation which will allow you to rapidly provision database-as-a-service. We will disseminate key details on creating golden image templates from the virtual machine to the Oracle binaries and databases. You will learn from industry experts how to troubleshoot production Oracle database servers running in VMware virtual infrastructures.

Database Admins/Architects, VMware Admins, System Admins/Architects, Project Architects
This book designed for Oracle DBAs and VMware administrators needing to learn the art of virtualizing Oracle.