Viscosity North America (Viscosity), a leading Oracle-centric, IT solution, consulting firm, announced that they will acquire Sumner Technologies, an Oracle Application Express (APEX) training, consulting, and solutions firm with locations in Columbus, Ohio and Ashburn, Virginia.  Sumner Technologies is one of the industry’s top Oracle APEX related services and training firms, having served customers from all verticals in North America. Led by former Oracle, APEX product manager, Scott Spendolini, Sumner Technologies successfully implemented many APEX projects; as well as developed and delivered custom Oracle APEX training curriculum.
“Adding Sumner Technologies solutions, experts, and vast knowledge of Oracle solidifies  our position as the leader of APEX development in the industry and exponentially increases our presences in Oracle’s full-stack cloud and on-premise solutions,” said Jerry Ward, Chief Operating Officer at Viscosity. 
One of Viscosity’s pillars of concentration will be on delivering SaaS applications on Oracle Cloud.  Viscosity specializes in Oracle Enterprise Performance Management (EPM) Cloud with a heavy focus centered on Financial Consolidation and Close Cloud and Planning and Budgeting Cloud.  With Enterprise Resource Planning (ERP) Cloud, Viscosity sets their focus to increase productivity, lower costs, and improve controls with Financials, Procurement, and Project Portfolio Management Cloud.
Viscosity will continue to provide innovative approaches to integrate complex data between Oracle’s SaaS applications and on-premise applications.  Platform as a service (PaaS) will be our de facto standard to extend and bridge SaaS products. “Spendolini is expected to exponentially grow our PaaS for SaaS business by transforming business processes and building robust dashboard solutions with APEX to financial executives and lines of business” said Charles Kim, Chief Executive Officer. 
With Spendolini joining Viscosity, the number of books centered around the Oracle ecosystem grows to twenty books. Viscosity provides to the user community more intellectual knowledge than any other company in the world. For more information about Viscosity’s growing APEX practice, visit

Viscosity North America was founded by industry and authored experts with backgrounds in Oracle engineered systems, private/public cloud, application development, big data, and E-Business Suite. As an Oracle Platinum Partner, Viscosity is known as the “Trusted Advisors”, specializing in the delivery of full stack solutions and resolving complex data challenges. Their vast experience in verticals such as Oil & Gas, Healthcare, Finance, and Retail, provides their customers with insight into what is driving IT complexity. Viscosity offers solution services in the areas of Cloud implementation and Integration, Big Data, Analytics, Mobility, Engineered systems, Middleware, and Enterprise apps, as well as solutions in full stack health checks, license assessments, and custom application development.

This will be one of my 1 of 4 webinars that I will be presenting on Linux for SQL Server DBAs. The first session is the primer of Linux for administrators.

Since Microsoft SQL Server became generally available on Linux, more and more companies have started to adopt Linux as the operating system (OS) of choice for SQL Server. Linux may not be embraced as the mainstream operating system for mission critical Microsoft SQL Server databases yet, but many have rolled out development and test environments on Linux. To be an effective database administrator in today’s market, DBAs and Architects must possess extensive operating experience and database expertise; this includes SQL Server on Linux.

Come learn how to navigate in the world of Linux and learn the essential Linux commands covering commonly used distributions, such as Red Hat Enterprise Linux and SUSE Enterprise Server (SLES v12) to perform the day-to-day management of the Linux environment such as how to stop, start, enable and disable SQL Server Services on Linux.

Webinar Date: WEDNESDAY, MARCH 21, 2018 11:00 AM – 12:00 PM ET

Register for this event on SentryOne’s Website

You will also learn the essentials of installing SQL Server on a bare metal or virtualized infrastructure. We will also discuss the following topics so that you can navigate in the Linux environment effectively:

  • Shell scripting
  • File system and directory maintenance
  • Cron job scheduling
  • Finding large files and directories
  • Regular expressions
  • Manipulating files with vi
  • Performance monitoring
  • Reviewing top CPU and memory consumers
  • Identifying top consuming processes
  • Log management
  • Firewall management
  • Setting up ssh equivalence between Linux servers
  • Setting up a vncserver

One of the key benefits of virtualization is the ability to achieve a high consolidation ratio, thereby getting higher utilization of the hardware. This is especially true of the CPUs, since software licenses are usually tied to the number of CPUs in the hardware. During times of heavy utilization, the environment needs to be configured to make sure VMs with SLAs have their resources protected.

Remember best practices are recommendations which are dependent on their context. One often looks at two sources, stating a best practice that conflicts with each other. The difference can be the context. Here is a perfect example; one best practice is to never overcommit VMs running Oracle applications with stringent performance SLAs. Instead, protect the VMs with features like memory reservations, right size virtual CPUs, resource pools, allocation management mechanisms; such as Storage I/O Control (SIOC) and Network I/O Control (NIOC), etc. This is strongly recommended when virtualizing high workload intensive critical Oracle applications. The virtualized environment should be able to guarantee the resource and the Quality of Service (QoS), needed to meet the business requirements.

Another best practice in conflict with the previous one is to allow some level of overcommitment from Oracle environments. This is a great way to leverage all the features of virtualization by squeezing every ounce of utilization from your hardware that you can.  vSphere can manage resource sharing with its algorithms for fair share CPU scheduling, memory entitlement, NIOC, SIOC, and resource pools. However, this approach requires that the virtualization team have a lot of expertise and experience managing the overcommitting of resources and at the same time, ensuring the business SLA requirements are met. Latency-sensitive environments need to perform operations at the millisecond level. In order to achieve both goals, not having the required skill set will severely affect applications, especially when the environment grows or the application encounters increased utilization.

The best way to approach this issue is to start conservative and grow into aggressive, as and when you attain the required level of confidence with the workloads. The recommended way to go with over commitment would be:

  • Overcommit your development and test environments as much as you can, staying within common sense and meeting defined requirements.
  • Try not to overcommit production high profile environments, unless your expertise is ready for it. Initially, be conservative and do not overcommit production environments with SLAs. Then you can begin overcommitment of databases that do not have high utilization or strict performance SLAs. Use this strategy to build success and confidence with your users that the Oracle software will perform well in a VM.
  • Once you develop the right level of expertise, you can overcommit some production environments, if you have guidelines that ensure SLAs are always met. Your team must be good with resource pools, setting DRSpriorities and rules, I/O controls (Storage and Network), and SR-IOV, etc. To say you absolutely do not overcommit production environments is a simple answer, but it is not always the correct one.    Over committing allows much higher utilization of your hardware, but requires you to be smart as to when and how you overcommit.


Goals of Best Practices for Virtualizing Oracle

A goal for best practices is to reduce the possibility of errors and minimize variables when trouble shooting.

  • Develop virtualization best practices and make sure they are consistently followed.
  • Build analytical skills and metricknowledge around the four areas you are virtualizing: Memory, CPU, Storage, and Networking.
  • Understand dependencies and inter-dependencies of the various layers of the stack.
  • Educate the DBAs about the key metrics they need to understand about the virtual infrastructure, so they can determine if it is a virtualization issue or an Oracle issue.
  • Building custom widgets, using vCOPSfor DBAs, to be able to look at the virtual infrastructure the same way they would look at storage and networking in physical server environments.
  • Your bench marking should allow you to create consistent and reproducible results that you can compare against. Metrics should always be quantitative.
  • With VMware, develop best practices around vCenterand vCenter Operations, or the management and monitoring software you are using. Understand this is going to take time as well as the development of skill and expertise.
  • The approach that since the Oracle software has no knowledge if the underlying platform is physical or virtualized, the DBAs do not need to know about the virtual infrastructure will not help solve problems. DBAs and vAdmins need to work together as a team to effectively troubleshoot issues.

What are key metrics for virtualization? With any infrastructure it comes down to people, processes, and technology; learning to understand key metrics with tools like esxtop will be helpful. KISS (Keep It Simple Stupid) applies, because complex systems fail in complex ways. Good design is critical. It’s important to develop internal best practices, management processes, and guidelines for managing and monitoring a virtualization environment. It is vital to ensure your infrastructure management is ready to handle tier one workloads and the dynamics they can create.


Posted by Charles Kim, VMware vExpert, Oracle ACE Director

In a RAC Configuration, OPatch supports 3 different patch methods:

1. ALL-Node Patch: The patch is applied to the local node first.  Then the patch is propagated to all the other nodes, and ultimately updates the OraInventory. For this patching process, all instances in the RAC configuration must be shutdown during the entire patching process.

2. Minimum Downtime Strategy Mode Patch:  With this strategy, OPatch first applies the patch on the local node then prompts the user for a sub-set of nodes, which will become the first subset of nodes to be patched. After the initial subset of nodes are patched, Opatch propagates the patch to the other nodes and finally updates the inventory. The downtime would happen between the shutdown of the second subset of nodes and the startup of the initial subset of nodes patched.  Here’s an example of how the minimum downtime process flow would look like:

. Shutdown all the Oracle instances on node 1 
. Apply the patch to the RAC home on node 1 
. Shutdown all the Oracle instances on node 2 
. Apply the patch to the RAC home on node 2 
. Shutdown all the Oracle instances on node 3 
. At this point, instances on nodes 1 and 2 can be brought up
. Apply the patch to the RAC home on node 3 
. Startup all the Oracle instances on node 3

3. No down time (Rolling Patch): With this method, we do not incur a downtime. Each node would be patched and brought up while all the other nodes are up and running, resulting in no disruption of the system. Some rolling patches may incur downtime due to post-installation steps, i.e. typically by running SQL scripts to patch the actual database. You must read the patch README.txt file to find out whether the post-installation steps requires downtime or not. Here’s how the Rolling Patch process will look like:

. Shutdown all the Oracle instances on node 1 
. Apply the patch to the RAC home on node 1 
. Start all the Oracle instances on node 1 
. Shutdown all the Oracle instances on node 2 
. Apply the patch to the RAC home on node 2 
. Start all the Oracle instances on node 2 
. Shutdown all the Oracle instances on node 3 
. Apply the patch to the RAC home on node 3 
. Start all the Oracle instances on node 3

 In order for a patch to be be applied in a “rolling fashion”, the patch must be designated as a “rolling updatable patch” or simply “rolling patch” in the README.txt file. When patches are released, they are tagged as either a “rolling” or “not rolling” patch. In general, patches that could be tagged as a rolling fashion are patches that do not affect the contents of the database, patches that are not related to the RAC internode communication infrastructure, and patches that change procedural logic and do not modify common header definitions of kernel modules. This includes client side patches that only affect utilities like export, import, sql*plus, sql*loader, etc. 

Only individual patches — not patch sets — will be “rollable”. It should also be noted that a merge patch of a “rolling patch” and an ordinary patch will not be a “rolling patch”. 

From onwards, all patches released will be marked as a “rolling” or “not rolling patch”, based on defined set of rules. Patches previously released are packaged as “not rolling”.

Because the set of rules currently defined are very conservative, patches released as “not rolling patches”, either before and after, may be eligible to be re-released as “rolling patches”, after analysis from Oracle Development.
If you plan to apply a patch that is marked as “not rolling” and want to check if is possible to take advantage of the rolling patch strategy, please contact Oracle Support. You can determine if a patch is a “rolling patch” or not by executing one of the following commands:

    – 9i or 10gR1: opatch query -is_rolling
    – 10gR2: opatch query -all  [unzipped patch location] | grep rolling
    – 10gR2 on Windows: opatch query -all [unzipped patch location] | findstr rolling
    – Later 10gR2 or 11g: opatch query -is_rolling_patch [unzipped patch location]

Please refer to patch readme to find out whether the patch is rolling patch or not.  For additional details, you can view Rolling Patch – OPatch Support for RAC (Doc ID 244241.1)


Posted in RAC

I am proud of our technical management team at Viscosity North America. We spent some time at Oracle OpenWorld providing our input on Oracle Linux. Surprisingly, each of us provided a different perspective and ended up being a rather comprehensive review. Please listen to our reviews of Oracle Linux and why it is the differentiator in the industry: (Rich) (Nitin) (Charles) (Jerry – Docker Containers)

Posted by Charles Kim, Oracle ACE Director

I was recently in LA (Santa Monica in particular). I saw a food truck and had to take a picture because this food truck was a little different in their advertising technique. I was more curious about how do they get away with this kind of public advertising?

IMG 6CAFC1E6022A 1

So I thought about how can I market my company, Viscosity North America. So here’s the new slogan for Viscosity:

“Oracle Experts, Cloud Experts, Apex Experts, Exadata Experts, and Experts of Other Shit !!!”

I wander if I can put a big sign up in our next building when we move offices.

Everyone knows me as an Oracle, Linux and VMware Expert. Few know me as a Certified Microsoft SQL Server expert from days of old. I am venturing into the SQL Server world again and plan on leveraging my expertise from the Oracle database world. I love the fact that Microsoft ported their SQL Server database to Linux. Stay tuned as I write future articles on how to deploy SQL Server on Linux and expose best practices to scale a SQL Server database on Linux.

For the first part of many series on virtualizing Microsoft SQL Server on VMware, let’s focus on the storage aspect of the virtalized infrastructure.

Improper storage configuration is often the culprit with performance issues. Majority of the SQL Server performance issues can be correlated back to storage configuration. Typically, relational databases, especially in the production workloads, produce heavy I/O workloads. When storage is misconfigured, performance degradations and additional latency can be introduced especially during heavy I/O workloads.

Storage is always about understanding throughput (IOPS) and disk latency. Understand your workload I/O usage patterns, thresholds, and times of high activity; benchmark and confirm you are achieving the true throughput of your hardware. Bad settings and incorrect configurations will keep the true throughput of the system from being achieved. It is important to understand the total IOPS your disk system can handle from the following formulas.

  • Total Raw IOPS = disk IOPS x number of disks
  • Functional IOPS = (disk IOPS x write%)/ (RAID overhead) + (Raw IOPS x Read%)

You need to find a balance between performance and capacity. Larger drives typically correlates to less performance. The more spindles you have, the more IOPS you can generate. Keep in mind that ESXi host demand is an aggregate demand of all VM’s residing on that host at that time. Low latency high I/O SQL Server databases are very sensitive to the latency of the I/O operations. Storage configurations are very important in achieving an optimal database configuration.

Recommendation for the best performance is always Eager Zeroes thick vmdk’s created in an Independent Persistent mode, to avoid any performance issues. Thick provisioned Lazy Zeroed vmdk’s or Thin provisioned vmdk’s can be used, as long as the Storage array is VAAI capable, improving the performance for first-time-write performance for these two types. Vmdk’s created in an Independent Persistent mode, i.e. Persistent refers to changes persistently written to disks. Independent refers to the vmdk being independent of VM based snapshots.

vAdmins can thinly provision a virtual disk. Thinly provisioned disks equate to storage on demand. Thin provisioning at the storage level and at the virtualization layer is commonly practiced in many companies, as it is a technique used to save space and to over-commit space on the storage array. Make sure how your storage is layed out for your SQL Server environments.

Development databases an be provisioned on thinly provisioned disks and can grow on-demand; however, for production workloads, make sure that you are always leveraging Eager Zeroed Thick VMDK.

This blog post touches on one of the key elements of virtualization to successfully deploy a highly performant SQL Server environment. For more details, sign up for one of my upcoming webinars on “Ten Surprising Performance Killers on Microsoft SQL Server” on Oct 12 at 1:00 PM CST.

Join Oracle’s Andy Rivenes along with three Oracle ACE Directors next week for the IOUG Master Class: Oracle Database 12c Release 2!

Register today for this on-site jammed packed technology day of learning where we will focus on two of the most compelling features – Multitenancy and Database In-Memory (DBIM) – with ton of discussions on all the new features. Come see:

Jim Czuprynski, Oracle Enterprise Architect, ViON Corporation
Rich Niemiec, Chief Innovation Officer, Viscosity North America
Charles Kim, Founder and President, Viscosity North America

You’ll also have the opportunity to win some great prizes from our event sponsors, ViON Corporation and Viscosity North America. And the best part – there is no cost to you! We have our sponsors to thank for covering all attendee costs for the day. All attendees need to do is register and attend! After registering, you will receive a confirmation of your registration. Location details and the full agenda can be found here.

Here’s the detailed agenda:
8:00 a.m – 9:00 a.m.: Breakfast, registration, networking
9:00 a.m. – 9:15 a.m.: Welcome
9:15 a.m. – 10:00 a.m.: Keynote on Oracle Future: 12cR2, Multitenant, Database In-Memory, Cloud
10:00 a.m. – 12:30 p.m.: Track 1 – PDB Me, ASAP! Oracle 12cR2 Multitenant HOL, Pt 1

Track 2 – Oracle Cloud On the Horizon and Oracle Database In-Memory Deep Dive
12:30 p.m. – 1:30 p.m.: Lunch and networking
1:30 p.m. – 4:00 p.m.: Track 1 – PDB Me, ASAP! Oracle 12cR2 Multitenant HOL, Pt 2

Track 2 – Oracle Database In-Memory By Example and Analytic Views
4:00 p.m. – 4:45 p.m.: 12cR2 Experts Panel Discussion
4:45 p.m. – 5:00 p.m.: Closing Remarks and Prize Giveaways

Click here for details and to register for this incredible event

I feel like I barely made it alive this week. My crazy life started out on Sunday evening as I delivered a webinar for the New Zealand and Australia Oracle User Group on Backing Up Your Oracle Databases to the Cloud. Because the target audience was in Australia and New Zealand, my webinar was scheduled for 7PM on Sunday. Immediately next day afternoon, I delivered another presentation for Database Trends and Applications on Database Replication for Zero downtime Migrations, High Availability and Disaster Recovery presentation leveraging SharePlex. I haven’t delivered a speech on SharePlex and Data Replication since 2000 at IOUG in Orlando. The turnout for this event was great!

Next day, I flew to San Francisco to deliver the full-day Hands-On-Lab workshop on Oracle Cloud on Wednesday. This is our featured workshop that we deliver for the Oracle Cloud SIG on behalf of the IOUG – Independent Oracle Users Group sponsored by the Northern California Oracle User Group. There was lot of content that had to be updated and lot of new content on Oracle Ravello Cloud and Bare Metal Cloud Services that had to be added.

On Thursday, I delivered the keynote at the Northern California Oracle Users Group (NoCOUG) on Why You Need to Consider Upgrading to Oracle Database 12c Release 2 (Oracle 12.2) to Securely and Easily Migrate to Oracle Cloud. This was my first time presenting for NoCOUG and the turnout was great. After my keynote, I also sat on the Expert Panel on Oracle Cloud fielding questions from the audience.

I do not think I want to repeat a week like this again. On a positive note, I did get to meet a lot of experts from our industry. If you would like to download a copy of any or all of the presentations, please feel free to visit our Viscosity North America website. I feel like this crazy presentation life style is becoming more and more of a norm for me these days. I do enjoy doing the research and geeking out with fellow Oracle technologists.

Posted by Charles Kim, Oracle ACE Director

Oracle Database 12c Release 2, packs a multitude of new features for Data Guard with high availability, data protection, and disaster recovery. Through the new functionality shared in this paper, DBAs can provide better protection for mission critical production databases from human errors, data corruptions, failures, and disasters. With the new features in Oracle 12.2, DBAs can deliver a robust reporting environment while addressing corporate disaster recovery goals.

Download the Data Guard 12.2 white paper here.

Posted by Charles Kim
Oracle ACE Director