Advantages, Disadvantages, & Risks Of Virtualization For Your Database — And How To Get It Right!

Advantages, Disadvantages, & Risks Of Virtualization For Your Database — And How To Get It Right!

Types Of Virtualization

There are many types of virtualization. Storage, network, server, database. For the purposes of this article I will discuss server virtualization, but with a special focus on servers that will house databases.  

Server virtualization essentially refers to abstracting the services that make up a computer server from the underlying hardware resources. Database virtualization on the other hand refers to abstracting the services that make up a database system from the servers that provide those services. It is essentially another layer of abstraction.

I am choosing to write about server virtualization in this article because it is has been more widely adopted so far than database virtualization and is implemented in a database agnostic way. In other words, when you virtualize your database servers, the advantages and disadvantages will apply to any database that you are using on that server. 

Advantages Of Virtualization

There are many advantages to server virtualization.  I will discuss two key advantages here.

Rapid Provisioning

Probably the greatest advantage of server virtualization is rapid provisioning. Virtualization platforms like VMWare enable us to build new servers in seconds based on existing servers or server templates. This is a major improvement over needing to configure servers individually in the past.  This saves time, money, and perhaps most importantly, improves consistency and can be used to enforce policies if administered and controlled properly.  By narrowly configuring servers to handle one database or a collection of related databases, we can extend the benefits of rapid provisioning to the database, facilitating rapid creation or refresh of test, dev, or qa database environments, for example. 

Resource Utilization

Another advantage of virtualization is greater resource utilization. Servers on many virtualization platforms can be configured to use resources such as memory on an as- needed basis. This minimizes the amount of resources that need to be maintained for burst times, assuming that all servers do not burst at the same time. Of course, taking advantage of this capability requires careful planning and an understanding of the resource usage profiles of your servers. 

Disadvantages/Risks Of Virtualization

While there are many advantages of virtualization, there are also key disadvantages, which come mostly in the form of risk.  These disadvantages are not inherent problems with virtualization. Instead, they can be the result of a lack of strict planning and management of a virtual environment. 

Management and Accountability

Rapid and simple provisioning comes with a cost. The ease and speed of spinning up new servers tends to promote server and database sprawl, causing management and accountability problems. When virtualizing, strict policies and procedures must be implemented and enforced to avoid future problems, especially in environments with multiple system managers.

Hardware Cost

In addition to management and accountability problems, actual costs can spin out of control. In a cloud environment like AWS (one type of virtualization), cloud provider costs that seem small on a server-by-server basis, quickly add up as server sprawl kicks in. Similarly with in-house virtualization infrastructure, easily created servers eventually overwhelm the resources in the system and more hardware must be purchased, often with difficulty tracking those costs to specific projects or departments. 

Quality

Database and System management involves a number of skills;  there are the hard technical skills like knowing what command commands to execute in order to download and install a Linux distribution or an Oracle Patch.  And for every one of those hard skills, there are a hundred soft skills, like knowing what downstream impact a Linux patch may have, what the likely security implications are of granting access to folder required by a piece of software that needs to be installed, or knowing how to determine the most efficient way to configure resources for Oracle.  A huge risk in a virtualized environment is that the ease and speed of provisioning may give the false impression that the need for highly skilled system and database managers has diminished. On the contrary, I think that the ease and speed of provisioning increases the need for those skill sets, because the potential to propagate a poor configuration throughout the system is much greater in a virtualized environment.  And fixing 10 servers later is much more expensive than provisioning the first server properly in the beginning.

Licensing

Major database vendors price their software based on the underlying resources on the machine that it is running on. In a virtual environment, we can assign a certain amount of computer power to a database server, and that can be a small fraction of the total computer power of the virtualization cluster. But the vendors don’t see it that way, Oracle for example, bases the cost on the total CPU power across the whole cluster regardless of how much power we assign to a given server. This is true unless we use Oracle’s virtualization platform, where it honors the resource partitioning of the virtualization platform. Misunderstanding about this licensing model has caused many companies to be unexpectedly charged very large back licensing fees.  Note that a potential solution to this may be to create a separate virtualization cluster for the database environment but this limits some of the advantages described above.

Security

Security is always a concern when provisioning a server or database. There are many configuration settings, folder access rights restrictions, OS and database users that need to be deactivated, removed, or restricted. The rapid cloning and perceived lower skill requirement for provisioning new servers can take a small security problem and rapidly propagate it throughout the environment. So while there may not be new security vulnerabilities introduced simply because we now operate in a virtual environment,  as with all of the other disadvantages and risk that I mentioned, the risks are magnified in a rapid provisioning environment. 

How To Get It Right?

I spoke to two experts who are responsible for virtualization platforms for their organizations or for client organizations that run mission critical applications. I wanted to find out the keys to success in building and maintaining a solid virtualization platform. Here are some of the takeaways.

I first spoke to the CIO of a financial institution that runs their entire shop on virtualized servers. He said that in-house server virtualization is a mature technology and risks are low for an organization with a relatively stable application mix and resource load, and with a small system management staff. He also feels that compromising like throwing a whole blade server at a specific application, is somethings worth it to limit the risk of resource contention, even though it may reduce the benefit of efficient resource utilization.

I also spoke with Rocco Guerriero, CEO of Contour Data Solutions. Rocco says that having the right policies and procedures in place is the key to ensuring a trouble free environment. For example, he points out that if you are implementing a mission critical database server with a standby database, it is important to ensure that the virtual server holding the primary database will never be migrated to the same physical server that holds the standby. This can be done using rules that can be specified using the virtualization tools.

Rocco also cautions that a good rule of thumb is to have enough resources in each cluster so that you don’t exceeding 50% usage during typical load.  This ensures that servers can acquire the resources they need when demand spikes.

And finally, he recommends carefully assigning priorities to virtual machines if you need to ensure that certain servers always get the resources they need even at the expense of others. 

Conclusion

Server Virtualization can be an effective way to reduce costs and speed up provisioning of hardware and software for our IT projects. But we must mitigate the risks from the beginning. Here are a few steps that I believe should be part the management plan for any virtualized environment.

  • Establish policies and procedures that must be followed for all servers to be provisioned. Based on the conversations that I had with these experts, and on my experience administering databases in virtual environments, I think policies should be in place that require the following;   Evaluation and implementation of necessary virtualization rules as described above, change control, security review, database license review, before and after cluster resource capacity review, and a record of each server stating what application(s) it is to be used for, resource cost estimate, provenance tracking of the server image, backup and restore requirements, and other data that will assist with management.
  • Ensure that only highly skilled system and database administrators are responsible for provisioning and configuring new servers and databases. Resist the temptation to enable regular users or developers who do not have system management experience to do this.

At the end of the day the thing to remember is that provisioning servers is very easy, and very easy to get it wrong.

Managing Server Sprawl With AWS Management Console Alerts

Managing Server Sprawl With AWS Management Console Alerts

A DBA’s Transition Guide for Hosting on the AWS Cloud

So your organization has decided to migrate your traditional on-premises IT infrastructure to the AWS Cloud in the hopes of realizing cost savings, and to cut down on the time it takes to provision and configure services to support new and changing application workloads. Applications can evolve over time to cloud-centric architectures in order to realize cost savings. But what about all the extra administrative tasks and pressures that go along with the additional speed and agility that cloud hosting provides? How do you keep a handle on all the new instances and know when there are server sprawl issues? Or, even better, avoid server sprawl issues in the first place?

Every DBA knows that whenever anything goes wrong it is always the database that is guilty until proven innocent. So how can DBAs adapt to the new challenges of AWS hosting to remain valuable assets to our organizations?

For the purposes of this blog we will focus on database monitoring and management using the AWS CloudWatch service. CloudWatch ingests performance data from a wide range of AWS resources, applications and services, sends alerts when needed, and keeps a 15-day historical record of performance information. You can even configure CloudWatch with alarm actions to automatically take corrective measures in response to certain predefined event types (but that is a blog for another time). As an added bonus, the CloudWatch “free tier” should be sufficient to perform the heavy lifting of issue detection and identification for most application databases.

Monitoring Performance Metrics of Databases Managed with Amazon RDS

As with traditional on-premises databases, CPU utilization and available memory are two sides of the same performance tuning coin for databases in the AWS Cloud.

You can use the CPUUtilization metric in CloudWatch to keep a historical view of CPU usage for databases managed with Amazon Relational Database Service (Amazon RDS). To get a more complete picture of how an RDS database instance is performing, you can combine CPU monitoring with these additional metrics:

  • FreeableMemory, which shows the amount of available memory
  • SwapUsage, which shows how much data in memory is being paged to disk due to memory shortages

You can also configure CloudWatch to send alerts when thresholds are crossed.

One of the best features of cloud hosting is you are no longer locked into a specific database footprint based on hardware that was purchased. If you start to see a trend of CPU availability consistently running above 80%, or you’re seeing a shortage of free memory, it could be time to take advantage of the cloud’s on-demand scalability and plan to grow your DB instance to increase capacity. Likewise, if you notice that your databases are consistently showing a large amount of free memory and CPU, then think about scaling down the database instance class to save money.

Storage Monitoring and Auto Scaling To Avoid Server Sprawl

In the AWS cloud, there is never a good reason for running out of available storage on a production database, or any database for that matter. For example, you can use the CloudWatch FreeStorageSpace metric to measure the amount of storage space available to a database instance and trigger alerts as needed. Amazon RDS hosted databases also support storage auto scaling on all major RDS database offerings. This option automatically increases the storage by 5 GB or 10% of currently allocated storage, whichever is higher.

The amount of input/output operations per second (IOPS) for a given database is derived from the storage type you are using together with the amount of storage allocated. It is important to know what IOPS numbers your current storage supports, and you can define the CloudWatch metrics ReadIOPS and WriteIOPS to notify you if you are approaching that level to avoid an issue.

You can get additional IOPS by moving to faster storage or growing your storage footprint to a certain degree. If you exhaust those options and are certain that poor application coding is not leading to excessive read/write activity, it may be time to start thinking about moving to the Provisioned IOPS (PIOPS) storage type, which can provide a higher level of guaranteed I/O for an additional cost.

CloudWatch also offers metrics for ReadLatency, WriteLatency, and DiskQueueDepth for you to configure if you want to keep a closer eye on those parameters.

Monitoring Database Connections

The CloudWatch DatabaseConnections parameter lets you monitor the number of active connections to your database and can alert you when the value approaches the max_connections property for the database.

The default value for max_connections is derived from the total memory and is database-specific, so it is important to check the setting for each database. You can also modify the default value of this parameter if required.

As you can see, CloudWatch simplifies a number of key database monitoring and management tasks. But CloudWatch is just one of several DBA support options you can try on AWS Cloud. You can also subscribe to Amazon RDS events to be notified about changes to a database instance, leverage the Performance Insights dashboard to help analyze database issues, and more.

If your company is thinking of migrating your databases to a cloud or managed hosting provider, Buda Consulting can help you choose the best option for your workloads, and can act as your “first line of defense” when problems like server sprawl arise. We also offer “personalized” managed database environments for Oracle, SQL Server and MySQL workloads.

Contact us to schedule a free consultation today.

For more information:

Why You Should Not Thin Provision Your Production Storage

Why You Should Not Thin Provision Your Production Storage

Virtualization brought with it some other amazing technologies, one of which was thin provisioning for storage. Thin provisioning offers significant value by allowing an administrator to create a server with a large supply of storage, while actually allocating only what is currently needed.

Thin provisioning is a good option when used in a development environment or other scenario where only test data would reside on the virtual server. But there are some thin provisioning disadvantages for production environments because of the high potential for downtime and data loss.

Over-Subscription

Thin provisioning causes trouble when more storage is provisioned than is available in the underlying hardware. For instance, when you thin provision, you can store a 4 TB virtual disk on a 400 GB physical volume. As long as you use less than 400 GB of space inside the virtual volume, the setup works well. But what happens when you use 401 GB?

Over-subscription is what happens when you subscribe more storage than is available. This will cause I/O errors on your server, which can lead to irreparable damage to your virtual server and/or application. The damage is exacerbated in a production environment because it can result in partial or total data loss.

Two Ways To Remedy Over-Subscription

There are two approaches that help to remedy an over-subscription problem.

Increase Your Virtual Disk Repository

The first is to increase the over-subscribed virtual disk repository. This usually requires a SAN administrator to increase the repository volume in the SAN storage manager and rescan the volume via the SCSI bus.

Create A New Virtual Disk Repository

If you cannot extend the initial volume, the second approach is to create a new virtual disk repository and migrate some of your virtual servers to this new storage.

One of our Oracle clients is using Oracle virtualization software for storage. The original DBA (before we were involved), created 30 TB of thinly provisioned vdisks. But the entire size of the volume on which these vdisks were subscribed was only 20 TB. Everything worked fine until the total actual usage of those vdisks reached 20 TB, at which point I/O errors signaled that something was very wrong.

After gaining access to Oracle’s VM manager, we were able to see that the volume was oversubscribed. We corrected the problem using the second approach described above. We created a new disk repository and moved data files one at a time until everything was moved over and the space issue was resolved.

Conclusion

In summary, using thin provisioning can be wonderful if managed correctly with proper forecasting. But if not managed carefully, it can lead to negative disadvantages that outweigh the storage saving benefits.

If you’re using thin provisioning today, or looking for other ways to make the best use of your physical and virtual storage, contact Buda Consulting to talk over potential options and what’s right for your environment.