by Robert Buda | Dec 8, 2017 | Backup and Recovery, Best Practices
Don’t let the Backup Grinch steal Christmas
I have written about this topic in the past but it seems that it can never be shared too many times. As we approach the holiday season, we don’t want to be recovering data when we should be spending precious time with family and friends. So I will share another cautionary tale about how what we don’t know can hurt us. And how we can improve the chances of an uninterrupted holiday celebration.
This past weekend, during a routine health check that we perform for one of our clients, one of our Oracle DBAs found that backups starting failing a few days prior. The client had no idea that there was any problem with the backup.
In this case, the database backup itself was successful, but the ongoing control file and archive log backups failed. Basically this means that in the event of a failure, they would be unable to restore any transactions that happened since the backup. This could result in a major loss of data.
This is another illustration of the critical importance of having a dedicated team whose responsibility it is to check the health of your database and your backups on a regular and frequent basis.
And now with my own special version of a holiday classic (a thousand apologies to Clement Moore), I wish you all healthy databases and consistent backups all through the year!
A Visit from The Buda Team
Twas the night before weeks-end, when all through production
Not a backup was running, there was an obstruction
The schedule had been set, so long ago now
No-one thinks they should check it, its just running somehow
And while you were nestled all snug in your bed
You had no way of knowing your backup was dead
But the Buda Team, ready to check out your logs
Are poised to find out that your system has hogs
Bad programs preventing the backup from storing
But now that we fixed it you can go back to snoring
In the morning we’ll tell you what happened last night
And with one great big sigh you’ll say “So glad we’re alright!”
Happy Holidays!
by Robert Buda | Oct 2, 2017 | Backup and Recovery, Best Practices
Planning for Disaster
As we think about the enormous cleanup effort taking place in the wake of
Harvey, Irma, and Maria, we are reminded of the importance of planning to keep your business up and running when disaster strikes. In this blog post I share the thoughts of an expert in the field of Business Continuity Planning and Disaster Recovery.
I recently had an interesting and fun conversation with Bob Cohen, Business Continuity and Disaster Recovery Practice Director at Pivot Point Security, about some key aspects of Business Continuity Planning and some cautionary tales about the risks of not doing so.
[table id=3 /]
Notes:
I hope you enjoyed the conversation that I had with Bob. PivotPoint and Buda work together to ensure that our client’s data assets are secure and protected against all types of threats.
If you have any questions about how to ensure that you are prepared to recover your databases in the event of a disaster, please give us a call at (888) 809-4803 x 700 and if you have further thoughts on the topic, please add comments.
by Robert Buda | Sep 5, 2017 | Backup and Recovery
How is a database backup like a portable generator with a gasoline leak? And why should you care on a beautiful sunny September morning? How does any of this relate to Business Disaster Recovery and Business Continuity planning and testing? Read On!
Lack of Preparation
Until Superstorm Sandy in 2012, we had lived in our NJ town for 25 years without ever having a power outage for more than a few minutes. I thought it would never happen, and never prepared for it. I had no portable generator or other storm supplies.
Then on October 8. 2012, I learned how important it is to anticipate and prepare for the the unexpected. We lost power for about 7 days when the storm ravaged our state. Even after the power went out, I thought for sure it would be back on at any moment. A transformer must have blown, I thought, or a tree came down on a wire, they just have to reroute the power! We don’t get power outages in Robbinsville!
By the time I learned that our sub-station was three feet under water and it would be days before we had power, not a generator was to be had anywhere in NJ. Even if Home Depot had any, we could not get out of our development to get there because of fallen trees. If only I had prepared…
I learned my Lesson
So this morning, for the 10th time in 5 years, on this beautiful Sunny Labor Day, I went out to perform my twice-yearly test of the Generac GP 5500 Generator that I purchased a few months after Sandy when they again became available. The generator sits in my garage, waiting to be hooked up the transfer switch that I installed when I purchased it.
Leakage
I pulled the generator into the driveway, opened the gas valve, and gasoline proceeded to leak all over my driveway! After pulling up a you-tube video, I learned the the cause was likely to be a stuck float in the generator. In order to fix it, I had to take apart the carburetor, and for that I needed a 3/8 inch socket with a 1/4 inch drive and a pivoting socket extension. Why am I telling you this? Because in the middle of a hurricane, you don’t want to run out to Home depot to get a 3/8 inch socket with a 1/4 inch drive and a pivoting socket extension which you likely do not have in your garage.
Are you as prepared as you think you are?
Back to my story, so since I don’t already have the aforementioned tool in my garage, I decide to ask my friend and neighbor if he happens to have something that might work for my task. He did not but while in his garage talking about generators, he told me about his nice new generator that he bought a few years ago that was still sitting in a box in the garage. He had never tested it or even removed it from the box! I thanked him for looking for the tool and then left him with some friendly advice to test the generator soon, and offered assistance to install a transfer switch to make the generator even more useful when the time comes. It made me wonder how many people have generators sitting in a box in their garage.
All Fixed
After a trip to Home Depot to get the proper tool, which I could not have taken during Sandy, I took apart and cleaned the carburetor, with the help of the you tube video, which I could not have watched during Sandy, and the generator started on the first pull, with no leakage! So now at least when it comes to power, I am better prepared for the next storm.
Back to Database Backups
So what does all this have to do with database backups? The generator sitting in the box in my neighbor’s garage is like a database backup sitting in your data center, but that you never tested. It makes you feel good that it is there, but when the wind is blowing and the rain is falling and the disks are crashing and the data center is filling up with water, you don’t really know that it is going to work.
However, a database backup in your data center (with an off-site copy) that gets tested, cleaned up, and maintained on a regular basis, is more like the generator sitting in my garage that is far more likely to be ready for the next storm.
Which database backup would you rather rely on?
If you would like to discuss how to ensure that your backups will work when you need them, give us a call at (888) 809-4803 x 700 and if you have further thoughts on the topic, please add comments!
If you enjoyed this article please like and share!
by Robert Buda | Aug 3, 2017 | Backup and Recovery, Best Practices, Database Architecture, Oracle
Architecting for Backup and Recovery Efficiency with Oracle RMAN
Database architecture is critical to achieving many business objectives. These include application performance, business continuity, security, and geographic distribution. This article will explore another often overlooked objective that can be influenced by the database architecture: backup optimization and recovery efficiency.
Very large databases are common in today’s business environment. Multi-terabyte databases are prevalent in all but the smallest organizations. Despite these large sizes, we tend to find that data that pertains to earlier time periods tends not to change, and tends to be used less frequently than more current data. When architecting the database, the age of the data, and therefore the possibility of data changing can be used as a factor in the physical database design that can optimize backup and recovery efficiency and performance.
RMAN Backup Optimization
It is a given that all data must be backed up. But taking that backup puts load on the database that impacts application performance during the time that the backup is running. Common approaches to mitigating this impact include backing up from a standby database rather than the production database, taking offline database backups when the application is not available, and restricting the backup time to non-peak times so the machine resource usage is minimized.
However, in some environments, those options are not available. The first option, backup up from a standby database, may not be an option of you don’t have a high availability environment. Bringing the database down is not an option in a 24×7 production environment. And there are many databases that are so large that the time it takes to back up the entire database is simply too long and exceeds the non-peak times for the application.
Partitioning and Archiving
Another technique that may be used is to build partitioning and archiving into the database architecture. We can partition the data into physically separate tablespaces, and place each partition into a separate tablespace. This allows us to isolate data from past time periods that are kept for archiving purposes but are not frequently queried and are never going to be updated. These separate tablespaces can be backed up when the data reaches the point that it will not be changed again, and then it can be excluded from the normal backup routine. In many databases, older data represents a very large portion of the overall database, so such a scheme can significantly reduce the backup time, thereby significantly reducing the impact on the application.
There are a number of ways to exclude tablespaces from the backup after they have reached the point where they will not be updated again, including:
- Making the tablespaces readonly, and configuring Backup Optimization in Oracle RMAN. After this, RMAN will backup the the tablespace enough times to satisfy the retention policy and then will exclude them on subsequent backups.
- Using the RMAN command CONFIGURE EXCLUDE FOR TABLESPACE command. Once configured, the specified tablespace will be excluded from future backups. These tablespaces can be manually included explicitly in other backup sets to ensure that the data is backed up but they can be excluded from full backups.
Here is an example of how we might use this: lets say that we have an Oracle Database Application that collects traffic sensor data. Each day we collect a large set of data from traffic sensors from municipalities around the country. We have very large tables that contain hundreds of datapoints per sensor. Each table contains hundreds of gigabytes of data stretching back 10 years. The tables are partitioned so that a new partition is created for each month, and as the data is collected, it is automatically placed into the proper partition. At the beginning of each year, we can take a single backup of all the tablespaces that hold the data from the prior year. We know that data will never change, so we do not have to include those tablespaces in future backups. We can set these tablespaces as readonly, and then with backup optimization turned on, RMAN will then exclude them from subsequent backups, but will still enforce the backup retention policy so you wont lose backup sets that are necessary to restore those tablespaces. An added benefit is that the backup set each week will be significantly smaller thereby reducing disk requirements for the ongoing backup sets.
Restore Efficiency
In addition to significantly reduced backup time, partitioning the data in this way also improves the efficiency of the restore process because if one partition fails, the others do not need to be restored. This can result in significant time savings during a restore.
Other Benefits
There are other benefits to partitioning your data beyond the improvements to the backup and restore process. By separating older data which typically does not change, and is accessed less frequently, from the newer data, we have the ability to place the older data on less costly media. And regardless of the media type, there are performance benefits to separating data onto different drives/controllers (particularly useful when using separate storage arrays as opposed to SAN environments).
Thinking Ahead
When architecting database, think about what the impact of the backup and RMAN database recovery process will look like after 10 years. Architecting the backup and restore efficiency into the database design at that time will save lots of redesign later on.
If you are struggling with cumbersome backup optimization and restore processes or are about to do a database design or redesign, please give us a call at (888) 809-4803 x 700 and if you have further thoughts on the topic, please add comments!
If you enjoyed this article please like and share!
by Robert Buda | Jul 21, 2017 | Best Practices, Cloud
“Lets Move to The Cloud!”
We often hear people talk about moving their business or their data center or their software to “The Cloud“. For many, this concept seems confusing and vague.
That’s because The Cloud does not exist!
There is no “The Cloud”. In reality, there are many clouds. And therefore we can’t just decide to move to “The Cloud”.
Instead, there are many clouds with services being offered by many vendors. A cloud at its core is a collection of hardware and software combined by a vendor into a service offering that provides some level of computing service for their customers. Depending on our risk tolerance, bandwidth requirements, data custody and security requirements, level of technical expertise, and business model, one or more of these levels may make sense for our organization. These levels are known as Infrastructure as a Service, Platform as a Service, and Software as a Service. The table below describes these levels in more detail. The technical reader will recognize that these levels are fuzzy and that the things that are included in each level and the things that we control in each level can vary from vendor to vendor but this table gives us a sense of each level.
[table id=1 /]
A source of confusion when thinking of the cloud is that it is often thought of as an external organization abstracting away the underlying technical details of our computing environment. For example, PAAS (Platform as a service) offerings abstract away everything from the physical hardware up through the operating system, leaving us to have to manage only the software frameworks we are using, which may be database management systems like Oracle, or software development platforms like Visual Studio. But in reality, it is the abstraction that is the essence of a cloud, not the fact that it is an external party providing it. Therefore we can have on-premises clouds hosted at our own data center, and private clouds managed by our own team but that are hosted at external data centers. It is a stack of software that provides the abstraction that makes it a cloud, not the vendor. The foundation of this stack is mostly virtualization and automation related software.
Journey into the Clouds
Jumping right into the clouds is difficult and scary. We can’t see things clearly with all these clouds around. We don’t know what and who we can trust.
The good news is that we can take advantage of some of the huge benefits of cloud computing without some of the riskier aspects. When we run a private cloud or on-premises cloud, we still benefit from the virtualization and automation when provisioning servers, databases, etc, while minimizing the risk that may be introduced by using shared services or relying on external vendors. Additionally, if we transition our software to use our own private cloud services, they will be much further along when it comes time to move to public cloud services in the future.
There are other options to make the journey less scary as well: Some vendors are providing ways to simplify the process of taking meaningful steps toward the public cloud while staying on premises. Oracle offers the Oracle Cloud Machine, a machine that can live in our own data center, offer IAAS and PAAS capabilities, and is installed and managed by Oracle, behind our own firewalls. When we are comfortable moving to the public cloud, the entire environment can be picked up and moved to Oracle’s public cloud. And Microsoft has just announced the Azure Stack. This will enable us to use the same software stack that is running in the Microsoft Public Azure cloud, but we can run it in our own data centers, on our own hardware. Again, after transitioning our software to use cloud services, a future shift to the Azure public cloud will be greatly simplified.
Clouds are not one size fits all
So when we think about how to transition to cloud based technology, we should stop thinking about moving to “The Cloud” because that is too simplistic. We need instead to look at each component of our IT services and think about what level of computing resources we would like abstracted away for that component, and then choose from the available clouds that provide that level of service.
For example, we may decide that for our Customer Relationship Management System, SAAS is the proper level because we are comfortable with the cloud vendor providing all of the IT administration, Disaster Recovery, and Security services, but for our Chemical Inventory Management system that holds highly sensitive formula information, we may choose to go with a PAAS solution or even an IAAS solution because we want to have more control over network and data security. And for a Financial Trading System we may insist on a private cloud IAAS solution so we have full control over all aspects of redundancy, connectivity, and security.
Get your head out of “The Cloud”
We are all thinking about the Cloud these days. I heard a talk by the great physicist and author Michio Kaku recently who predicts that through Artificial Intelligence and technology that can read and write our memories, we will all essentially think in “The Cloud” some day.
But for our businesses today, we have to think about the individual clouds so that we don’t get lost. For each service being provided to our employees, partners, customers, regulators, etc, we must think about the appropriate level of service and abstraction (IAAS, PAAS, SAAS), and then evaluate offerings of the cloud vendors at that level.
So when we think about “The Cloud”, we must think instead of “The Clouds”. And we may see things a bit more clearly.
If you would like to discuss more about your Journey into The Clouds please give us a call at (888) 809-4803 x 700 and if you have further thoughts on the topic, please add comments!
If you enjoyed this article please like and share!
