Watch Your FRA!

Watch Your FRA!

When it comes to Oracle database administration, one of the most revered parts of your database structure is the fast recovery area (FRA). This is an Oracle managed area where DBAs usually store some of the following files:

  • Redo logs
  • Archive logs
  • Flashback logs
  • Control files
  • RMAN backups

The purpose of the Oracle FRA is to simplify database recovery. The Oracle database process automatically manages items stored in the FRA and will delete items that are no longer needed. 

Oracle FRA Settings

Now the Oracle FRA may sound like a magical area that should never run into storage-related issues—but that could not be farther from the truth. Even though the Oracle database process will manage items and attempt to delete files that aren’t needed, the DBA also has to be aware of instance settings that may block Oracle from being able to remove the files. Some settings that can cause storage issues with your FRA include:

  • RMAN backup retention setting – If you set your backup retention to store two weeks’ worth of RMAN backups, but your FRA fills up to 100% before any backups can be purged, this will cause your database to halt.
  • RMAN archive log deletion policy – If you set the deletion policy to delete archive logs after they are applied to all standby databases, but haven’t noticed that your primary and standby databases have been out of sync for a long period of time, your FRA can fill to 100% and cause your database to halt.
  • RMAN archive log backup copies setting – By default, backup copies are set to 1. But what if you want to make sure your backups contain more copies of your archive logs in the event that one of your incremental backups became corrupted? When you set this setting higher than 1, you will not be able to delete any archive logs unless they have been backed up however many times this setting is set to. So if you set this option to 3, you will need to have taken at least three backups of each archive log before said log can be deleted from your system. If you opted to store archive logs in your FRA, then this can fill the FRA to 100% and cause your database to halt.
  • Db_flashback_retention_target setting – If you have enabled the flashback database option this is stored in the FRA by default. As with the archive logs, depending on the time value of the setting, it will store all flashback logs needed to guarantee that you can flashback your database as per the setting. If you set this to a high setting, this can fill the FRA to 100% and cause your database to halt.

Those are just a handful of the many ways you can accidentally fill your Oracle FRA, which is why you need to make sure that your FRA is adequately sized to store all files as per all retention settings. You should also create a script that queries the v$recovery_area_usage and have this result sent to the email of all DBAs, as this will tell you how much of your FRA is used and what in particular is taking up the space:

For remote and onsite DBA support to help keep your databases running smoothly, including 24×7 live support, contact Buda Consulting.

Schemas, Databases and Instances—Defined and Discussed

What is a schema as opposed to a database as opposed to an instance? And how do schemas differ between Oracle and MSSQL? Or between Oracle and PostgreSQL? Or between PostgreSQL and MongoDB?

These terms can be confusing, but they are very important when planning a database architecture. So let’s define these terms and discuss conceptually how they are similar and how they differ between database software implementations. This post will focus mostly on schemas, with some references to the other terms for context. 


What is a database ?

A database is the collection of database files that contain the data being stored. These files hold both the user data and the metadata (data dictionary) that the database needs to make sense of the user data.  The metadata includes the schema definitions (where applicable) as described below.


What is a database instance?

A database instance is the collection of all of the database software processes plus any memory structures required by those processes, plus the database files where the database data is stored. (See diagram)

The different software vendors treat the relationship between databases and instances in different ways.

Oracle supports one database per instance unless you are working with 12c and above and using Oracle Multitenant.

PostgreSQL supports multiple databases per instance.  Some system catalogs are shared across all databases in an instance.

MSSQL supports multiple databases per instance. Each instance has a set of system databases that are shared across all databases served by that instance. 

MongoDB supports multiple databases per instance.


What is a schema?  

The concept of a schema can be a little confusing because there are three different relevant uses of the word “schema” in the context of an IT project. 

    1. Mirriam Webster defines a schema as “a structured framework or plan, an outline.”  
    2. In the realm of database technology, a schema means a structural definition of the data that you are storing. This essentially defines the datatypes of the data you are storing, and the organization of that data (into tables, documents, indexes and constraints, etc). This can be expressed in the form of a diagram such as an entity relationship diagram (ERD), or in a set of data dictionary language (DDL) statements, or in a JSON object. 
    3. Some database vendors have extended the concept of a schema to include not just a definition of the structure of a set of data, but also a particular collection of objects that contain the data (tables, etc), and even the data itself. This is sometimes a named collection and is typically based on one of these factors: 
      • Who owns the objects (a database user)
      • Who should have access to the objects (e.g., a database role that may be assigned to users)
      • What the objects are used for (e.g., all objects for a given application or function within an application)   

When implemented in this fashion, a schema can also be thought of as a namespace. An object can have the same name in two different schemas and the two objects will be distinct from each other. 

It is interesting to note that MongoDB, which is a document database as opposed to a relational database, is sometimes called a schema-less database. MongoDB also has the concept of a schema, but it is purely a description of the structure of the data, more like definition 2 above than 3. A MongoDB schema does not represent the actual instance of the data, as it does with the relational databases mentioned. 

To summarize, within the context of database management software, a schema is either a set of objects that contain data that is related in some logical way (user, access, application), or simply a definition of the structure of data. 

Examples

Here are some example of schemas (see the diagram below):

    1. Schema JSMITH:  A schema that contains all of the tables that belong to user Jsmith. This schema would typically simply be named the same as the user, and is often created automatically when that database user account is created. When the user connects to the database, this will typically be his default schema. So any objects that he creates will automatically be part of that schema. When he issues a query, unless he specifies a schema name as part of the name of the object he is querying, or changes his schema search path (this is done differently by each database vendor),  the result set will come from the object by that name that exists in his default schema. 
    2. Schema PAYROLL: A schema that contains all of the tables for the payroll application. This schema would typically be named for the application or a functional area within the application. When accessing data from the Payroll schema, users will need to either set their schema search path to the Payroll schema, or prefix all object names in the query with PAYROLL.
    3. Schema DBO: This built-in schema in MSSQL is the default for all users unless otherwise specified. In many SQL Server databases, almost all objects end up here.  This is similar to the public schema in PosgtreSQL.


How does a schema differ between database vendors? 

MSSQL and PostgreSQL have an actual object in the database called a schema. You can create and drop a schema, and you can assign access rights and ownership to a schema as a whole. In these environments, there is a loose connection between a schema and a database user. A schema may be owned by a database user. But a database user does not have to own any schemas. A schema may also be owned by a role instead of an individual user. If you want to drop a user that owns a schema, they can and must first transfer ownership of the schema to another user. 

Oracle has the concept of a schema but it does not really have an object in the database called a schema. It is more conceptual. In Oracle, each database user may be an owner of objects, and the collection of objects owned by a given user is considered a schema. If a database user is dropped, all objects owned by that user (in that user’s schema) must be dropped first. A schema in Oracle does not exist independently of a database user. There is a command in Oracle called Create Schema that is essentially a wrapper that lets you create a database user and a set of objects to be owned by that user all at once.  The Drop Schema command is really the same thing as Drop User. One cannot transfer ownership of an object from one user to another. The new user would need to recreate the object. (A CTAS query may be helpful here).

MongoDB, as mentioned earlier, uses the concept of a schema in database design and in the validation of the structure of incoming data. (Nice blog on this here.) But there is no object in the database known as a schema. 

I hope this post helps pull together the concept of a schema and the way the different vendors have implemented schemas.

Please comment with any questions or examples that you think might be helpful, including for database vendors that are not listed here. Also, if you disagree with the way I defined schema, please let me know how you see it.

To talk over any questions you may have around schemas or database architecture in general, contact Buda Consulting.

Which Edition of Oracle RDBMS is Right for My Business?

Which Edition of Oracle RDBMS is Right for My Business?

Do I really need Oracle Enterprise Edition? Do I need Oracle Enterprise Edition in order to have a standby database? What version of Oracle do I need?  

We get questions like these a lot. Oracle’s editions and features can be a bit overwhelming. This post shares some key information that I hope will make your choice easier.

To keep this post to a manageable length, this post deals only with Oracle Version 12c and Version 19c, and excludes Oracle cloud offerings. A future post will cover those feature sets. 

Oracle Standard Vs. Enterprise

For each version covered below, I took the feature set from Oracle’s licensing documentation and listed those things that are available only in the Enterprise Edition of Oracle, and not in any of the Standard Editions.

But before we look at the specific features available in each version, let’s explore two important categories of features:

Data Guard

One of the more frequently used options in Enterprise Edition is Data Guard. A set of Data Guard features is listed in the first table for each edition below. These are used to implement high availability; that is, having a standby database available that is kept up to date with your primary production database in case your primary database goes down.

If Data Guard related features are the only set of “Enterprise-only” features that you need, then you may still be able to use Oracle’s Standard Edition. Here’s why:

DBvisit Standby, an alternative to Oracle’s Data Guard, provides an active standby database using any version of Oracle. You will still need to license the database software on both the primary and the standby server, but both can be much less expensive Standard Edition licenses. 

Oracle Security

Oracle has a number of important security options. 

The advanced security option is an extra cost option that is only available with Enterprise Edition.  This includes transparent data encryption, data redaction, data masking and subsetting (very helpful for development environments), and other security options. To adequately secure sensitive data at the database level, we advise that you use advanced security, and thus Oracle Enterprise Edition. 

With that said, if you are not using advanced security, and the only Enterprise-only option you need is Data Guard, then we recommend taking a good look at Dbvisit.

Oracle Version 12c Features

Below are the features that are only available in Enterprise Edition for Oracle 12c. If you need anything on this list, then you need Oracle Enterprise Edition. Period! 

Category Feature
Data Warehousing and Business Intelligence Advanced Index Compression
Data Warehousing and Business Intelligence Automatic Data Optimization
Data Warehousing and Business Intelligence Basic Table Compression
Data Warehousing and Business Intelligence Bitmapped index, bitmapped join index, and bitmap plan conversions
Data Warehousing and Business Intelligence Deferred Segment Creation
Data Warehousing and Business Intelligence Exadata Flash Cache Compression
Data Warehousing and Business Intelligence Heat Map
Data Warehousing and Business Intelligence Hybrid Columnar Compression
Data Warehousing and Business Intelligence Hybrid Columnar Compression Row-Level Locking
Data Warehousing and Business Intelligence In-memory Parallel Execution
Data Warehousing and Business Intelligence Oracle Advanced Compression
Data Warehousing and Business Intelligence Oracle OLAP
Data Warehousing and Business Intelligence Oracle Partitioning
Data Warehousing and Business Intelligence Parallel capture and apply via XStream
Data Warehousing and Business Intelligence Parallel Data Pump Export/Import
Data Warehousing and Business Intelligence Parallel index build/scans
Data Warehousing and Business Intelligence Parallel query/DML
Data Warehousing and Business Intelligence Parallel Statement Queuing
Data Warehousing and Business Intelligence Parallel statistics gathering
Data Warehousing and Business Intelligence Prefix Compression (also called Key Compression)
Data Warehousing and Business Intelligence Summary management—Materialized View Query Rewrite
Data Warehousing and Business Intelligence Transportable tablespaces, including cross-platform and full transportable export and import
High Availability Application Continuity
High Availability Automatic Block Repair
High Availability Block change tracking for fast incremental backup
High Availability Block-level media recovery
High Availability Cross-platform Backup and Recovery
High Availability Duplexed backup sets
High Availability Fast-start fault recovery
High Availability Flashback Database
High Availability Flashback Table
High Availability Flashback Transaction
High Availability Flashback Transaction Query
High Availability Global Data Services
High Availability Lost Write Protection
High Availability Online Datafile Move
High Availability Online index rebuild
High Availability Online index-organized table organization
High Availability Online table redefinition
High Availability Oracle Active Data Guard
High Availability Oracle Data Guard—Far Sync Standby
High Availability Oracle Data Guard—Real-Time Cascading Standbys
High Availability Oracle Data Guard—Redo Apply
High Availability Oracle Data Guard—Snapshot Standby
High Availability Oracle Data Guard—SQL Apply
High Availability Parallel backup and recovery
High Availability Recovering tables and table partitions from RMAN backups
High Availability Rolling Upgrade Using Active Data Guard
High Availability Rolling Upgrades—Patch Set, Database, and Operating System
High Availability Tablespace point-in-time recovery
High Availability Transaction Guard
High Availability Trial recovery
High Availability Unused block compression in backups
Integration Advanced Replication
Integration Messaging Gateway
Integration Sharded Queues
Manageability Database Resource Manager
Manageability Oracle Cloud Management Pack for Oracle Database
Manageability Oracle Data Masking and Subsetting Pack
Manageability Oracle Database Lifecycle Management Pack for Oracle Database
Manageability Oracle Diagnostics Pack
Manageability Oracle Fleet Patching and Provisioning

(formerly Rapid Home Provisioning)

Manageability Oracle Real Application Testing
Manageability Oracle Tuning Pack
Manageability SQL Plan Management
Manageability SQL Tuning Sets
Networking Infiniband Support
Networking Network Compression
Networking Oracle Connection Manager
Performance Adaptive Execution Plans
Performance Attribute Clustering
Performance Client Side Query Cache
Performance Concurrent Execution of UNION and UNION ALL Branches
Performance Database Smart Flash Cache
Performance Fault Tolerant In-Memory Column Store
Performance In-Memory Aggregation
Performance In-Memory Column Store
Performance PL/SQL Function Result Cache
Performance Query Results Cache
Performance Support for Oracle Exadata Storage Server Software
Performance TimesTen Application-Tier Database Cache
Performance Zone Maps
Scalability Oracle NoSQL Database Basic Edition
Scalability Quality of Service Management
Security Enterprise User Security
Security Fine-grained Auditing
Security Oracle Advanced Security
Security Oracle Database Vault
Security Oracle Label Security
Security Privilege Analysis
Security Real Application Security
Security Redaction
Security Transparent Sensitive Data Protection
Security Virtual Private Database
Snapshots and Cloning Storage Snapshot Optimization
Spacial Multimaster replication of SDO_GEOMETRY objects
Spacial Parallel spatial index builds
Spacial Partitioned spatial indexes

Now we’ll look at features that are only available in certain versions of Oracle 12c Standard Edition, or that are limited in Standard Edition. The “Notes” column explains the limitations.

Category Feature SE1 SE/SE2 EE Notes
Data Warehousing and Business Intelligence Oracle Machine Learning (formerly Advanced Analytics) N Y N Included with SE2 and EE

Not included with SE1 and SE

OML4R (Oracle R Enterprise) Server is limited to operation on Oracle Linux 6 or higher

High Availability Optimization for Flashback Data Archive N Y N Basic Flashback Data Archive is in all editions. Optimization for Flashback Data Archive requires EE and the Oracle Advanced Compression option.
Integration Basic Replication Y Y N SE/SE1/SE2: read-only, updateable materialized view
Integration Oracle Streams Y Y N SE/SE1/SE2: no capture from redo
Manageability Instance Caging N Y N Included with SE2 and EE

Not included with SE1 and SE

Spacial Oracle Spatial N Y N Included with SE2 and EE

Not included with SE1 and SE

Spacial RDF Graph N Y N Included with SE2 and EE

Not included with SE1 and SE

If you need any of the above features, then you may need Oracle Enterprise Edition, depending on the specifics as listed in the notes.

That’s it!

Your DBA team should be able to tell you if you need/use these Enterprise-only features. 

If you don’t need any of these features, then potentially you can save many thousands of dollars by using the Standard Edition instead. 

Oracle version 19c Features

OK, that’s it for 12c. Oracle 19c is essentially the same but with a somewhat different, more “evolved” set of options. The considerations mentioned above remain valid. 

First, let’s look at the 19c features that definitely require Enterprise Edition:

Category Feature
High Availability Application Continuity
High Availability Oracle Sharding
High Availability Oracle RAC One Node
High Availability Oracle Data Guard—Redo Apply
High Availability Oracle Data Guard—Far Sync Standby
High Availability Oracle Data Guard—SQL Apply
High Availability Oracle Data Guard—Snapshot Standby
High Availability Oracle Data Guard—Real-Time Cascading Standbys
High Availability Oracle Active Data Guard
High Availability Rolling Upgrades—Patch Set, Database, and Operating System
High Availability Rolling Upgrade Using Active Data Guard
High Availability Online index rebuild
High Availability Online table organization
High Availability Online table redefinition
High Availability Duplexed backup sets
High Availability Block change tracking for fast incremental backup
High Availability Unused block compression in backups
High Availability Block-level media recovery
High Availability Lost Write Protection
High Availability Automatic Block Repair
High Availability Parallel backup and recovery
High Availability Tablespace point-in-time recovery
High Availability Trial recovery
High Availability Fast-start fault recovery
High Availability Flashback Table
High Availability Flashback Database
High Availability Flashback Transaction
High Availability Flashback Transaction Query
High Availability Optimization for Flashback Data Archive
High Availability Online Datafile Move
High Availability Transaction Guard
High Availability Cross-platform Backup and Recovery
High Availability Global Data Services
High Availability Recovering tables and table partitions from RMAN backups
Integration Messaging Gateway
Integration Sharded Queues
Manageability Oracle Cloud Management Pack for Oracle Database
Manageability Oracle Data Masking and Subsetting Pack
Manageability Oracle Database Lifecycle Management Pack for Oracle Database
Manageability Oracle Diagnostics Pack
Manageability Oracle Tuning Pack
Manageability Oracle Real Application Testing
Manageability Database Resource Manager
Manageability SQL Tuning Sets
Manageability Oracle Fleet Patching and Provisioning (formerly Rapid Home Provisioning)
Networking Infiniband Support
Networking Oracle Connection Manager
Networking Network Compression
Performance Client Side Query Cache
Performance Query Results Cache
Performance PL/SQL Function Result Cache
Performance Oracle TimesTen Application-Tier Database Cache
Performance Database Smart Flash Cache
Performance Adaptive Execution Plans
Performance Concurrent Execution of UNION and UNION ALL Branches
Performance Oracle Database In-Memory
Performance In-Memory Column Store
Performance In-Memory Aggregation
Performance Attribute Clustering
Scalability Oracle Real Application Clusters (Oracle RAC)
Scalability Oracle NoSQL Database Basic Edition
Security Column-Level Encryption
Security Tablespace Encryption
Security Oracle Advanced Security
Security Oracle Database Vault
Security Oracle Label Security
Security Enterprise User Security
Security Centrally Managed Users
Security Fine-grained Auditing
Security Privilege Analysis
Security Real Application Security
Security Redaction
Security Transparent Sensitive Data Protection
Security Virtual Private Database
Snapshots and Cloning Storage Snapshot Optimization
Spatial and Graph Data Parallel spatial index builds
Spatial and Graph Data Multimaster replication of SDO_GEOMETRY objects
Spatial and Graph Data Partitioned spatial indexes
VLDB, Data Warehousing, and Business Intelligence Oracle Partitioning
VLDB, Data Warehousing, and Business Intelligence Oracle On-Line Analytical Processing (OLAP)
VLDB, Data Warehousing, and Business Intelligence Oracle Advanced Compression
VLDB, Data Warehousing, and Business Intelligence Advanced Index Compression
VLDB, Data Warehousing, and Business Intelligence Prefix Compression (also called Key Compression)
VLDB, Data Warehousing, and Business Intelligence Hybrid Columnar Compression
VLDB, Data Warehousing, and Business Intelligence Hybrid Columnar Compression Row-Level Locking
VLDB, Data Warehousing, and Business Intelligence Heat Map
VLDB, Data Warehousing, and Business Intelligence Automatic Data Optimization
VLDB, Data Warehousing, and Business Intelligence Basic Table Compression
VLDB, Data Warehousing, and Business Intelligence Deferred Segment Creation
VLDB, Data Warehousing, and Business Intelligence Bitmapped index, bitmapped join index, and bitmap plan conversions
VLDB, Data Warehousing, and Business Intelligence Parallel query/DML
VLDB, Data Warehousing, and Business Intelligence Parallel statistics gathering
VLDB, Data Warehousing, and Business Intelligence Parallel index build/scans
VLDB, Data Warehousing, and Business Intelligence Parallel Data Pump Export/Import
VLDB, Data Warehousing, and Business Intelligence In-memory Parallel Execution
VLDB, Data Warehousing, and Business Intelligence Parallel Statement Queuing
VLDB, Data Warehousing, and Business Intelligence Parallel capture and apply via XStream
VLDB, Data Warehousing, and Business Intelligence Transportable tablespaces, including cross-platform and full transportable export and import
VLDB, Data Warehousing, and Business Intelligence Summary management—Materialized View Query Rewrite

If you need any of the above, the you must use Oracle Enterprise Edition 

Now we look at the features that are included in the Oracle 19c Standard edition, but are limited.

Category Feature Notes
Consolidation Oracle Multitenant – # of PDBs The number value in each column indicates the maximum number of pluggable databases (PDBs) that can be created for each offering.

For all offerings, if you are not licensed for Oracle Multitenant, then you may have up to 3 user-created PDBs in a given container database at any time.

Manageability SQL Plan Management SE2: Only one SQL plan baseline per SQL statement is allowed and SQL plan evolution is disabled. Limits on baseline capture methods as well.

The following DBMS_SPM functions and procedures are not allowed: CONFIGURE, LOAD_PLANS_FROM_AWR, LOAD_PLANS_FROM_SQLSET, and all functions and procedures associated with SQL plan evolution.

If you are not using pluggable databases, or if three databases are enough, and if you are not using sophisticated plan management, then you can likely use Standard Edition.

If you know of any other enterprise-only options that are not listed here, or if you think I misstated anything, please comment.  I’d like to hear from you. Also, if you know of other good disaster recovery alternatives, let me know.

If you are not sure about any of this, or you just want to double-check your thought process, leave a comment or give me a call and I will be happy to have a conversation with you about it.

Oracle Active Data Guard Replication Error with Cisco Firewall: Root Cause and Workaround

Oracle Active Data Guard Replication Error with Cisco Firewall: Root Cause and Workaround

One of our customers had issues over the past several months with Oracle Active Data Guard replication to a standby server. The database on the standby server would intermittently fall out of sync with the database on the primary server, and log files were not consistently shipping to the standby server. 

Active Data Guard Replication Error

The Oracle database version was 12.2.0.1 running on Oracle Linux 6. The firewall was a Cisco ASA-5585-SSP-40, and the ASA version is 9.6(4)8. 

TNS tracing showed: CORRUPTION DETECTED: In redo blocks starting at block #…

By the time I got involved, the firewall administrators had already implemented all the recommended firewall changes to disable the following:

    •     SQLNet fixup protocol 
    •     Deep Packet Inspection (DPI
    •     SQLNet packet inspection 
    •     SQL Fixup

The following errors were noted in the primary database alert log:

    •     ORA-16055: FAL request rejected 
    •     ARC6: Standby redo logfile selected for thread 2 sequence 33351 for destination LOG_ARCHIVE_DEST_2 
    •     ARC6: Attempting destination LOG_ARCHIVE_DEST_2 network reconnect (12152) 
    •     ARC6: Destination LOG_ARCHIVE_DEST_2 network reconnect abandoned 

The following errors were noted in the standby database alert log:

    •     CORRUPTION DETECTED: In redo blocks starting at block #…
    •     RFS[20]: Possible network disconnect with primary database
    •     Error 1017 received logging on to the standby 
    •     FAL[client, USER]: Error 16191 connecting to…
    •     ORA-16191: Primary log shipping client not logged on standby 

The root cause of the problem turned out to be a bug in the Cisco firewall.  For reasons unknown, when the primary and standby database listeners were using port 1521, the firewall would ignore the settings the admins had implemented for the Oracle Data Guard connections and revert to the default settings. As a workaround, we changed to a different port.

If you are experiencing intermittent or hard-to-diagnose database issues in your environment, contact Buda Consulting.

Time to Validate Your Database Backup

Another new client, Another bad backup

I originally wrote this post in July but I am updating it today because it is so important.  Yesterday we did an initial Oracle Database Health Check for another new client and found yet again that their backups are useless. This time the client was taking backups regularly, but was not preparing the database first and was just copying the database files. Unfortunately this renders the backup useless.

Here is the original post:

One of the first things that we do when engaging a new client for Oracle or SQL Server Database Administration support is to do an inventory of their current databases and to validate that they have a good backup and recovery scheme in place.  The vast majority of the time, we find that there are either no backups being taken at all, or that not all of the data and configuration files are being backed up properly, despite the fact that the client thinks they are.

Often, there was a proper backup in place at some time in the past, but over time, the configuration of the system changed rendering the backup useless. These changes include changes in disk configuration, addition of data files, and other changes, that were made at the operating system level, configured properly in the database, but never modified in the backup configuration.

It is a simple oversight that can have catastrophic effects. We recently spent over three months helping a new client recover many years worth of data that they thought they were backup up properly, but they were wrong!

A regularly scheduled review of the backup scheme and a test recovery is the best way to ensure that your Oracle or SQL Server database is being properly backed up.

Contact Buda Consulting today for a comprehensive Disaster Recovery review of your Oracle or SQL Server Databases.