How We Delivered a 3500% Increase In Performance!

A major telecommunications vendor tracks every log entry (event) on every network device in their infrastructure.

These log entries are all inserted into one table so they can be used for correlation operations.

When we began working with the client, they were able to load at most 25 million events per day. These records were loaded by applications on four servers that prepared the data.

We conducted a careful analysis of the environment to systematically remove bottlenecks. Employing unix performance analysis tools, oracle statspack reports, Oracle TKPROF results, database design reviews, and application design reviews, we discovered and removed numerous issues that impacted insertion performance.

We made changes including redo log buffer modifications, storage infrastructure changes, table partitioning, I/O redistribution, and sql loader option changes, as well as application architecture changes. Each of these changes produced measurable improvement.

The results were impressive! The throughput improved from 25 M records per day to 900M records per day (4166/sec). A 3500 % increase in performance!

They Searched For Days, We Found It In One Hour

A major pharmaceutical firm had just completed a large software development project using Oracle.

After significant testing in the test and UAT environment, including volume testing, the project rolls out and goes into widespread use. Shortly after production, the users begin to complain about slow performance.

The development team is puzzled, the system works fine in UAT, and the production environment has the same amount of data, in fact, it is the SAME DATA, as the data that was present in UAT. The data was simply copied over into production for the launch.

How can the system be slower in production? After three days of working through all the possible causes that they could think of, the client calls Buda on Friday night to help resolve the problem.

By employing our systematic approach to finding and resolving bottlenecks, we quickly identify the problem, stale statistics.

While the DBA thought (insisted) that statistics were being refreshed each night, in actual fact, only the structure was being validated and the statistics were stale.

A quick revision and execution of the database maintenance scripts and the system returned to expected performance! What seemed to be a disaster for the implementation team turned out to be a simple change and all was well by Monday Morning!