Be Very Afraid: An interview with the CTO of database security at McAfee

Follow @Oratweets

As published in the 105th issue of the NoCOUG Journal (February 2013)

Slavik Markovich is vice president and chief technology officer for Database Security at McAfee and has over 20 years of experience in infrastructure, security, and software development. Slavik co-founded Sentrigo, a developer of leading database security technology that was acquired by McAfee in April 2011. Prior to co-founding Sentrigo, Slavik served as VP R&D and chief architect at db@net, a leading IT architecture consultancy. Slavik has contributed to open-source projects, is a regular speaker at industry conferences, and is the creator of several open-source projects like FuzzOr (an Oracle fuzzer) and YAOPC (Yet Another Oracle Password Cracker). Slavik also regularly blogs about database security at www.slaviks-blog.com.

Down in the street little eddies of wind were whirling dust and torn paper into spirals, and though the sun was shining and the sky a harsh blue, there seemed to be no color in anything except the posters that were plastered everywhere. The black-mustachio’d face gazed down from every commanding corner. There was one on the house front immediately opposite. BIG BROTHER IS WATCHING YOU, the caption said, while the dark eyes looked deep into Winston’s own. … Behind Winston’s back the voice from the telescreen was still babbling away about pig iron and the overfulfillment of the Ninth Three-Year Plan. The telescreen received and transmitted simultaneously. Any sound that Winston made, above the level of a very low whisper, would be picked up by it; moreover, so long as he remained within the field of vision which the metal plaque commanded, he could be seen as well as heard. There was of course no way of knowing whether you were being watched at any given moment. How often, or on what system, the Thought Police plugged in on any individual wire was guesswork. It was even conceivable that they watched everybody all the time. But at any rate they could plug in your wire whenever they wanted to. You had to live—did live, from habit that became instinct—in the assumption that every sound you made was overheard, and except in darkness, every movement scrutinized.—1984 by George Orwell

Is my financial and medical information safe from the bad guys? After watching Die Hard 4, I’m not so sure, because it seems that bad guys can access, change, or erase anybody’s information with a few keystrokes.

Although life is not a movie, and the situation is not quite as bad as Die Hard 4, it is not that good either. You can read about breaches with varying degrees of severity every week. While the “bad guys” require a bit more than a few keystrokes to access/change information, they have very sophisticated tools at their service. World-spanning global botnets, automated hacking tools, a flourishing underground market, and a strong financial incentive all motivate the “bad guys” to continue breaking into systems.

On the flipside, there have been many significant changes and improvements to the applicable regulations associated with protection of PHI and ePHI healthcare information. In addition, the enhanced enforcement of HIPAA, and the newer HITECH, regulations has increased the visibility of—and, arguably, attention to—affected organizations complying with these regulatory mandates. SOX, GLBA, and other financial regulations are intended to address the integrity and authenticity of financial records. So, the organizations keeping your records are forced to think about security.

I would also add that it isn’t always “the bad guys” that cause data compromise—sometimes it’s caused accidentally, either by human, or system(s), error. To summarize, if you are being targeted, I’d say that there is a pretty good chance that the hackers will succeed in compromising your details. On the other hand, your liability is limited, at least on the financial front.

Why is information security so poor in general? Is it because administrators and users—me included—are clueless about information security, or is it because the operating systems, databases, networks, languages, and protocols are inherently vulnerable, which makes our task much harder than it really ought to be?

Indeed, there is a big awareness issue when it comes to security. Users, developers, and administrators generally lack deep understanding of security and, as everybody knows, security is only as strong as your weakest link. The “bad guy” just needs one successful try on a single attack vector, while the security protections need to cover all bases, all the time. It’s an asymmetric game where currently the “bad guys” have the advantage.

When specifically talking about “database security,” the reality is that the overall risk posture for these systems, and the often highly sensitive and/or business-critical information they contain, is most often grossly underestimated by the respective organizations. A comparison can be made to what the famous 1930s bank robber Willie Sutton was often quoted as saying, when asked by a reporter why he robbed banks: “Because that’s where the money is.” The “bad guys” often target these databases, and the valuable data assets they contain, because they know that’s where they can get the biggest bang for their buck (i.e., the highest return for their exploit efforts).

Also, the associated risk to them of being caught and subsequently penalized is very often quite low combined with the associated payoff (return) being quite high. So from an ROI perspective, their motivating rationale is abundantly clear.

Finally, if you were indeed “clueless” about security, you probably wouldn’t be asking these types of targeted questions.

The analogy is that certain cars are the favorites of car thieves because they are so easy to break into. Why are salted password hashes not the default? Why are buffer overflows permitted? Why was it so easy for China to divert all Internet traffic through its servers for 20 minutes in April 2010? Why is Windows so prone to viruses? Is it a conspiracy?

My motto is “always choose stupidity over conspiracy.” It goes back to the issue of lack of awareness. Developers that are not constantly trained on security will introduce security issues like buffer overflows or passwords stored in clear text or encrypted instead of hashed with a salt, etc. Some protocols were not designed with security in mind, which makes them susceptible to manipulation. Some targets are definitely softer than others.

At an absolute minimum, measures should be taken to harden the respective systems, as per the individual vendors’ guidelines and instructions. Unnecessary system services and processes should be disabled to reduce the attack surface, appropriate access control mechanisms should be properly configured, critical system patching should be done on a regular basis, etc.

But, unfortunately, these minimal security measures are often insufficient to address the rapidly expanding threat landscape. System visibility, in as near real time as possible, is required. Automated user process monitoring, vulnerability assessment, event correlation, and accompanying security policy notifications/alerting for these systems needs to be provided.

Is the cloud safe? Is SaaS safe?

I do not believe that the cloud or the SaaS model is inherently more or less safe—it is just a different kind of safe. Depending on the organizations’ risk appetite, they can be provided with the appropriate safeguards and controls to make implementation of private and public cloud-based services correspondingly “safe.” Technological controls, as well as organizational and administrative controls, need to be tailored for these types of deployments.

It’s also critical that the database security model be extensible and scalable to accommodate virtual and cloud-based environments.

Do we need better laws or should we trust the “enlightened self-interest” of industry? Enlightened self-interest—the mantra of Fed chairman Alan Greenspan—didn’t prevent the financial collapse Will it prevent the digital equivalent of Pearl Harbor?

“Enlightened self-interest,” by itself, is usually insufficient. At least it has been proven to be up to now. On the other hand, over-regulation would not be a good alternative, either. There has to be a happy medium—where government and private industry work together to promote a more secure environment for commercial transactions to occur, and where consumers’ privacy is also protected. But, unfortunately, we’re not there yet.

If not laws, how about some standards? Why aren’t there templates for hardened operating systems, databases, and networks? Or are there?

There are numerous standards for applying security controls to these systems, including Center for Internet Security (CIS), which includes “hardening” benchmarks for a variety of different systems and devices, as well as the NIST 800 Series Special Publications that offer a very large set of documents addressing applicable policies, procedures, and guidelines for information security. In addition, most of the more significant IT product vendors provide specific hardening guidelines and instructions pertaining to their various products.

The problem is how to consistently measure and make sure that your systems do not deviate from the gold standard you set. Unfortunately, systems tend to deteriorate with use—parameters are changed, new credentials and permissions are introduced, etc. An organization without a consistent, proven way to scan systems is going to have issues no matter how close it follows the standards. A recent scan we did with a large enterprise discovered over 15,000 weak passwords in their databases. In theory, they followed very strict federal policies.

Who will guard the guards themselves? As an administrator, I have unlimited access to sensitive information. How can my employer protect itself from me?

There’s a fundamental tenet in information security called “principle of least privilege,” which basically says that a user should be given the necessary authorization to access the information they need to perform their tasks/job—but no more than that level of privileged access. In addition, there’s another concept called “separation (or “segregation”) of duties,” which states that there should be more than one person required to complete a particular task, in order to help prevent potential error or fraud.

In the context of databases, this translates to not allowing users and administrators to have more access than is required for them to do their jobs—and for DBAs, that the DB administrative tasks will be monitored in real time and supervised by a different team, usually the information security team. A security framework that enforces these database access control policies is critical, because the inconvenient fact is, many compromises of DBs involve privileged access by trusted insiders.

While there is a much higher probability that someone who is not a DBA would try to breach the database, the DBA is in a much better position to succeed should he or she really want to do that.

If risk is the arithmetical product of the probability of an incident happening and the potential damage that incident could cause, then due to the latter factor, DBAs as well as other highly skilled insiders with access privileges pose a significant risk.

In 2007, Computerworld and other sources reported that a senior DBA at a subsidiary of Fidelity National Information Services Inc. sold 8.5 million records, including bank account and credit card details, to a data broker. An external hacker would find it very difficult to achieve this kind of scale without insider cooperation.

It is important, for security as much as for regulatory compliance reasons, to monitor and audit DBA activity. In fact, this should be done for all users who access the database. DBAs are the first to understand this. If you work in a bank vault, you know there are CCTV cameras on you. You want those cameras on you. DBAs are in a similar situation, and they understand this requirement completely.

What DBAs should not accept are solutions that hinder or interfere with the DBA’s daily tasks—DBAs are primarily concerned with running databases efficiently. Any solution that jeopardizes this primary objective is counter-productive and doomed to fail anyway, because DBAs and other staff will find ways to circumvent it.

What DBAs should not accept are solutions that hinder or interfere with the DBA’s daily tasks—DBAs are primarily concerned with running databases efficiently. Any solution that jeopardizes this primary objective is counter-productive and doomed to fail anyway, because DBAs and other staff will find ways to circumvent it.

At the risk of getting lynched by Journal readers, I have to ask your opinion about certification. Information Technology is the only profession whose practitioners are not subject to licensing and certification requirements. Can we really call ourselves “professionals” if we are not subject to any rules? Doesn’t the cost-benefit analysis favor licensing and certification? Even plumbers and manicurists in the state of California are subject to licensing and certification requirements but not IT professionals. Do you advocate security certification?

Well—while there’s certainly value in conducting user security awareness training and in promoting and achieving professional security certification, there are some issues. Like who would the accrediting body be? Who exactly needs to be certified? Will there be different levels of certification? Will each OS, DB, network device, application, etc., require its own distinct cert? It can quickly get very complicated.

But a shorter answer could be yes—I advocate security certifications.

In the novel 1984, George Orwell imagined that a device called a “telescreen” would allow “Big Brother” to listen to everything you said. The reality in 2013 is much worse since so much is digital, including my every message, phone call, and commercial transaction, and the cell phone is everybody’s personal electronic monitoring bracelet. What steps should we take to protect ourselves in this brave new digital world?

One possible answer might depend on how much security an individual is willing to trade for a potential reduction of features and functionality. For example, when “location services” are enabled on your phone, a variety of enhanced proximity-based services are then available, like several kinds of mapping services, driving directions and conditions, identification of nearby retail outlets, restaurants, gas stations, etc.

In addition, you can also locate your phone if it gets lost, wipe it of its contents, and/or have emergency services find you to provide help. But you also potentially get location-based advertisements, and there’s the specter of the device and application vendors (browser and service providers, too) aggregating and mining your various voice/data transmission location(s), for their own commercial purposes. The ongoing “privacy vs. commerce” battles involved in the “Do Not Track” discussions are good examples of these often-conflicting forces.

My personal assumption is that anything I publish on any network (text message, Facebook, Twitter, etc.) is public, no matter what settings it is published with. If I want to keep something private, I encrypt it. But, I’m willing to make privacy sacrifices in the name of convenience. I do use GPS; I do use Facebook and LinkedIn, etc.

Thank you for spending so much time with us today. Would you like to tell Journal readers a little about today’s McAfee? What are your current products? What is in the pipeline?

Well, I’m glad you asked. The McAfee Database Security solution comprises a core set of three products that serve to scan, monitor, and secure databases:

• McAfee Vulnerability Manager for Databases, which automatically discovers databases on the network, detects sensitive information in them, determines if the latest patches have been applied, and performs more than 4,700 vulnerability checks.
• McAfee Database Activity Monitoring, which provides automatic, non-intrusive, and real-time protection for heterogeneous database environments on your network with a set of preconfigured security defenses, and also provides the ability to easily create custom security policies based on configurable, and very granular, controls. In addition, it has the capability to deliver virtual patching updates on a regular basis to protect from known vulnerabilities.
• McAfee Virtual Patching for Databases (vPatch), which protects unpatched databases from known vulnerabilities and all database servers from zero-day attacks based on common threat vectors, without having to take the database offline to patch it. Additionally, vPatch has been accepted as a “compensating control” in compliance audits.

The McAfee Database Security solution is also tightly integrated with McAfee’s centralized security management platform, ePolicy Orchestrator (ePO), which consolidates enterprise-wide security visibility and control across a wide variety of heterogeneous systems, networks, data, and compliance solutions.

At McAfee, we do not believe in a silver bullet product approach. No security measure can protect against all attacks or threats. However, McAfee’s Security Connected framework enables integration of multiple products, services, and partnerships for centralized, efficient, and effective security and risk management. ▲

How Not to Interview a Database Administrator (The Google Way)

As suggested by the following story, Google would have preferred to hire my teenage daughter as the manager of their database team instead of me. I was on a long drive with my family so—to pass the time—I asked them to solve the problem that the Google interviewer had asked me to solve:

“Four men are on one side of a rickety bridge on a dark night. The bridge is only strong enough to support two men at a time. It is also necessary for the men crossing the bridge to carry a lantern to guide their way, and the four men have only one lantern between them. Andy can cross the bridge in 1 minute, Ben in 2, Charlie in 5, and Dan in ten minutes. How quickly can all four men be together at the other side?”

My daughter’s first solution was identical to mine.

Andy and Ben cross the bridge first. This takes two minutes.
Andy returns with the lantern. This takes one minute.
Andy and Charlie cross the bridge next. This takes five minutes.
Andy returns with the lantern. This takes one minute.
Andy and Dan cross the bridge last. This takes ten minutes.

The total time for the above solution is 19 minutes. However, I had googled the answer after returning from my interview (at Google) and knew that the four men could cross in 17 minutes, so I asked my daughter to try again. She “solved” the problem on her second attempt which suggests that Google would have preferred to hire her as a manager of database administration instead of me. Click here to see the “solution.”

I quoted the words “solved” and “solution” in the above paragraph because we still need a rigorous proof that the above “solution” is in fact the optimal solution; that is, is there a solution that takes less than 17 minutes? Neither does the above “solution” provide any insight into the general case. For example, the above “solution” is not optimal if Ben takes 4 minutes to cross the bridge instead of 2 minutes. The above “solution” needs 23 minutes if Ben takes 4 minutes to cross the bridge but it can be done in 21 minutes. And what if there are more than four people who need to cross? I am willing to bet that my Google interviewer would not have been able to prove the optimality of the above “solution” or solve the general case. If you’re interested, a comprehensive mathematical treatment of the above case as well as the general case can be found in this mathematical paper by Prof. Gunter Rote of the Free University of Berlin.

The Google interview technique is not the best technique for finding the best database administrators (or those with the right aptitude). Please feel free to comment. Is it just a case of sour grapes on my part?

Show Me the Way—with the innovator behind Statspack and AWR

Follow @Oratweets

As published in the 102nd issue of the NoCOUG Journal (February 2012)

Show Me the Way

with Graham Wood

Graham Wood has been working with Oracle Database for 25 years. He is currently a product manager for the Oracle RDBMS based in Redwood Shores, Calif. He has architected and tuned some of the largest Oracle databases, and has presented around the world on Oracle performance–related topics.

I have it on very good authority (Tom Kyte in the current issue of Oracle Magazine) that you are the genius and innovator behind Statspack and Automatic Workload Repository. I am in awe. Tell me the story behind that.

Wow, starting with a memory test! When Oracle V6 was introduced it contained the first V$ views, such as V$SYSSTAT and V$FILESTAT. These structures were created to allow database development to understand which parts of the code were being executed, and how often, during the running of the OLTP benchmarks that had started to appear at that time. The database shipped with two scripts that were used to produce a report from the V$ views during a benchmark run. These were bstat.sql, which captured the current contents of the V$ views at the start of the benchmark into a set of tables, and estat.sql, which captured the contents at the end of the benchmark into another set of tables, produced a report from the two sets of tables, and then dropped them. I was working in a small specialist performance group in Oracle UK at the time and it occurred to us, being database guys, that it might be useful for production systems to do regular captures of the V$ views and to keep this data around for rather longer as a repository of performance data. We wrote some scripts and started to distribute them inside Oracle, and they also found their way out to several customers. This was the original “Stats Package,” as we called it. As new releases of the database came out, I upgraded the scripts, probably most notably with the inclusion of V$SQL in Oracle V7 in the Stats7 package. In 1996 I moved to Oracle HQ in Redwood Shores to work in the Server Technologies Performance Group, and one of the goals that I set myself was to get the scripts shipped with the product so that all customers could use them. They finally made it into the database distribution in Oracle 8i as Statspack after being updated and enhanced by Connie Green. And the rest, as they say, is history, with almost all big Oracle sites using Statspack to keep a history of performance data.

When we started development of Oracle 10g, one of the main focus areas for the release was to be manageability, and a key part of that was to simplify performance analysis and to make recommendations for performance improvement. The most important part of this for me was to be able to automate performance analysis for a database instance and to identify the key areas where improvements could be made. Basically, unless the analysis is correct, there is no point in trying to make recommendations. In order to do this we needed to have a common currency for reporting across the components of the database and for quantifying the performance of a system. This led to the introduction of the concept of DB Time, the time spent in the database by user sessions, which allowed us to do quantitative comparisons between different components and also to quantify the impact on the system of an issue—for example that a single SQL statement represents 27% of all of the user time spent in the database. One of the main objectives of this was to make DBAs more effective by directing them to areas where they were likely to be able to make the greatest improvements in performance, rather than them spending time and effort on making changes that produced little benefit. To do all of this needed much more infrastructure than there was in Statspack and in Oracle 10g, and a lot of effort went into ensuring that we had adequate data available to do analysis of a performance problem the first time that it occurred. This resulted in an automatically managed repository of data (AWR), which contained not only data from normal V$ views containing cumulative statistics but also metric data and sampled activity data in the Active Session History. The combination of all of these data sources has taken performance analysis to a different level.

Tom Kyte’s favorite performance story is about a database that was always slow on rainy Mondays. What’s your favorite performance story from your own experiences?

One company that I worked with early on in my Oracle career asked me to help them improve the performance of a large batch report which was produced every night and went out to six people around the organization. It was causing problems for all of the rest of their batch operations by consuming a large amount of resources. The first improvement was to run the report once and print six copies rather than run the same report six times! Then I spoke to the folks who received the report and found out that three of them immediately tossed it in the trash (this was before the days of recycling), and the other three never looked beyond the first four summary pages as they now had an online system that allowed them to look at the details. We ended up changing the report to just produce the summary, and the overnight batch load on the system dropped by about 95% from the start point. It was definitely a case of it always being faster to not do something than to do it.

The most common problem that I see is that of flawed analysis: fixating on a particular statistic or event, which means that you never get to the root cause of the problem.

What are the typical issues you see when you are asked to look at a performance problem? Indexes? Statistics?

Well by the time I get called in to look at a performance problem these days there have probably already been quite a few people looking at it before, so all of the obvious things have already been tried. So, to be honest, the most common problem that I see is that of flawed analysis: fixating on a particular statistic or event, which means that you never get to the root cause of the problem and you end up trying to deal with a long list of symptoms. Much better to take a top-down approach and make sure you have the real cause before trying to fix things. If you have a really bad headache you could try and find a better aspirin or lie down in a darkened room, but you might be better to just stop banging your head against the wall. Having said that, I do still see a lot of problematic SQL, and drilling down to the root cause has become so much easier with the introduction of SQL Monitor. It is one of my top features of Oracle 11g, both for DBAs and developers, as it makes it so easy to find out exactly where in the plan the high resource usage and bad cardinality estimates are coming from, without even having to look at the details of the SQL itself. And, of course, I still see applications that have poor connection management and perform unnecessary parsing, even though we have been telling folks how to do it right for a couple of decades now.

I’ve heard a rumor that attendees of the Real World Performance events are being told that “tune” is a four-letter word. Is that some sort of insider joke? What does it mean?

I think that you have me confused with Cary Millsap! Cary differentiates between “tuning” and “optimizing.” The four-letter word that we talk about in the Real World Performance Day is “hack.” We define hacking as making changes without having diagnosed the root cause of the problem, without having scoped the problem or solution, and without being able to detail the expectation, in terms of what changes can be expected in the database performance statistics, of applying the “fix.” Most commonly these days the supporting argument for applying a hack is “well, I found a website that said if I set _go_faster in the init.ora I will run at least three times faster.” While Google can obviously be a good source of information, you have to remember that not everything that you read on the Internet is true. There really is no good alternative to doing proper performance analysis (although the availability of DB Time and ADDM make it easier) and proper testing, in your environment and with your data.

The title of software professional comes with a requirement to deliver quality product, not just hope that hardware will bail you out

In Oracle on VMware, Dr. Bert Scalzo makes a case for “solving” performance problems with hardware upgrades. What’s your opinion about this approach? [Footnote]

Ah, the “hardware is the new software” approach, as my colleague Andrew Holdsworth calls it. Software was called software because it was the part of the system that was “soft’ and could easily be changed. These days we often see customers who will do anything they can to avoid changing the application, no matter how bad it is. Hardware upgrades can only ever “ameliorate” a subset of performance problems. If the system is CPU bound, then adding more CPU cycles may make things better, but the benefits that you get will be, at best, the 2x every 18 months of Moore’s Law. But most systems with performance problems these days are not CPU bound, and even when they are, there is also a real possibility that adding more CPU will actually further reduce the performance of the system by increasing contention on shared structures. The performance benefits of fixing the software can be orders of magnitude greater and, if done well, make it so that the system is better able to scale with hardware upgrades. The cheap hardware theory primarily applies to CPU, although larger, cheaper memory can also help but often requires that the box is changed anyway. Storage system upgrades are rarely cheap. Although $/GB has been falling rapidly, $/GB/s and $/IOP/s have not, and reducing I/O performance problems will always involve increasing either one or the other of these throughput metrics. I would guess that most of the readers of your magazine would think of themselves as software professionals. To me that title comes with a requirement to deliver quality product, not just hope that hardware will bail you out.

Saying No to NoSQL

Just when I thought I’d finished learning SQL, the NoSQL guys come along and tell me that SQL databases cannot deliver the levels of performance, reliability, and scalability that I will need in the future. Say it isn’t so, Graham.

Well we hear much pontificating about the benefits of NoSQL, but so far I haven’t seen any audited industry-standard benchmark results as proof points. I have seen many claims from NoSQL evangelists that traditional RDBMSs cannot meet their requirements, only to find on further analysis that they tried a single open-source RDBMS, ran into some problems, and generalized from there. It is also interesting in the light of your previous question about using cheap hardware to try and resolve performance problems, that NoSQL solutions are developer intensive, as much of the functionality that would be provided by a SQL RDBMS has to be hand-crafted for each solution. But I’m sure over time we will see winners appear from the current plethora of NoSQL products.

What about Big Data. Can’t SQL databases handle big data then?

To me the case for Big Data comes down to two key areas: unstructured data and high-volume, low-value data such as web logs. This data could be stored in an RDBMS, but more typically what we are seeing customers doing is using Big Data techniques to extract information from these types of data sources and then storing this data in their RDBMS. This is the type of environment that Oracle’s recently announced Big Data Appliance is designed to help with.

The NoSQL salesmen insist that I need “sharding” instead of partitioning. Did they get that right?

Partitioning in the database has the huge benefit of being transparent to your application and your application developer. Using sharding requires that you move the management of the shards into your own application code. Do you want to develop your own code to perform queries across all of your shards and to do two-phase commits when you need to do a transaction that would affect multiple shards? And is such custom code development really cheap?

Professor Michael Stonebraker claimed in the 100th issue of the NoCOUG Journal that traditional SQL databases should be “sent to the home for tired software.” Has innovation really stopped at 400 Oracle Parkway? Has Larry sailed off into the sunset?

There have been many technologies that have claimed that they will replace SQL RDBMS over the last 30 years, including object databases and XML. SQL databases are still alive and well and contain the mission-critical data that is the lifeblood of businesses. Having a standard language, SQL, and a sound basis on relational theory means that SQL databases have stood the test of time in an industry where hype and fashion are rampant. In terms of 400 Oracle Parkway (where most of database development is housed) there are still many new features being built into the Oracle database that will increase the benefit that customers get from using the product. But you will have to wait for the product announcements to hear about those. And, of course, as the next America’s Cup is in San Francisco. Larry is still very much around and involved.

The Whole Truth About Exadata

Is Exadata a case of solving performance problems with hardware upgrades? Put another way: is the performance improvement from Exadata exactly what one might expect from the bigger sticker price, no more and no less?

Well the stock answer is that it is an engineered system that is designed to be capable of very high throughput. The software allows us to utilize the hardware much more effectively. There are customers who have upgraded to Exadata and seen the hardware upgrade benefits, typically 5–10x performance improvement, which is enough to get them into ads in The Economist and airports around the world. But the customers who have fully exploited the capabilities of Exadata have seen orders of magnitude more benefit. In our Day of Real World Performance presentations we load, validate, transform, collect optimizer statistics, and run queries on 1TB of raw data in less than 20 minutes. That sort of performance can transform what IT can deliver to the business and has far greater value than the sticker price.

Is Exadata as good for OLTP workloads as it is for OLAP? (You can be frank with me because what’s said in these pages stays on these pages!)

Well Exadata is certainly a very capable OLTP box. It has fast CPUs and can perform huge numbers of very fast I/Os with large numbers of IOPS by utilizing the flash cache in the storage cells. And OLTP performance is all about CPU horsepower and large numbers of IOPS. But I think it is fair to say that there is less “secret sauce” in Exadata as an OLTP platform than there is for data warehousing.

Show Me the Way

Thank you for answering my cheeky questions today. Someday, I hope to know as much about Oracle Database performance as you. Can you show me the way? Your book, perhaps?

Well I think that the key to being a good performance analyst is making sure that you spend time upfront correctly scoping the problem and then avoid jumping to conclusions while doing a top-down analysis of the data. When you are looking for solutions, make sure that the solution that you are implementing matches the scope of the problem that you started with, as opposed to a mismatched scope. The classic example of scope mismatch is making a database-level change, like changing an init.ora parameter, to solve a problem that is scoped to a single SQL statement. Much better to use techniques like SQL Profiles or SQL Baselines that will only affect the single SQL. Using that approach will get you a long way. As far as my book, I guess I still need to write it; it will be a cheeky book!

Footnote: Here’s the full quote from Dr. Scalzo’s book: “Person hours cost so much more now than computer hardware even with inexpensive offshore outsourcing. It is now considered a sound business decision these days to throw cheap hardware at problems. It is at least, if not more, cost effective than having the staff [sic] tuned and optimized for the same net effect. Besides, a failed tuning and optimization effort leaves you exactly where you started. At least the hardware upgrade approach results in a faster/better server experiencing the same problem that may still have future value to the business once the fundamental problem is eventually corrected. And, if nothing else, the hardware can be depreciated, whereas the time spent tuning is always just a cost taken off the bottom line. So, with such cheap hardware, it might be a wiser business bet to throw hardware at some solutions sooner than was done in the past. One might go so far as to make an economic principle claim that the opportunity cost of tuning is foregoing cheap upgrades that might fix the issue and also possess intrinsic value. Stated this way, it is a safe bet that is where the business people would vote to spend.”

Download the 102nd issue of the NoCOUG Journal