Demo code for Ghost Data in Indexes

NOTE: all demo data is fake.

This is the demo code for encrypting data where there is an existing index. We are starting with a table customers_tst that is in the unencrypted tablespace dat.

  1. start with dropping the old test objects.
  2. create two small tablespaces small_idx and dat.
  3. create the customers_tst table as a subset of customers.
  4. create an index on customers_tst(ssn)
  5. alter the table customers_tst to add encryption to ssn and cc_nbr. Because ssn has an index we are not using salt.
  6. We then alter the index to rebuild. Because Oracle marks block as free and does not erase them, the old index still exists.
  7. We check for ghost data in small_indx.dbf and dat.dbf.
  8. We confirm that there is ghost data and then drop the index.
  9. Once we drop the tablespace, we can then turn our attention on shredding the ghost data.
1 DROP TABLE customers_tst; 2 DROP MATERIALIZED VIEW customer_sales; 3 DROP TABLESPACE small_idx INCLUDING CONTENTS; 4 DROP TABLESPACE dat INCLUDING CONTENTS; 5 -- rm /opt/oracle/oradata/DEV/datafile/small_idx.dbf 6 -- rm /opt/oracle/oradata/DEV/datafile/dat.dbf 7 CREATE TABLESPACE small_idx DATAFILE '/opt/oracle/oradata/DEV/datafile/small_idx.dbf' SIZE 10M; 8 CREATE TABLESPACE dat DATAFILE '/opt/oracle/oradata/DEV/datafile/dat.dbf' SIZE 10M; 9 10 -- create a test table from customers. 11 CREATE TABLE customers_tst 12 tablespace dat 13 as (select * 14 from customers 15 where rownum <= 1000); 16 17 -- we are going to build an index on SSN 18 CREATE INDEX customers_ssn_idx ON customers_tst(ssn) TABLESPACE small_idx; 19 20 ALTER TABLE customers_tst MODIFY 21 (ssn encrypt USING 'AES256' NO SALT, 22 cc_nbr encrypt USING 'AES256'); 23 24 -- now lets do an index rebuild and test for ghost data 25 ALTER INDEX customers_ssn_idx REBUILD; 26 27 -- in root container run flush the buffer cache 28 29 -- in shell run strings /opt/oracle/oradata/DEV/datafile/small_idx.dbf 30 -- in shell run strings /opt/oracle/oradata/DEV/datafile/dat.dbf 31 32 SELECT * FROM customers_tst 33 where ssn = '347631761'; 34 35 drop index customers_ssn_idx; 36 37 DROP TABLESPACE small_idx; 38 -- in the shell shread the datafile 39 -- in the shell shred /opt/oracle/oradata/DEV/datafile/small_idx.dbf 40 -- in the shell rm /opt/oracle/oradata/DEV/datafile/small_idx.dbf 41 -- we are going to build an index on region so support FK to regions. 42 create tablespace small_idx datafile '/opt/oracle/oradata/DEV/datafile/small_idx.dbf' size 10M; 43 44 -- a small tablespce to hold a materialized view. 45 CREATE TABLESPACE dat 46 DATAFILE '/opt/oracle/oradata/DEV/datafile/dat.dbf' SIZE 10M; 47 48 -- now lets recreate our test data and encrypt it. 49 50 CREATE TABLE customers_tst 51 tablespace dat 52 as (select * 53 from customers 54 where rownum <= 1000); 55 56 ALTER TABLE customers_tst MODIFY 57 (ssn encrypt USING 'AES256' NO SALT, 58 cc_nbr encrypt USING 'AES256'); 59 60 -- we are going to build an index on SSN 61 CREATE INDEX customers_ssn_idx ON customers_tst(ssn) TABLESPACE small_idx; 62 63 64 CREATE MATERIALIZED VIEW CUSTOMER_SALES 65 TABLESPACE DAT 66 NOCACHE 67 PARALLEL 68 USING INDEX 69 REFRESH 70 START WITH SYSDATE NEXT SYSDATE + 1/24 71 COMPLETE 72 WITH ROWID 73 USING DEFAULT ROLLBACK SEGMENT 74 DISABLE QUERY REWRITE AS 75 SELECT 76 c.fname, 77 c.lname, 78, 79 c.state, 80, 81 c.cc_nbr, -- cc_nbr is sensitive and encrypted. 82 c.ssn, -- ssn is sensitive and encrypted. 83 s.price, 84 s.sales_date, 85 86 FROM customers_tst c, 87 sales_tst s, 88 products p 89 where = s.cust_id 90 and s.product_id =; 91 92 CREATE INDEX CUSTOMER_SALES_IDX ON CUSTOMER_SALES (SSN) TABLESPACE small_idx; 93 CREATE INDEX CUSTOMER_SALES_IDX2 ON CUSTOMER_SALES (cc_nbr) tablespace small_idx; 94 95 --DROP MATERIALIZED VIEW customer_sales; 96 97 -- lets check for sensitive data in the datafiles. 98 -- in shell strings /opt/oracle/oradata/DEV/datafile/dat.dbf 99 -- in shell strings /opt/oracle/oradata/DEV/datafile/small_idx.dbf 100 DROP MATERIALIZED VIEW customer_sales; 101 DROP TABLESPACE small_idx INCLUDING CONTENTS; 102 DROP TABLESPACE dat INCLUDING CONTENTS; 103 -- in shell shred /opt/oracle/oradata/DEV/datafile/dat.dbf 104 -- in shell shred /opt/oracle/oradata/DEV/datafile/small_idx.dbf 105 -- in shell rm /opt/oracle/oradata/DEV/datafile/dat.dbf 106 -- in shell rm /opt/oracle/oradata/DEV/datafile/small_idx.dbf 107 108 CREATE TABLESPACE small_idx DATAFILE '/opt/oracle/oradata/DEV/datafile/small_idx.dbf' SIZE 10M; 109 CREATE TABLESPACE dat DATAFILE '/opt/oracle/oradata/DEV/datafile/dat.dbf' SIZE 10M; 110 111 112 CREATE TABLE customers_tst 113 (ID, 114 FNAME, 115 LNAME, 116 CITY, 117 STATE, 118 ZIP, 119 DISCOUNT, 120 CC_NBR ENCRYPT USING 'AES256', 121 REGION, 122 SSN ENCRYPT USING 'AES256' NO SALT) 123 TABLESPACE dat 124 AS 125 (select * 126 from customers 127 where rownum <= 1000); 128 129 130 -- we are going to build an index on SSN 131 CREATE INDEX customers_ssn_idx ON customers_tst(ssn) TABLESPACE small_idx; 132 133 134 -- we are going to recreate the materialized view, this time 135 -- we will add the encrypt clause. 136 137 CREATE MATERIALIZED VIEW CUSTOMER_SALES 138 ( fname, 139 lname, 140 city, 141 state, 142 zip, 143 cc_nbr encrypt USING '3DES168', 144 ssn encrypt USING '3DES168' NO SALT, -- because ssn has an index we will not use salt. 145 price, 146 sales_date, 147 product_name ) 148 TABLESPACE DAT 149 AS 150 (SELECT 151 c.fname, 152 c.lname, 153, 154 c.state, 155, 156 c.cc_nbr, 157 c.ssn, 158 s.price, 159 s.sales_date, 160 161 FROM customers_tst c, 162 sales_tst s, 163 products p 164 WHERE c.ID = s.cust_id 165 and s.product_id =; 166 167 CREATE INDEX CUSTOMER_SALES_IDX ON CUSTOMER_SALES (SSN) tablespace small_idx; 168 CREATE INDEX CUSTOMER_SALES_IDX2 ON CUSTOMER_SALES (ZIP) TABLESPACE small_idx; 169 170 -- in shell strings /opt/oracle/oradata/DEV/datafile/dat.dbf 171 -- in shell strings /opt/oracle/oradata/DEV/datafile/small_idx.dbf

#infosec issues on moving to the #cloud #DBaaS

Last week I was at Oracle Cloud World working at the ODTUG booth. This gave me the opportunity to talk to a lot of people who are seriously looking at moving their environment to the cloud. While chatting with these people, I started to pull together some thoughts on the security issues that come with moving to the cloud. Many of those issues are the same for hosting your own database applications. There are several issues with moving to the cloud and if you don’t address them it can become dark and stormy.

What security questions do you need to address prior to moving to the cloud? Note: many of these issues also applies to hosting your own databases! This subject is complex and I’m just touching on some of the issues.  If you don’t do your due diligence you will get burned.


Will and How will your data be encrypted? First off, all of your data should be encrypted by default. I am also of the <OPINION> cloud provider should not even offer to store your information unencrypted. </OPINION>. With the advent of hardware encryption modules, encryption performance is a non-issue.

There are a couple of options on encrypting your data, both have strengths and both have weaknesses. First of the easiest encryption option to implement is tablespace encryption. This option is used to encrypt all of your data stored in the tablespaces.  The down side is the data is unencrypted in the SGA.

The other option is column encryption.  This requires a bit of work upfront to setup. You are going to need to identify the atomic pieces of data that need to be encrypted then go through your indexing scheme to make sure you have not put indexes on columns that are encrypted with salt, and you don’t have foreign key constraints on columns that are encrypted. The upside of column encryption is the data stays encrypted in the SGA.

Will your backups be encrypted? Again, the answer must be yes and this is where it gets a bit tricky. RMAN backups are block level copies of the data files, so if your data is encrypted, your backups will be encrypted. However, if someone runs a datapump export of your data to refresh a lower environment and they do not specify encryption in the options, then your data will be saved unencrypted. The cloud provider must audit for this event and if it does happen, then you need to be informed and the cloud provider must make every effort to find and destroy that datapump file and any copies that have been made of it. You notice I used the word destroy as apposed to delete. Well there is good reason for that, if you delete a file there is still ghost data that can be recovered. So that or those file(s) will need to destroyed by a utility such as Linux shred.

The trusted insider attack surface has changed. <OPINION> It is safe to assume Oracle and other Tier 1 cloud providers will vet their system administrators. </OPINION> However; people change, that is just a fact of life. I frequently use the example of Edward Snowden. Prior to his leaking NSA documents he had gone through polygraph examinations, and his entire background put under a microscope, then he changed.

How will your cloud provider protect you against their trusted insider? The concept is easy, wall off your data from being seen by the system administrator. I’ve been a DBA for decades and can tell you with complete honesty, the DBA or SA does not need access to your data in order to do their job. <OPINION> Oracle has a great product Database Vault that is designed to wall off your data from the SAs. Any cloud solution should include the implementation of Database Vault. </OPINION>

Your cloud provider must provide a proven tool that protects your information from trusted insiders at the cloud provider.

Your cloud provider must also provide an integrated audit solution that tracks all audit events and allows you to report on audit events. Oracle Audit Vault comes with BI. You can use caned reports and customize those reports for your requirements.

Can you make customization’s to the security? Oracle Real Application Security (RAS) gives you, Redaction, Virtual Private Databases and audit on all connections. A full discussion of RAS is beyond the scope of this paper.

<OPIONION> At the very least, you should be able to implement, Virtual Private Databases, and Redaction to protect your data from the normal use of you applications. </OPINION> (I say normal use of you applications. Using different tools and grants it is possible to bypass these features.)

Will the cloud provider implement and configure Database Firewall. Database Firewall is a good tool to defend against sql injection attacks. It takes a lot of work to properly configure it especially if you are using a custom application. Will the cloud provider be responsible for the configuration of database firewall?

How are you going to get your data back if you decide to break up with your cloud provider?

If your cloud provider is using Oracle 12C multitennant an encryption key is generated with the container database and that is used to decrypt the encryption key for the pluggable database. I’m not going to dive too deep into this. The cloud provider can unplug your database and provide you with a set of keys to decrypt your data.

Then the worst happens, there is a data breach. You need to know, how will your cloud provider make you whole. The truth is, your customers will be upset with you and maybe your cloud provider.

End the end, you are the steward of your customers data and with that stewardship comes responsibilities.

2015 #InfoSec in review. We get a big fat “F”

We are stewards of our customers data and need to do better. <OPINION> I would give us a big fat “F” for data security in 2015.</OPINION> What happened and what needs to be improved? We saw weak passwords, lack of encryption, malware and social engineering over and over again. One very sad aspect of these attacks is once the system was compromised, the attack went on for months, even years prior to the attack being uncovered. So again we really need to do better, reading the logs, doing analytics on system behavior and locking down the data.

High level the attack vectors have not changed much over the years. Malware payloads are still being delivered by drive by downloads and infected emails. Businesses and medical groups are still leaving sensitive data unencrypted, trusted insiders can still get to sensitive information. We are also seeing encrypted connections being made to unknown servers and allowing that traffic to go through our firewalls.

I’m going to do my best to keep my opinion clear by using the <OPINION> </OPINION> tags so you know what my personal opinion is. I’m also not going to go through every attack that happened in 2015. In here I will also let you know what I think should / could have been done to mitigate the attacks.

1) IRS Data breach

In IRS’s effort to make things easy for users to access their data they exposed very sensitive tax and financial data to hackers. Over 100,000 people were compromised with this system and $50,000,000 is false tax refunds have been stolen from the US Government.

When we design systems, one of the top requirements we have are user experience. If we make it to hard to access the systems they will not be used, make it to easy and the data can be compromised. We need to weigh the value of the data with user experience. The users expect their information to be respected and protected.

2) OPM data breach

The OPM hack impacted me personally along with my wife. The impact was over 22 million people had full background and biometric information leaked to a foreign intelligence agency. I watched the congressional hearings and was very disappointed by the <OPINION>incompetence of the people </OPINION> testifying. The Director of OPM resigned but <OPINION> the CIO of OPM should have been walked out the door. </OPINION> it was her job to make sure this information was secure. I still don’t know why this information was not stored on the classified network as it should have been. <OPINION> As an added insult, the government is offering two years of credit monitoring. As if a foreign intelligence agency is really interested in taking out credit cards in our names. The big threat is we are now at risk for blackmail. </OPINION>

The OPM breach was malware that was making encrypted connections to unknown servers. This is a case where black listing IP’s would not work, but white listing connections would work. Sensitive data should only be transmitted over trusted paths and <OPINION> if encrypted connections are being made, then those connections should be treated as sensitive. </OPINION>

3) UCLA Medical

UCLA Medical lost 4.5 million records of unencrypted patient data including PII and medical information. There is no excuse to not encrypt sensitive data. I still hear the old excuse of there is a performance impact of encryption. With the availability of hardware encryption modules, this argument does not hold water.

After encrypting data, we still have to be careful about ghost data and data leakage. A DBA can still run database pump and get an unencrypted copy of the data then copy that data to another location. We do this all the time to refresh an environment. Controls need to be placed on data pump copies so any information that is exported from the database will stay encrypted and the location of those copies are known. When moving data from unencrypted to encrypted, all ghost data must be shredded.

4) Ashley Madison

This one did not really interest me very much other then the disrespect Ashley Madison showed their customer base. This hack ruined some reputations and exposed a large number of people to blackmail. Yes credit card numbers were encrypted, but geolocation and email addresses was not encrypted. <OPINION> The large number of people who used their work and government email addresses was shocking. These people who are so blind to opsec deserve to be caught. </OPINION>

5) Hyatt

Just recently we learned about the Hyatt payment processing data breach. Not much is known at this time other then malware sent encrypted data to an unknown server. This is yet another case of needing to have a trusted path for sensitive data by using white list and denying access to any unknown IP address.

6) Trump Hotels

Trump Hotels, in a year long campaign, credit card and security code information was stolen from customers of Trump properties. I’m going to keep beating this drum, you need a trusted path from point of sales to the processing database, so <OPINION> implement white lists and deny any encrypted traffic to unknown ip’s.</OPINION>

7) T-Mobile and Experian

T- Mobile placed their trust in Experian and suffered a massive breach of 15 million customers full name, social security number and date of birth and some passport numbers. In this case no payment card data was compromised. Yet this is still enough information for identity theft. Not a lot of information has been provided on the attack vector used.

In December 2013 T-Mobile suffered another data breach with vendor Decisioning Solutions that is owned by Experian. In both of these cases, T-Mobile is offering credit monitoring through ProtectMyID that is owned by Experian. <OPINION> Why does T-Mobile continue doing business with Experian? </OPINION>

This is not an exhaustive list of breaches for 2015.


VTECH the toy manufacturer exposed data on 4.8 million customers due to password insecurity.

9) Securus

Securus lost 70 million call logs and recorded conversations of people in prison. These recordings also included attorney client privileged conversations.

10) FBI

The FBI LEO Portal was hacked, the attack vector and damage is still classified.

11) Scott Trade

Scott Trade lost data on 4.6 million customers under a two year campaign. Krebs on Security reported that the data was used for stock scams.

12) Excellus Blue Cross Blue Shield.

Excellus Blue Cross Blue Shield lost data on 10 million customers. The attack started in 2013 and was not discovered until 2015.

13) Anthem

Anthem lost data on 78.8 million customers. I have read the count was actually 80 million customers and 19 million rejected customers.

14) Anonymous vs ISIS.

I only add this because of the interest in ISIS. After the Paris attacks Anonymous started OpParis that is turning into a interesting game of wack a mole. Anonymous is using brute force to shut down ISIS controlled accounts and servers. The results are debatable, <OPINION> it would be better to allow some of the systems to stay online to gather intelligence on ISIS. By shutting them down you are forcing them onto the dark web where it’s harder to gather intelligence.</OPINION>

<OPINION> Sadly, many times after a breach the offending company offers one year or two years of credit monitoring. The customer will be exposed for the rest of their life. Two yours of credit monitoring is wholly inadequate./OPINION>

What do we need to do.

  1. Secure the data. Encrypt data at rest so if the data is compromised then it will be useless to the criminal.
  2. Encrypt the data on the network when there is sensitive data going through it. Man in the middle attacks happen.
  3. Build trusted paths for sensitive information. All sensitive information must go through that path.  If an encrypted session is being built to an unknown server, deny that connection.
  4. Secure the parameter. We are letting encrypted traffic go to unknown servers. This has to stop by using white list. If a workstation or node can process sensitive data, then that workstation or node should not be able to access unknown servers.
  5. Secure programming practices. I still see first hand sloppy programming that is vulnerable to sql injection. Organizations must impalement secure coding practices with code reviews that also include looking to vulnerability. A couple months ago, I came across a piece of code that was vulnerable to sql injection, when I brought it up to program management I was told, going back to fix the problem would put the program behind schedule, move forward and we will fix it after going production. <OPINION> This is the wrong attitude. </OPINION>If the program had standards in place before coding started, then the problem would not have gotten as far as it did.
  6. Secure the data from trusted insiders. I wont get into the political issues of Bradly Manning or Edward Snowden. Both of them were vetted and had access to sensitive information, they broke their trust and stole information that did incalculable damage.
  7. Routinely review audit logs to look for unusual behavior. I’m still seeing audit logs get ignored until there is a problem. Products like Oracle Audit Vault, brings all of your audit into one package where you can create BI dashboards to find out when something is happening that is outside of the norm.

#infosec #LetsSecureThisTogether This is going to piss some people off. The C suite needs to have this conversation.

Get the words best practice out of your vocabulary.” I have been at many customer sites that needed my expertise, and someone says to me in a meeting, “well what is the best practice to secure our information.” I’m going to tell you right now, the bad guys are reading the same best practice white papers and poking holes through them left and right. In addition, that audit report you received saying you are in compliance with <state your regulation> may be factually correct at that moment in time, but your information is still not secure.

For each of your systems, bring five of your senior system administrators into a room and ask them a simple question. “How would you compromise your system?” Then sit back and listen. If they are good inside of thirty minutes you are going to start hearing things that will scare you. Let me give you some examples from my life.

We are generating audit reports but no one is actually reviewing them. At one customer, I generated audit reports that showed invalid logins’, ip address and username logins when the the connections were simultaneous, and a host of other audit reports. I then went to our security people and asked them how often they wanted this report, daily, or weekly. I was told to review them and only bring things up to them when I find something interesting. Well for one, under the concept of separation of duties your SA should not have sole responsibility for reviewing audit logs. Yes, they should but a copy needs to be sent out for review. As the Oracle DBA I can logon as oracle or logon as sys, I am god. I can do almost anything I want and cover my tracks. Can you spell Edward Snowden? Can you say with complete confidence that you do not have an Edward Snowden in your shop?

The patch schedule is to drawn out. One shop we were twelve months behind in our Oracle CPU patches because of the perception that patching the database would impact the development schedule. This is one of those times where I got maybe a little bit forceful in a meeting and pushed for getting everything patched. But the customer was adamant, “you will not do anything that will slowdown development and testing.” We finally got everything patched when it was announced there was going to be an audit. The audit happened, and there was no findings. Our immediate management was happy, me, I was not so happy. When a patch comes out, the bad guys are reading what is being fixed and they are quite adapt at exploiting the weaknesses that are being patched.

A webserver that is miss-configured, I walked into one customer and as I was setting up my audit scripts, I noticed there were over a thousand invalid login attempts from a handful of webservers every thirty minutes. The DBAs were not talking to the webserver SA’s and the webserver SA’s would allow the invalid login attempts to continue if the application that webserver supported was no longer in use and again, no one was actually reviewing the audit logs.  In fact this was a known issue that was explained as “normal” to the security group. This is the perfect way to hide password cracking attempts. One of the webservers that was in the DMZ had not been used for over a six months, but was still running. When I dug into the audit trail to see what was actually going on, I found several attempts to connect to the database as sys, system, admin, sa, root and a host of other attempts. That server had been compromised along with a few others. When I brought the evidence up to management they were shocked. Finally passwords were changed on the webservers and those webservers that were no longer in use were pulled from the network and a complete scan of the network was completed.

Excessive privileges to developers and developers using a common development account. Yup, this still happens. I walk into a shop, and given the application username password to do all my work. Folks your audit trail is now toast. At that same shop all developers have the passwords for sys and system. I never got a good reason for this. NO ONE should ever logon at sys. Application accounts should never be used for development.

This is just a small fraction of the issues I have seen in different shops. 

All of these shops either followed “best practices” or modified their practice when the learned there was going to be an audit. Everyone of those shops are staffed with professional SA’s who can tell you where the weakness is. And every shop I have ever worked in has weaknesses. Your job in the C suite is to ask these professional SA’s to come into a meeting and have a safe and secure conversation on how they would crack the system. If they can not come up with anything you are either working at NSA, they are trying to hide something or you need smarter SA’s.

And don’t just do this once and say, we’re done, schedule these meetings either quarterly or semiannually. This is a conversation between the C suite and your professional SA’s. You need to understand where your risk are, these people are real smart and if you listen to them they can tell you what’s wrong and what needs to be done to fix it.

Once you have this information, give your professional SA’s and their management the tools and resources they need to close these security holes.

#Oracle #TDE Ghost Data Teaser

Here is a teaser for the Oracle Transparent Data Encryption presentation

We look at having an existing table with existing indexes. A policy comes out that says we need to encrypt SSN and Credit Card Numbers. Once we encrypt the columns and rebuild the indexes, does the unencrytped data in the index get encrypted?

Watch and find out.

Oracle Transparent Data Encryption Baseline Practices webinar

I will be giving a webinar on Oracle Transparant Data Encryption Baseline Practices August 27, 2015 at 3PM. Sponsored by @odtug 

Why “Baseline Practices?” well best practices does not seem to be working so we are going to start improving “your game” by setting the baseline and getting you to think how information is secured.

This one-hour presentation includes how to, gotchas (where data can leak in clear text), and baseline practices for implementing Oracle Transparent Data Encryption. We will walk through setting up Oracle Transparent Data Encryption and establish baseline practices to secure your information, explaining where and how to encrypt the data, and where and how data leaks in plain text and HEX format. We will also explore these questions: When sending data to a standby database, when does the data stay encrypted and when can it transfer over the network in clear text? When using Oracle Streams, does data go across the network in clear text? When gathering statistics on encrypted data, where can the data be exposed unencrypted? As we discuss each of the leaks, we will also review how to mitigate the leakage and eliminate ghost data.

Register here: