Passing Variables to EBS Start Scripts (without having them show up in the environment)

In earlier releases of Oracle E-Business Suite, many of the standard startup/shutdown scripts would accept a password as a command-line argument.  This gave us the ability to automate and/or script certain maintenance tasks.

E-Business Suite R12.1.3

In E-Business Suite R12.1.3, for example, you could start the appsTier with the following:

cd ${ADMIN_SCRIPTS_HOME}

./adstrtal.sh apps/${APPSPW}

E-Business Suite R12.2

When we get to E-Business Suite R12.2, the startup procedure is pretty much the same, but the command line arguments are no longer there.  To get around this, we’re told to pass things on the command line u sing the echo command.  Per MOS 1902776.1, the recommendation is:

cd ${ADMIN_SCRIPTS_HOME}

{ echo "APPS" ; echo ${APPSPW} ; echo ${WLADMINPW} ; }|\
./adstrtal.sh @ -nopromptmsg

Although the command line is complicated, it works well. 

The Problem

However, the variables that are set will persist in the Unix environment of the adstrtal.sh script (and all processes that is spawns).

Test Script

So, to figure this out (and test our approaches), we have this simple test script:

#!/bin/bash
echo -e "\n--------------------------------------------------\n"
echo -e "Variables:\n"
echo -e "\tAPPSPW:  ${APPSPW}"
echo -e "\tAPPSUN:  ${APPSUN}\n"

echo -e "Full ENV DUMP in /tmp/env.txt"
echo -e "Searching through /tmp/env.txt for APPSUN and APPSPW"
echo -e "\n--------------------------------------------------\n"
env >/tmp/env.txt

grep "APPSPW\|APPSUN" /tmp/env.txt
echo -e "\n--------------------------------------------------\n"

We can set these variables and run the script normally:

{ echo ${APPSUN} ; echo ${APPSPW} ; }|./test.sh

--------------------------------------------------

Variables:

   APPSPW:  appspw
   APPSUN:  apps

Full ENV DUMP in /tmp/env.txt
Searching through /tmp/env.txt for APPSUN and APPSPW

--------------------------------------------------

APPSPW=appspw
APPSUN=apps

--------------------------------------------------

As you can see, the ${APPSUN} and ${APPSPW} variables are in the text file.

However, when we run the script with a different command line, we get a different result.

{ echo ${APPSUN} ; echo ${APPSPW} ; }|env -u APPSPW bash ./test.sh

--------------------------------------------------

Variables:

   APPSPW: 
   APPSUN:  apps

Full ENV DUMP in /tmp/env.txt
Searching through /tmp/env.txt for APPSUN and APPSPW

--------------------------------------------------

APPSUN=apps

--------------------------------------------------

The Solution

To get around that problem, we can actually DELETE the variables from the environment.

cd ${ADMIN_SCRIPTS_HOME}

{ echo "APPS" ; echo ${APPSPW} ; echo ${WLADMINPW} ; }|\
 env -u APPSPW -u SYSTEMPW -U WLADMINPW bash ./adstrtal.sh @ -nopromptmsg

The command line is even uglier. We still have ${APPSPW}, ${SYSTEMPW}, and ${WLADMINPW} in our shell, but, those variables are not inherited by the running EBS processes.

E-Business Suite 12.1 is now Certified on OEL7 and RHEL7

New this week from Steven Chan’s blog:

https://blogs.oracle.com/stevenChan/entry/oracle_e_business_suite_release5

One thing that Steven’s blog posting mentions that deserves specific emphasis is that Oracle Database 11.2.0.4 and 12.1.0.2 are ALSO certified on OEL7/RHEL7 [see MOS 1304727.1].  Please note that the certification is specific to the version of the database.  Most notably that 11.2.0.3 and 12.1.0.1 appear to be excluded from this certification.  As always, be sure to pay close attention to the certification status of your various components when planning any installation/upgrade.

It’s also important to note that, while there is a 32-bit version of RHEL6 (and E-Business Suite 12.1.3 is certified on it), there isn’t a 32-bit version of RHEL7.  This is important and, at the same time, it isn’t.  First of all, it’s highly unlikely that anyone is still using 32-bit hardware.  (Or that they ever were, for E-Business Suite on Linux).  Yes, it’s true that the appsTier components of E-Business Suite 12.1 are still 32-bit, running them on a 64-bit Linux requires only minor adjustments.  The bulk of which involve dependencies on kernel settings and Linux packages.

So, with all of this out there… Go forth and upgrade!

— James

Listing Installed Packages on Linux

First, let me mention that, unless otherwise indicated, when I blog about Linux it will be about the RPM-based distributions that are certified with the Oracle Database (RedHat Enterprise Linux, Oracle Enterprise Linux).

Normally, when you’re looking to see which packages are installed on Linux (RedHat, Oracle, CentOS), you would use this command:

rpm -qa

Unfortunately, the standard output of that command omits alot of useful information. It may or may not indicate if you have the 32 or 64 bit version of a package installed, for example.

So, for a command that will show you which packages are installed in a format that looks like the name of the RPM file:

rpm -qa --queryformat \
"%{NAME}-%{VERSION}-%{RELEASE}.%{ARCH}.rpm\n" |\
sort > pkglist_`date +%Y%m%d-%H%M`.txt

— James

E-Business Suite R12.1.1 is Certified on Oracle Linux 6!

Back in February, I blogged about the pending certification of Oracle E-Business Suite on Oracle Enterprise Linux 6 and RedHat Enterprise Linux 6. In that blog post, I noted that the certification announcement was “planned” but, of course, Oracle doesn’t provide dates.

Well, guess what? The waiting is finally over. As these things go, the announcements come out in parts.

First, on March 22, 2012, Oracle announced that Oracle Database 11gR2 and Fusion Middleware 11gR1 were certified. (The press release can be found here.)

And today (April 4, 2012), through the Oracle E-Business Suite Technology blog (known to many of us as “Steven Chan’s blog”), we have the E-Business Suite announcement (available here)!

While this is fantastic news, read the announcements carefully!

These certifications are ONLY for Oracle Enterprise Linux 6 on the x86-64 with the Unbreakable Enterprise Kernel (UEK) version 1.

These certifications are ONLY for Oracle Enterprise Linux 6 on the x86-64 with the Unbreakable Enterprise Kernel (UEK) version 1.

According to the database announcement, certification on RedHat Enterprise Linux 6 (and Oracle Enterprise Linux 6 [without UEK]) should be available within 90 days. I would expect the E-Business Suite R12 announcement to follow shortly behind.

What about other E-Business Suite releases? At this point, I have no actual information. But, I can speculate (with a good degree of certainty) that you won’t see any 11.5.10.2 certifications against OEL/RHEL 6. E-Business Suite 11.5.10.2 is currently in Extended Support. Even though the support fees have been waived (through the end of Extended Support, November 30, 2012), Oracle doesn’t typically certify new platforms once a product goes into Extended Support. (A more detailed discussion of Oracle’s recent support announcements can be found here.)

The other question mark out there is OEL/RHEL 6.0 on x86-32. Personally, if you’re implementing R12 or upgrading to R12 on Linux, you should be using an x86-64 distribution on x86-64 hardware. However, certification on x86-32 is also forthcoming.

As always, be sure to read/follow the relevant notes through the Certify Tab on My Oracle Support before you start any project to make sure that the combination of components you intend to use are, in fact, certified. These certifications will also detail the various always steps, operating system parameters, packages, and even patches specific to your combination that you will need to follow.

All of this is excellent news, as the OEL and RHEL 5.x line is getting pretty long in the tooth and is approaching it’s end of life.

Now… when will we get that R12.2 announcement? Collaborate, maybe? OpenWorld? … the waiting continues.

– James

UPDATE 6/27/2012:  Oracle has just announced certification for Oracle Enterprise Linux 6.0 (x86-32), Red Hat Enterprise Linux 6.0 (x86-32 and x86-64), and  Novell SUSE Linux Enterprise Server (SLES) version 11 (64-bit).  See Steven Chan’s blog for more details:  https://blogs.oracle.com/stevenChan/entry/oracle_e_business_suite_release3

Stupid Unix Tricks… Part 1

So, let’s say you’re trying to figure out if the database (or E-Business Suite) is down. Now, the logical way is use the Unix commands ps and grep to check for a particular process. Generally speaking, we would look for the SMON process for that particular instance.

However, maybe you’re looking for something else that has multiple processes and you want to see that they’re all shut down.

We’re going to use a database as an example (largely because I assume you are familiar with the database). The basic command would be:

ps -ef|grep ora_smon_PROD
oracle 10445 6643 0 15:32 pts/0 00:00:00 grep ora_smon_PROD
oracle 19710 1 0 Feb28 ? 00:00:36 ora_smon_PROD

However, the problem here is that it also gives our grep command. To get around that, we can strip it out using grep -v grep (which would strip from our results anything that contains the string grep). Additionally, maybe we want to get something we can use in an if statement. The simplest way to do that is to count the number of lines returned by the command. That can be done by piping the output through the wc -l command. Our final command will look like this:

ps -ef|grep ora_smon_PROD|grep -v grep |wc -l

So, assuming that we just wanted to look for SMON we can build our if statement like this:

if [ `ps -ef |grep ora_smon_PROD|grep -v grep |wc -l` -gt 0 ]; then
   echo "SMON is UP"
else
   echo "SMON is DOWN"
fi

Now, let’s assume that you want to check for PMON as instead:

if [ `ps -ef |grep ora_pmon_PROD|grep -v grep |wc -l` -gt 0 ]; then
   echo "PMON is UP"
else
   echo "PMON is DOWN"
fi

But what if you wanted to make sure that they were BOTH down?

if [ `ps -ef |grep -e ora_pmon_PROD -e ora_smon_PROD|grep -v grep |wc -l` -gt 0 ]; then
   echo "PMON and SMON are UP"
else
   echo "PMON and SMON are DOWN"
fi

The key here is grep -e. Because grep allows you to use the -e flag more than once per invocation, you can specify multiple strings to search for. Multiple -e strings are treated as a logical “or” by grep when it’s parsing the input.

As with everything, your results may vary. Different platforms may have different versions of grep with different capabilities. This example was tested on Linux.

– James

Password-less Login Using SSH Pre-Shared Keys

Way back when I started working with Unix (otherwise known as “the olden days” or “days of yore”), one of the tricks we used was a concept known as “remote login” and the “Berkeley R commands”. This was based on a number of things, most of them depending on either the /etc/hosts.equiv or the ${HOME}/.rhosts file to establish the trusting relationship. Configuring these would allow you the ability to do some really neat things. Among them, copying files from one host to another using a command like rcp /tmp/file user@remotehost:/tmp/file without being asked for a password. This made for some really neat scripting opportunities and made it much easier to manage multiple systems.

Unfortunately, the Berkeley “R” commands are notoriously insecure. The way that the trusting was done was based entirely on the username and hostname of the remote user on the remote host. Literally, you told the server to trust “jmorrow@remotehost.mydomain.com”. The problem with this is that all that was required was knowledge of the trusting relationship. All you had to do was set up a machine named “remotehost.mydomain.com” and create a “jmorrow” user on it. Then you could go anywhere that that trusting relationship allowed.

Fortunately for us, the cool features that were introduced by the Berkeley “R” commands are implemented much more securely in the SSH protocol and toolset.

The SSH Protocol can use pre-shared keys to establish trusting relationships. In this case, each node has both a public and a private key. When the client talks to the server, the client offers a ” key”. The server, which maintains a list of trusted “public keys”, then compares that key to it’s database to determine if it actually trusts the client. If the client passes the test, then it is allowed in without any further challenge. This can be very useful for administrators, automated file transfer, also for scripting interactions between hosts. Note that this is not a “Machine A” trusts “Machine B” relationship. It is “user@machinea” trusts “user@machineb”.

For the purposes of this article, the “server” is the node that you are logging into from the “client”. So, the “server” is the one that is doing the trusting. The terms “server” and “client” refer only to the role being played by each component in the ssh communications session. I should also mention that Oracle Real Application Clusters (RAC) depends on this relationship as well.

Generate your public/private key pairs [Both Client and Server]

The server (user@host) needs to have one, and each client (user@host) that is being trusted needs to have one.

Execute these two commands (in a Unix/Linux environment) to create both your rsa and your dsa keys. You will be prompted for a location to store the files (typically under ${HOME}/.ssh), and for a passphrase. In all cases, it’s ok to accept the defaults.

ssh-keygen -t rsa
ssh-keygen -t dsa

If you know you don’t want to use a passphrase, you could generate the keys with these two commands:

ssh-keygen -t rsa -f ${HOME}/.ssh/id_rsa -N ""
ssh-keygen -t dsa -f ${HOME}/.ssh/id_dsa -N ""

Transfer the public key files from the client to the server

I prefer to make sure that I have a uniquely named copy of the public keys (makes it easier to transfer to another box when first establishing the relationship).

cd ${HOME}/.ssh
ls -1 id_[dr]sa.pub |while read LINE
do
cp ${LINE} ${LINE}.`whoami`@`hostname -s`
done

Now copy these files to the server:

scp ${LINE}.`whoami`@`hostname -s` trustinguser@trustingserver:.ssh/.

Copy the public keys you’re trusting into the authorized_keys file

Here, we’ll need to put those keys into the authorized_keys file. Do this for each of the files that you transferred in the previous step.

cd ${HOME}/.ssh
cat filename >> authorized_keys

Make sure permissions are correct

If the permissions on these files are too open, the trusting relationship will not work. Here are my recommendations:

chmod 600 ${HOME}/.ssh/auth*
chmod 700 ${HOME}/.ssh
chmod 644 ${HOME}/.ssh/id_[dr]sa.pub*
chmod 600 ${HOME}/.ssh/id_[dr]sa

Now, you should be able to ssh from the client to the server witout being prompted for a password.

— James

Certification on Oracle (RedHat) Enterprise Linux 6?

One of the questions that getting with increasing frequency is “Can we upgrade our systems to Oracle (or RedHat) Enterprise Linux 6”. 

As always, my first response is to jump on My Oracle Support, navigate to the “Certifications” tab, and attempt to query up the certified platforms for the Oracle Server (RDBMS). In my experience, the Oracle Server is usually among the first products certified on any (major) platform. 

So, I go to the Certify page on MOS, and it appears that Oracle (or RedHat) Enterprise Linux 5 is the most recent certified with the current version of the database (11.2.0.3).

Now, I’m thinking to myself, “Hasn’t it (Linux 6) been out for some time?”. Turns out, it has. RedHat Enterprise Linux 6 was initially released in November, 2010.

So, a with a little effort (“the Google knows all”), I was able to find a few tidbits:

First, from a RedHat Blog Post back in August, 2011:

The certification process we conducted with Oracle 11gR2 and Red Hat Enterprise Linux 6 is the same process we have successfully completed a number of times with earlier Red Hat Enterprise Linux releases. With those releases, Oracle’s certification approval process took about six weeks from the day we submitted test results to Oracle to the day that Oracle posted the certification on their MetaLink support site (https://support.oracle.com). Based on this experience, we would expect certification of the 11gR2 database on Red Hat Enterprise Linux 6 to occur sometime in CYQ4 of this year. We look forward to Oracle’s response and to working with them to complete this certification process. In the interim, customers may also contact Oracle directly for updates on the certification status at gcp-customerfeedback_us@oracle.com.

In addition to the certification testing described above, we perform ongoing and extensive testing on Oracle 11gR2 at every minor release of Red Hat Enterprise Linux. Consequently, we confidently recommend the deployment of Oracle 11gR2 in Red Hat Enterprise Linux 6 production environments today.

And also, from a note on My Oracle Support:

Database Client or Database Server Install on Red Hat Enterprise Linux 6 (RHEL6) or Oracle Linux 6 [ID 1350000.1]

At the time of the last update of this article (30-Jan-2012), Red Hat Enterprise Linux 6 (RHEL6) and Oracle Linux 6 are not certified or supported for use with any Oracle Database version. Be sure to use only certified/supported combinations of Database version and OS version, which you can find under the Certifications tab of My Oracle Support (MOS). Please note that although Certify used to have a status of “Planned” for Red Hat Enterprise Linux 6 (RHEL6) / Oracle Linux 6, this does not imply any guarantee of certification. Also, please do not open a Service Request asking when the OS will be certified. Any additional information will be added to Certify when it becomes available, and Support is unable to provide any more information than what is there. The Certify information on MOS is the only official source for Oracle certification.

So there you have it. Oracle Server (and, consequently, E-Business Suite) are NOT currently certified to run on either Oracle Enterprise Linux 6.x or on RedHat Enterprise Linux 6.x. Unfortunately, anyone you ask at Oracle will basically tell you that “due to revenue recognition rules, we can’t say”.

What I can tell you is, despite the “this does not imply any guarantee of certification” message, I think it highly likely that they will certify it (eventually). Especially given Oracle’s large Linux installed base (and the fact that Oracle’s own Linux distribution has so much in common with RedHat’s).

Meanwhile… we wait.

— James

UPDATE:  I sent a few emails and heard back from some people at Oracle.  From the sound of it, a certification announcement for Oracle Enterprise Linux 6 is being “planned” but, naturally, they can’t provide any date information.  But, when you combine this with the information above, it leads me to believe that some certification might be coming sooner rather than later.  Let’s keep our fingers crossed!

UPDATE #2:  Oracle has now (22 March, 2012) certified 11.2.0.3 against Oracle Enterprise Linux 6 (Using the Unbreakable Enterprise Kernel V1) for x86-64 platforms. (Press Release).  They have also announced that certification against RHEL 6 will be complete within 90 days.

UPDATE #3 (6/27/2012):  Oracle has just announced certification for Oracle Enterprise Linux 6.0 (x86-32), Red Hat Enterprise Linux 6.0 (x86-32 and x86-64), and  Novell SUSE Linux Enterprise Server (SLES) version 11 (64-bit).  See Steven  Chan’s blog for more details:  https://blogs.oracle.com/stevenChan/entry/oracle_e_business_suite_release3