So long, BPEL, it's been swell

Took a new position with a company that doesn't run BPEL.

As such, I'll be neglecting this blog even harder now.

Refreshing your connection pool

If you're using the database adapter to "poll" an external database, and that database restarts, the connection pool on that BPEL instance, and, consequently, any BPEL processes using that connection pool, will not work properly.

More troubling is that, due to how BPEL processes instantiate from the DB Adapter, you might not see any errors.

To fix this, you can go into the EM, and navigate to the following location:

-- Administration -- JDBC Resources -- Connection Pools

Find the pool with the dropped database, click "Refresh Connection Pool" and confirm. Test your connectivity then and see what happens.

Possible Memory Leak in BPEL Process Manager

A while back, we had a nasty issue with all of our BPEL Instances in Production - what we saw, and eventually remedied, were the following symptoms:

Each instance would go down roughly every 14 hours, independent of load on the system.
We'd start by dropping OPMN ping requests, and, eventually, OPMN would declare the instance dead and would force a restart.

By enable remote-monitoring in OPMN.XML and binding JConsole to the instance, we determined that the PS MarkSweep never ran - consequently, everything that made into Eden space made it into Survivor, and then, of course, into OldGen. Eventually, the instance would get memory-bound and stop responding to OPMN pings. Interestingly enough, manually performing a Garbage Collection through JConsole did free up Eden and Survivor.

Oracle pointed to a potential flaw in Sun's JDK, which I believe is a good part of the answer, but, we did see this issue with the most current supported version of Sun's JDK. We ran the JRockit JDK, and did not see the issue, so, this does lend some credence to that theory. None of that really fixed the problem, though, because JRockit wasn't supported under OAS/BPEL 10.1.3.3.

The fix finally came down when we built a new instance from scratch - one of the steps we take in building a new instance is to disable unused services, like OWSM, ESB, and JavaSSO. Apparently, OWSM does something akin to making a scheduled explicit Garbage Collection, because as soon as we re-enabled the application, PS MarkSweep began running as regular as clockwork (literally).

Just something you might want to know...

Create BPEL Domains Programatically

To give you some background here, we have more BPEL instances than Carter has pills. Last time I counted, we're up to about 24 Production, and 28 Non-Production instances. I created a BIG batch file to perform all of our post-installation steps on these instances, including patching to a standard patch level, creating the standard set of domains, configuring logging parameters, configuring startup parameters, creating resource adapters and database connections, and some other fancy stuff.

Below I've posted my batch file for creating BPEL domains from the command-line. You'll need:

cURL (Command-line URL) - You can get that here
Your oc4jadmin password
The domain name you want to create
The hostname where BPEL is installed
The web port where the BPELConsole application is listening

Tweak the batch file to point to the location where curl.exe resides.

ECHO Off
SET DomainName=%1
SET HostName=%2
SET HostPort=%3
SET OC4JPassword=%4
SET CurlLoginLine=
SET CurlLoginLine="j_username=oc4jadmin&j_password=%OC4JPassword%" http://%HostName%:%HostPort%/BPELAdmin/j_security_check
SET CurlConfigLine=
SET CurlConfigLine="domainId=%DomainName%&datasourceJNDI=jdbc/BPELServerDataSourceWorkflow&txDatasourceJNDI=jdbc/BPELServerDataSource" http://%HostName%:%HostPort%/BPELAdmin/doCreateDomain.jsp
call curl.exe --trace-ascii c:\curltrace.txt -d %CurlLoginLine% -b cookie.txt -c cookie.txt
call curl.exe --trace-ascii c:\curltrace.txt -d %CurlLoginLine% -b cookie.txt -c cookie.txt
call curl.exe --trace-ascii c:\curltrace.txt -d %CurlConfigLine% -b cookie.txt -c cookie.txt

And, yes, the 2nd and 3rd from the last lines are identical. This isn't a typo. If you've dealt with 10.1.3.3 for more than 8 minutes, you'll have noted by now that you're forced to log in twice to the BPEL Console and EM Applications.

Make sure BPEL is up, and call the batch file.

Provide the requested information in this order:

1. The domain name you want to create
2. The hostname where BPEL is installed
3. The web port where the BPELConsole application is listening
4. Your oc4jadmin password

That should do it. Good luck. See the disclaimer.

Migrate BPEL to Another Windows Server

To start, the below are vital. When in doubt, don't!

• You're migrating from a Windows machine to a Windows machine.
• You have a compelling reason against using standard backup and recovery practices to perform this operation.
• You know how to export and import registry keys.
• You know how to use the Windows Resource Kit SC.exe command to create Windows Services.
• You're planning on keeping the new server name and IP Address the same as the old one. This is important!
• You know the right methodology, based on your organization's best practices, to migrate a server that's going to keep the same name as an existing server.
• You have no interest in actually trying this in a production environment, because it's not supported, nor is it very wise to go tossing your BPEL instances about like they're bags of candy and your servers are trick-or-treaters.

All that said, here we go.

We had to this to move our Production BPEL Instances to a new set of servers. The original servers were Windows 2003 Server, Standard, 32-Bit, the new are Windows 2003 Server, Enterprise, 64-Bit.

Reinstalling isn't an option, because then we'd have to redeploy several hundred BPEL processes, and go through rounds of regression testing for each process. Ugly. Oracle wasn't much help, and, really, nor should they be, because moving to another server is dicey.

I've performed migrations to different Dehydration Stores in the past, and what always struck me about Oracle on Windows is how self-contained the little buggers are. You have your Oracle Home directories, one Central Inventory, and some registry keys. I figured moving the whole shebang over to another server should be pretty simple. I was a bit off on that one.

This is how it's done.

1. Build your new server (We'll call this either New, or Target)
2. Disable all Oracle Services on Original Server (We'll call this Old, Original, or Source)
3. Perform a log cleanup to minimize the amount of data copied
   -- Remove backup OPMN logs
   -- Remove domain.log files
4. Copy all Oracle Home directories to exact same volume and path on new server
5. Copy C:\Program Files\Oracle to the target server
   -- Copy to C:\Program Files\Oracle on 32-Bit systems
   -- Copy to C:\Program Files (x86)\Oracle on 64-Bit systems
6. Create the following permanent environment variables on the target server
   -- ORACLE_HOME – (Leave Blank)
   -- PATH – Modify to include all Oracle references from the old server
   -- PHPRC – Should point to one of the php5.ini files in an Oracle Home - If your old server doesn't have this, don't freak out, you'll probably be fine.
7. Export the following registry keys from the source server
   HKLM\Software\ORACLE
   HKLM\Software\Apache Group
   HKLM\Software\ORACLE\inst_loc
8. On the exported keys, make the following changes (Also, do yourself a favor and read up on the Wow6432Node on 64-Bit Windows. It's an eye-opener)
   -- If moving to a 32-Bit System: No Change
   -- If moving to a 64-Bit System: Change Software\Oracle to Software\Wow6432Node\Oracle
   -- If moving to a 64-Bit System: Change the inst_loc value to reflect Program Files (x86)
9. Import the above keys into the Target server’s registry
10. Create Oracle Services using the SC command and the services on the old server as a template
11. Reboot
12. Check the OUI to make sure it sees installed applications
13. Check OPatch on the homes to make sure the inventory is intact.
14. Start BPEL

That's it.

You can leave comments if you have questions or, better yet, find this process wildly successful.

Introduction

My organization is one of the six or so companies that has implemented Oracle® SOA Suite 10.1.3.3, including BPEL Process Manager, on the Windows Platform.

The posts in this blog may help other Windows BPEL Administrators solve problems, and make their daily lives easier.

I've come up with ways to automate basic administration tasks, including domain creation and configuration, Data Source and Resource Adapter creation, and troubleshooting.

Understand this is not a programmer's blog. I do some scripting for SOA, but, if you're looking for BPEL program examples, I'd suggest the fantastic demo apps that come packaged with Oracle® Application Server 10.1.3.1.

More to follow...