Recently, we had to add a node to Oracle RAC 19c in one of our environments. It sounds simple enough — run a few scripts, verify configs, and you’re done. However, per Murphy’s Law — if something can go wrong, it usually will — and with RAC, it usually shall. The best way to avoid unnecessary surprises is to run the right checks up front and stick to proven practices. That’s exactly what this blog is about.
To give more context, node1 of the cluster had a hardware failure due to which it had to be removed from the cluster. After maintenance, we were adding it back to the cluster. This scenario is a classic case for needing to add a node to Oracle RAC 19c after a failure and recovery cycle.
Pre-Check With cluvfy
Before proceeding run cluvfy precheck. This step checks kernel parameters, packages, user groups, disk space, hostname resolution, network, storage and a lot more. It is extremly important to fix all FATAL errors before proceeding otherwise the node addition will fail.
# From existing node ie node2
cluvfy stage -pre nodeadd -n <new-node> -verbose
Warnings were encountered during execution of CVU verification request "stage -pre nodeadd".
OS Kernel Version ...WARNING
PRVF-7524 : Kernel version is not consistent across all the nodes.
Kernel version = "5.15.0-309.180.4.el9uek.x86_64" found on nodes: node1.
Kernel version = "5.15.0-307.178.5.el9uek.x86_64" found on nodes: node2.
CVU operation performed: stage -pre nodeadd
Date: Jun 30, 2025 2:27:56 PM
CVU version: 19.25.0.0.0 (100824x8664)
Clusterware version: 19.0.0.0.0
CVU home: /u01/app/19c/grid
Grid home: /u01/app/19c/grid
User: oracle
Operating system: Linux5.15.0-307.178.5.el9uek.x86_64In our case, everything passed except a kernel version mismatch. Since it was minor (a patch-level difference), we chose to proceed after internal review. The plan was to update the kernel on node2 after node 1 was added to the cluster – this would ensure zero downtime for the app.
Use addnode.sh to Add a Node to Oracle RAC 19c
Use addnode.sh with IGNORE_PREADDNODE_CHECKS=Y to bypass non-critical warnings. This is the main step when you add a node to Oracle RAC 19c and start integrating it into the cluster.
## node 2 (To be run from existing node)
cd /u01/app/19c/grid/addnode
export IGNORE_PREADDNODE_CHECKS=Y
cd $GI_home/addnode
./addnode.sh -silent "CLUSTER_NEW_NODES={node1}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node1-vip}" "CLUSTER_NEW_NODE_ROLES={hub}"
The output would looks as follows
$ ./addnode.sh -silent "CLUSTER_NEW_NODES={node1}" "CLUSTER_NEW_VIRTUAL_HOSTNAMES={node1-vip}" "CLUSTER_NEW_NODE_ROLES={hub}"
[WARNING] [INS-40111] The specified Oracle Base location is not empty on following nodes: [node1].
ACTION: Specify an empty location for Oracle Base.
[WARNING] [INS-13014] Target environment does not meet some optional requirements.
CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2025-06-30_03-20-39PM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2025-06-30_03-20-39PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
Copy Files to Remote Nodes in progress.
.................................................. 6% Done.
.................................................. 11% Done.
....................
Copy Files to Remote Nodes successful.
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 21% Done.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2025-06-30_03-20-39PM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 49% Done.
Saving cluster inventory in progress.
.................................................. 83% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/19c/grid was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2025-06-30_03-20-39PM.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 90% Done.
Update Inventory in progress.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2025-06-30_03-20-39PM.log
Update Inventory successful.
.................................................. 97% Done.
As a root user, execute the following script(s):
1. /u01/app/oraInventory/orainstRoot.sh
2. /u01/app/19c/grid/root.sh
Execute /u01/app/oraInventory/orainstRoot.sh on the following nodes:
[node1]
Execute /u01/app/19c/grid/root.sh on the following nodes:
[node1]
The scripts can be executed in parallel on all the nodes.
Successfully Setup Software with warning(s).
.................................................. 100% Done.Run root scripts
Run orainstRoot.sh and root.sh. As you can see in the logs, the root.sh script is a critical step when adding a node to an Oracle RAC cluster. Next, it configures the node, integrates it into the clusterware stack, and registers and starts services like ASM, CRSD, and listeners.
## To be run on the new node (ie node 1) that was added to the cluster
# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
## Root.sh
# /u01/app/19c/grid/root.sh
Check /u01/app/19c/grid/install/root_node1_2025-06-30_15-25-53-525336224.log for the output of root script
Performing root user operation.
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/19c/grid
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Relinking oracle with rac_on option
Using configuration parameter file: /u01/app/19c/grid/crs/install/crsconfig_params
The log of current session can be found at:
/u01/app/oracle/crsdata/node1/crsconfig/rootcrs_node1_2025-06-30_03-26-05PM.log
2025/06/30 15:26:22 CLSRSC-594: Executing installation step 1 of 19: 'ValidateEnv'.
2025/06/30 15:26:22 CLSRSC-363: User ignored prerequisites during installation
.......
..
2025/06/30 15:31:44 CLSRSC-594: Executing installation step 18 of 19: 'ConfigNode'.
clscfg: EXISTING configuration version 19 detected.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2025/06/30 15:31:59 CLSRSC-594: Executing installation step 19 of 19: 'PostConfig'.
2025/06/30 15:32:22 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded
Domain name to open is ASM/Self drc 4
Postchecks
Once the node was added and the root scripts were executed, I ran a final postcheck using cluvfy stage -post nodeadd -n <new-node> -verbose. This step confirms full integration of the new node into the cluster. It checks connectivity, SCAN and VIP consistency, NTP sync, and that all expected cluster resources are online. It also verifies things like multicast communication, subnet consistency, and listener registrations. In our case, everything passed cleanly — a good confirmation that the RAC node addition was successful end-to-end.
# To be run on the new node (ie node 1)
crsctl status res -t
cluvfy stage -post nodeadd -n node 1 --verbose
Post-check for node addition was successful.
CVU operation performed: stage -post nodeadd
Date: Jun 30, 2025 3:35:43 PM
CVU version: 19.25.0.0.0 (100824x8664)
Clusterware version: 19.0.0.0.0
CVU home: /u01/app/19c/grid
Grid home: /u01/app/19c/grid
User: oracle
Operating system: Linux5.15.0-307.178.5.el9uek.x86_64
Run Optional Health Checks After Adding Oracle RAC 19c Node
The post checks help us establish a baseline and validate if any issues remain after the node addition. These checks confirm that the new node integrates correctly with the existing RAC cluster. Run the following commands from node1 to verify connectivity, cluster health, and resource status.
- olsnodes
olsnodes -n
node1 1
node2 2- Check Listener
# ps -ef | grep tns
root 6 2 0 Jun28 ? 00:00:00 [netns]
oracle 149251 1 0 15:31 ? 00:00:00 /u01/app/19c/grid/bin/tnslsnr ASMNET1LSNR_ASM -no_crs_notify -inherit
oracle 149547 1 0 15:31 ? 00:00:00 /u01/app/19c/grid/bin/tnslsnr LISTENER_SCAN1 -no_crs_notify -inherit
oracle 151474 1 0 15:32 ? 00:00:00 /u01/app/19c/grid/bin/tnslsnr LISTENER -no_crs_notify -inherit
root 165388 97019 0 15:38 pts/3 00:00:00 grep --color=auto tns
srvctl status listener- ASM
srvctl status asm -a
ASM is running on node2,node1
ASM is enabled.
ASM instance +ASM1 is running on node node2
Number of connected clients: 2
Client names: +APX1:+APX:host-scan node2:_OCR:host
ASM instance +ASM2 is running on node node2
Number of connected clients: 4
Client names: +APX2:+APX:node -MGMTDB:_mgmtdb:host instance2:host:host-scan node2:_OCR:host-scan
- OCR & voting disk
ocrcheck
Status of Oracle Cluster Registry is as follows :
Version : 4
Total space (kbytes) : 901284
Used space (kbytes) : 85132
Available space (kbytes) : 816152
ID : 777450341
Device/File Name : +OCR
Device/File integrity check succeeded
Device/File not configured
Device/File not configured
Device/File not configured
Device/File not configured
Cluster registry integrity check succeeded
Logical corruption check succeeded
## crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE cbe24d8095734f4bbf884a2a4accd210 (AFD:OCR01) [OCR]
Located 1 voting disk(s).
Add RDBMS Home After Adding a Node to Oracle RAC 19c
Now that you’ve successfully configured the Grid Infrastructure, make sure to register the RDBMS home with the cluster.This is done by running addnode.sh from the RDBMS home on the existing node. This step ensures that srvctl is aware of the database home on the new node and can manage instances across the cluster properly.
nsure the DB home directory on node1 is empty. This is required before running the addnode.sh script. Importantly, the installer copies software from node2 and configures it here. Any pre-existing files can cause the operation to fail or behave unpredictably.
## On existing node ie node2
# pwd
/u01/app/oracle/product/19c/db/addnode
./addnode.sh -silent "CLUSTER_NEW_NODES={node1}"
[WARNING] [INS-13014] Target environment does not meet some optional requirements.
CAUSE: Some of the optional prerequisites are not met. See logs for details. /u01/app/oraInventory/logs/addNodeActions2025-06-30_03-42-37PM.log
ACTION: Identify the list of failed prerequisite checks from the log: /u01/app/oraInventory/logs/addNodeActions2025-06-30_03-42-37PM.log. Then either from the log file or from installation manual find the appropriate configuration to meet the prerequisites and fix it manually.
Prepare Configuration in progress.
Prepare Configuration successful.
.................................................. 7% Done.
Copy Files to Remote Nodes in progress.
.................................................. 12% Done.
.................................................. 18% Done.
..............................
Copy Files to Remote Nodes successful.
You can find the log of this install session at:
/u01/app/oraInventory/logs/addNodeActions2025-06-30_03-42-37PM.log
Instantiate files in progress.
Instantiate files successful.
.................................................. 52% Done.
Saving cluster inventory in progress.
.................................................. 89% Done.
Saving cluster inventory successful.
The Cluster Node Addition of /u01/app/oracle/product/19c/db was successful.
Please check '/u01/app/oraInventory/logs/silentInstall2025-06-30_03-42-37PM.log' for more details.
Setup Oracle Base in progress.
Setup Oracle Base successful.
.................................................. 96% Done.
As a root user, execute the following script(s):
1. /u01/app/oracle/product/19c/db/root.sh
Execute /u01/app/oracle/product/19c/db/root.sh on the following nodes:
[node1]
Successfully Setup Software with warning(s).
.................................................. 100% Done.
## Run root.sh on node 1
# /u01/app/oracle/product/19c/db/root.sh
Check /u01/app/oracle/product/19c/db/install/root_node1.ad.nyu.edu_2025-06-30_15-50-36-290767008.log for the output of root script
This completes the node addition process end-to-end, including both Grid and RDBMS configuration. If you’re ever in a similar situation, following these steps will help you successfully add a node to Oracle RAC 19c with minimal surprises. A good reference note to review would be this MOS 3019970.1
Discover more from oratrails-aish
Subscribe to get the latest posts sent to your email.