DBFS configuration

Prerequisites

1.The database that will be used for the DBFS file systems must be present. (OK)

2.Root access across the compute nodes/RAC nodes. (OK)

3.DBFS disk groups must be mounted and available in ASM across all compute nodes. (OK)

4.My Oracle Support access. (OK)

5.For RAC, the fuse libraries must be installed : kernel-devel, fuse, fuse-libs, fuse-devel.  Ensure the latest and most relevant versions are installed (e.g. 64bit). 

Install required OS packages

[root@rac-nodea ~]# rpm -qa|grep fuse 
fuse-devel-2.9.4-1.0.4.el6.x86_64 
fuse-libs-2.9.4-1.0.4.el6.x86_64 
fuse-2.9.4-1.0.4.el6.x86_64 
gvfs-fuse-1.4.3-27.el6.x86_64 
[root@rac-nodea ~]# rpm -qa|grep kernel-devel 
kernel-devel-2.6.32-754.11.1.el6.x86_64 

 NOTE If this is the first installation of DBFS on RAC, it will require a full cluster restart during the configuration.

Run following configuration steps on all nodes (except adding clusterware resource)

Create FS directories

[root@rac-nodea /]# mkdir -p /dbfs 

Set permissions on FS directories 

[root@rac-nodea /]# chown -R oracle:dba /dbfs 
[root@rac-nodea /]# cp /etc/fuse.conf /etc/fuse.conf_bkp 
[root@rac-nodea /]# cat /etc/fuse.conf_bkp 
# mount_max = 1000
# user_allow_other 
[root@rac-nodea /]# echo user_allow_other > /etc/fuse.conf 
[root@rac-nodea /]# chmod 644 /etc/fuse.conf 

Enable on all nodes (RAC)

[root@rac-nodea /]# chmod a+x /usr/bin/fusermount

Configure libraries

[root@rac-nodea /]# mkdir -p /usr/local/lib
[root@rac-nodea /]# ln -s /u01/app/oracle/database/18/dbhome_1/lib/libnnz18.a /usr/local/lib/libnnz18.a
[root@rac-nodea /]# ln -s /u01/app/oracle/database/18/dbhome_1/lib/libclntsh.so /usr/local/lib/libclntsh.so
[root@rac-nodea /]# ln -s /usr/lib64/libfuse.so.2 /usr/local/lib/libfuse.so.2
[root@rac-nodea /]# echo /usr/local/lib >> /etc/ld.so.conf.d/usr_local_lib.conf
[root@rac-nodea /]# ldconfig

Restart clusterware services on all nodes (rolling)

As ROOT

$ sudo su - 
[root@rac-nodeb ~]# cd /u01/app/grid/18/bin
[root@rac-nodeb /u01/app/grid/18/bin]# ./crsctl stat res -t 
[root@rac-nodea /u01/app/grid/18/bin]# ./crsctl stop crs
[root@rac-nodea /u01/app/grid/18/bin]# ./crsctl start crs
[root@rac-nodeb /u01/app/grid/18/bin]# ./crsctl stat res -t

Database configuration for DBFS

[oracle@rac-nodea ~]$ . oraenv
ORACLE_SID = [+ASM1] ? orcl1
[oracle@rac-nodea ~]$ sqlplus / as sysdba
SQL> create bigfile tablespace dbfsts datafile '+DATA' size 1G autoextend on next 1G maxsize 300G extent management local autoallocate segment space management auto;
SQL> create user dbfs_user identified by passw0rd default tablespace dbfsts quota unlimited on dbfsts;
SQL> grant create session, create table, create view, create procedure, dbfs_role to dbfs_user;

Create dbfs filesystem

[oracle@rac-nodea ~]$ . oraenv
ORACLE_SID = [orcl1] ?
[oracle@rac-nodea ~]$ sqlplus dbfs_user/passw0rd
SQL> spool dbfs_create_filesystem.log                               
SQL> @?/rdbms/admin/dbfs_create_filesystem_advanced.sql dbfsts orclb_fs nocompress nodeduplicate noencrypt non-partition
No errors.
SQL> spool off

 NOTE Name  orclb_fs  will be added to the path /dbfs, after the dbfs filesystem will be mounted

Configure Oracle networking

[oracle@rac-nodea /u01/app/oracle/admin]$ mkdir -p dbfs/tnsadmin
[oracle@rac-nodea /u01/app/oracle/admin/dbfs/tnsadmin]$ vi tnsnames.ora
orclb.local =
 (DESCRIPTION =
 (ADDRESS =
    (PROTOCOL=BEQ)
    (PROGRAM=/u01/app/oracle/database/18/dbhome_1/bin/oracle)
    (ARGV0=oracleorcl1)
    (ARGS='(DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=BEQ)))')
 (ENVS='ORACLE_HOME=/u01/app/oracle/database/18/dbhome_1,ORACLE_SID=orcl1')
  )
  (CONNECT_DATA=(SID=orcl1))
)
[oracle@rac-nodec /u01/app/oracle/admin/dbfs/tnsadmin]$ vi sqlnet.ora 
WALLET_LOCATION =
   (SOURCE=(METHOD=FILE)
           (METHOD_DATA=(DIRECTORY=/u01/app/oracle/admin/dbfs/wallet/rac-nodea))
   )
SQLNET.WALLET_OVERRIDE = TRUE 

Add similar tnsnames.ora on other nodes (replace with corresponding instance name)

Create Oracle wallet

In order for the file systems to be mounted, the OS must be able to login to the database to reach the APIs present there.  

[oracle@rac-nodea ~]$ mkdir -p /u01/app/oracle/admin/dbfs/wallet
[oracle@rac-nodea ~]$ mkdir -p /u01/app/oracle/admin/dbfs/wallet/`uname -n | cut -d . -f 1`
[oracle@rac-nodea ~]$ cd /u01/app/oracle/admin/dbfs/wallet/`uname -n | cut -d . -f 1`
[oracle@rac-nodea /u01/app/oracle/admin/dbfs/wallet/rac-nodea]$ . oraenv
ORACLE_SID = [oracle] ? orcl1
[oracle@rac-nodea /u01/app/oracle/admin/dbfs/wallet/rac-nodea]$ orapki wallet create -wallet . -auto_login_local
[oracle@rac-nodea /u01/app/oracle/admin/dbfs/wallet/rac-nodea]$ mkstore -wrl . -createCredential orclb.local dbfs_user passw0rd
[oracle@rac-nodea /u01/app/oracle/admin/dbfs/wallet/rac-nodea]$ TNS_ADMIN="/u01/app/oracle/admin/dbfs/tnsadmin" sqlplus /@orclb.local

Configure mount options

Download zipped mount_dbfs scripts from Oracle Support Note 1054431.1

Create following files and update conf file accordingly. Also change EOL to Linux.

/tmp/mount_dbfs.conf

/tmp/mount_dbfs.sh

[root@rac-nodea /tmp]$ cat mount-dbfs.conf
DBNAME=orclb
MOUNT_POINT=/dbfs
DBFS_USER=dbfs_user
ORACLE_HOME=/u01/app/oracle/database/18/dbhome_1
GRID_HOME=/u01/app/grid/18
LOGGER_FACILITY=user
MOUNT_OPTIONS=allow_other,direct_io,failover,nolock
PERL_ALARM_TIMEOUT=14
DBFS_PASSWD=welcome1
DBFS_PWDFILE_BASE=/tmp/.dbfs-passwd.txt
WALLET=true
TNS_ADMIN=/u01/app/oracle/admin/dbfs/tnsadmin
DBFS_LOCAL_TNSALIAS=orclb.local
IS_PDB=false
  
[root@rac-nodea ~]# mkdir -p /u01/app/grid/18/crs/script 
[root@rac-nodea ~]# cd /u01/app/grid/18/crs/script 
[root@rac-nodea /u01/app/grid/18/crs/script]# cp /tmp/mount-dbfs.sh . 
[root@rac-nodea /u01/app/grid/18/crs/script]# chmod 750 mount-dbfs.sh
[root@rac-nodea /u01/app/grid/18/crs/script]# chown oracle:dba mount-dbfs.sh
[root@rac-nodea /etc/oracle]# cp /tmp/mount_dbfs.conf /etc/oracle
[root@rac-nodea /etc/oracle]# chmod 640 /etc/oracle/mount-dbfs.conf
[root@rac-nodea /etc/oracle]# chown oracle:dba /etc/oracle/mount-dbfs.conf

NOTE

The nolock mount option is also required if using Oracle Database 18c. 

The direct_io mount option raise issues when installing GG binaries in DBFS.  

Test mount scripts

[oracle@rac-nodea /u01/app/grid/18/crs/script]$ ./mount_dbfs.sh start
mount_dbfs.sh mounting DBFS at /dbfs from database orclb
ORACLE_SID is orcl1
doing mount /dbfs using SID orcl1 with wallet now
nohup: redirecting stderr to stdout
Start -- ONLINE

[oracle@rac-nodea /u01/app/grid/18/crs/script]$ ./mount_dbfs.sh stop 
unmounting DBFS from /dbfs/orcl_ogg 
umounting the filesystem using '/bin/fusermount -u /dbfs'
Stop - stopped, now not mounted 

Add DBFS as resource to clusterware (only on one node)

[oracle@rac-nodea ~]$ crsctl add resource app.orclb.dbfs \
> -type cluster_resource \
> -attr "ACTION_SCRIPT=/u01/app/grid/18/crs/script/mount-dbfs.sh, \
> CHECK_INTERVAL=30, RESTART_ATTEMPTS=10, OFFLINE_CHECK_INTERVAL=60, \
> START_DEPENDENCIES='hard(ora.orclb.db)pullup(ora.orclb.db)', \
> STOP_DEPENDENCIES='hard(ora.orclb.db)', \
> SCRIPT_TIMEOUT=30"
[oracle@rac-nodea ~]$ crsctl start resource app.orclb.dbfs -n `uname -n | cut -d . -f 1`
[oracle@rac-nodea ~]$ df -h
[oracle@rac-nodea ~]$ crsctl stop resource app.orclb.dbfs -n `uname -n | cut -d . -f 1`
[oracle@rac-nodea ~]$ df -h

# relocate resource to another node (resource must be started first)

[oracle@rac-nodea ~]$ crsctl relocate resource app.orclb.dbfs -n rac-nodec
[oracle@rac-nodea ~]$ crsctl stat res app.orclb.dbfs
NAME=app.orclb.dbfs
TYPE=cluster_resource
TARGET=ONLINE
STATE=ONLINE on rac-nodec

NOTE Resource app.orclb.dbfs is registered on only one node, even though the type=cluster_resource. The reason for using cluster_resource is so the filesystem can only be mounted on a single node at one time, preventing the accidental mounting of DBFS from concurrent nodes, creating the potential of concurrent file writes, which causes file corruption problems.