DB system – OS patching

When using Oracle Cloud Database Services VM or Bare Metal, customer is responsible for OS updates and GI/DB patches so let’s first go through the steps to update an OL7 server.

This procedure is not applicable for Exadata DB systems.

General Recommendations

1) Don’t touch oraenv or .bash_profile

2) Don’t touch default local firewall rules


Backup your db prior to the OS update

NonProd first

Test this procedure on non prod server first

OS Update

1) Check if kernel is 4.1.12-124.27.1.el7uek then you need to change the bootefi label before updating the OS.

uname -r

To change bootefi label:

Edit /etc/fstab: Change the label bootefi to BOOTEFI (uppercase), reboot the server and run ls -l /etc/grub2-efi.cfg to check required link was created.

2) Run command below to identify the region server is running:

curl -s |grep region

3) Download the repo file and cp to yum dir:

wget https://swiftobjectstorage.<region>.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/oci_dbaas_ol7repo -O /tmp/oci_dbaas_ol7repo

and copy it to yum dir

cp /tmp/oci_dbaas_ol7repo /etc/yum.repos.d/ol7.repo

4) Download the version lock files and overwrite existing one:

wget https://swiftobjectstorage.<region>.oraclecloud.com/v1/dbaaspatchstore/DBaaSOSPatches/versionlock_ol7.list -O /tmp/versionlock.list

cp /etc/yum/pluginconf.d/versionlock.list /etc/yum/pluginconf.d/versionlock.list-`date+%Y%m%d`
cp /tmp/versionlock.list /etc/yum/pluginconf.d/versionlock.list

5) run YUM update

yum update

6) reboot

7) check new kernel version

uname -r

Hopefully I can rely on OS Management to update this type of server next time ūüôā So stay tuned !

Crash package on Oracle UEK kernel

Kdump is the Linux kernel crash-dump mechanism and Oracle recommends that you enabled it. You can configure Kdump using system-config-kdump utility.

So, if you are running Oracle Enterprise Linux UEK kernel and have faced a kernel panic, follow the steps below to install crash utility and analyse the vmcore file.

To use crash, packages debuginfo and debuginfo-common must also be installed.

To install packages on OL7 run:

yum install crash -y

export DLP="https://oss.oracle.com/ol7/debuginfo"

wget ${DLP}/kernel-uek-debuginfo-`uname -r`.rpm
wget ${DLP}/kernel-uek-debuginfo-common-`uname -r`.rpm

yum install kernel-uek-debuginfo*

Running crash:

syntax: crash <vmcore> <vmlinux>

crash vmcore /usr/lib/debug/lib/modules/`uname -r`/vmlinux

You will find vmcore files under /var/crash/<>

Output example:

[root@machine]# crash vmcore /usr/lib/debug/lib/modules/4.1.12-124.35.2.el7uek.x86_64/vmlinux
crash 7.2.3-10.el7
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"…

KERNEL: /usr/lib/debug/lib/modules/4.1.12-124.35.2.el7uek.x86_64/vmlinux
LOAD AVERAGE: 0.88, 0.26, 0.18
TASKS: 366
NODENAME: machine
RELEASE: 4.1.12-124.35.2.el7uek.x86_64
VERSION: #2 SMP Mon Jan 13 16:42:30 PST 2020
MACHINE: x86_64 (3400 Mhz)
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000030"
PID: 24893
COMMAND: "oracle_24893_ic"
TASK: ffff880168877000 [THREAD_INFO: ffff8801a6eb8000]
CPU: 0

Upgrade from OEL 6 to 7

Oracle 19c does not support OEL 6 so it is now time to upgrade my vm’s to OEL 7.

These are the steps:

  1. Make sure you have repositories ol6_x86_64_latest and ol6_x86_64_addons configured and enabled.
  2. Update to the latest OEL 6 version
    • yum update -y
  3. Install required packages
    • yum install redhat-upgrade-tool preupgrade-assistant preupgrade-assistant-el6toel7 preupgrade-assistant-el6toel7-data-0 preupgrade-assistant-tools preupgrade-assistant-ui openscap
  4. Run the upgrade assessment
    • preupg
  5. Review the output and if the assessment reports any fail, needs_action, or needs_inspection issues, read the remediation instructions for these issues and perform any required actions before proceeding with the upgrade.
  6. Download the ISO file from Edelivery.oracle.com. Choose version OEL 7.5.
  7. Copy the ISO to the server
  8. Run the upgrade tool
    1. redhat-upgrade-tool-cli –iso=V975367-01.iso –debuglog=/tmp/upgrade.log –cleanup-post
  9. Reboot the system to start the upgrade

Enjoy !

CRS crash after upgrade to Oracle 12.2

vector illustration of Crazy man cartoon

Cai num bug “interessante” na √ļltima semana ap√≥s a atualiza√ß√£o para o Oracle 12.2

I have hit an interesting bug this week after upgrade to Oracle 12.2.

A nota Doc ID 2460394.1 tem os detalhes do bug mas basicamente sua base será reiniciada pelo CRS de tempos em tempos.

MOS note Bug 28298447: cluster crashed due to mellanox driver related issue (Doc ID 2460394.1) has the details but basically your cluster will bounce the DB from time to time.

Ou seja, se você tem um Exadata, atualize o kernel para a versão 4.1.12-94.8.5 antes de realizar o upgrade do banco para 12.2

So if you running an Exadata, upgrade the kernel to version 4.1.12-94.8.5 before upgrading db to 12.2.

Essa corre√ß√£o de kernel tamb√©m est√° dispon√≠vel nas vers√Ķes (ou superior) ou (ou superior), ou seja, QFSDP abril/2018.

This kernel fix is also included on image version (or higher) or (or higher), or QFSDP from april/2018.

ODC Appreciation Day: LACP


The ODC Appreciation Day was proposed by Tim Hall and you can find more information here: https://oracle-base.com/blog/2017/09/25/odc-appreciation-day-2017-thanksodc/

Link Aggregation Control Protocol (LACP), also known as¬†802.3ad¬†is a methods of combining multiple physical network connections into one logical connection to increase throughput and provide redundancy in case one of the links should fail. The protocol requires both ‚Äď the server and the switch(es) to have the same settings to allow LACP to work properly.

To configure LACP you need to change your bonding device configuration in /etc/sysconfig/network-scripts/ifcfg-bond0 similarly to this:

BONDING_OPTS="mode=4 miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100 xmit_hash_policy=layer3+4 lacp_rate=1"

The key here is setting the MTU=9000 and establishing the transmit hash policy as layer3+4. This transmit policy is intended to more-evenly distribute the traffic across the physical links of the bonded device.

This is is the preferred mode of network bonding, but requires that the network is configured correctly on the switch.

You can then test the performance using iperf linux tool.

More information on https://wiki.linuxfoundation.org/networking/bonding





yum update error: Metadata file does not match checksum Trying other mirror.

If you get the error below:

http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/repodata/filelists.xml.gz: [Errno -1] Metadata file does not match checksum
Trying other mirror.
public_ol6_latest/primary                                |  39 MB     01:48
http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/repodata/primary.xml.gz: [Errno -1] Metadata file does not match checksum
Trying other mirror.
Error: failure: repodata/primary.xml.gz from public_ol6_latest: [Errno 256] No more mirrors to try.

Do not panic ūüôā Fix is simple, just:

1) Include http_caching=none in your /etc/yum.conf file
2) run yum clean metadata then
3) yum update