Crash package on Oracle UEK kernel

Kdump is the Linux kernel crash-dump mechanism and Oracle recommends that you enabled it. You can configure Kdump using system-config-kdump utility.

So, if you are running Oracle Enterprise Linux UEK kernel and have faced a kernel panic, follow the steps below to install crash utility and analyse the vmcore file.

To use crash, packages debuginfo and debuginfo-common must also be installed.

To install packages on OL7 run:

yum install crash -y

export DLP="https://oss.oracle.com/ol7/debuginfo"

wget ${DLP}/kernel-uek-debuginfo-`uname -r`.rpm
wget ${DLP}/kernel-uek-debuginfo-common-`uname -r`.rpm

yum install kernel-uek-debuginfo*

Running crash:

syntax: crash <vmcore> <vmlinux>

crash vmcore /usr/lib/debug/lib/modules/`uname -r`/vmlinux

You will find vmcore files under /var/crash/<127.0.0.1-TIMESTAMP>

Output example:

[root@machine 127.0.0.1-2020-01-26-20:29:08]# crash vmcore /usr/lib/debug/lib/modules/4.1.12-124.35.2.el7uek.x86_64/vmlinux
crash 7.2.3-10.el7
Copyright (C) 2002-2017 Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010 IBM Corporation
Copyright (C) 1999-2006 Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012 Fujitsu Limited
Copyright (C) 2006, 2007 VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011 NEC Corporation
Copyright (C) 1999, 2002, 2007 Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002 Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions. Enter "help copying" to see the conditions.
This program has absolutely no warranty. Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-unknown-linux-gnu"…

KERNEL: /usr/lib/debug/lib/modules/4.1.12-124.35.2.el7uek.x86_64/vmlinux
DUMPFILE: vmcore [PARTIAL DUMP]
LOAD AVERAGE: 0.88, 0.26, 0.18
TASKS: 366
NODENAME: machine
RELEASE: 4.1.12-124.35.2.el7uek.x86_64
VERSION: #2 SMP Mon Jan 13 16:42:30 PST 2020
MACHINE: x86_64 (3400 Mhz)
MEMORY: 15.7 GB
PANIC: "BUG: unable to handle kernel NULL pointer dereference at 0000000000000030"
PID: 24893
COMMAND: "oracle_24893_ic"
TASK: ffff880168877000 [THREAD_INFO: ffff8801a6eb8000]
CPU: 0
STATE: TASK_RUNNING (PANIC)

Upgrade from OEL 6 to 7

Oracle 19c does not support OEL 6 so it is now time to upgrade my vm’s to OEL 7.

These are the steps:

  1. Make sure you have repositories ol6_x86_64_latest and ol6_x86_64_addons configured and enabled.
  2. Update to the latest OEL 6 version
    • yum update -y
  3. Install required packages
    • yum install redhat-upgrade-tool preupgrade-assistant preupgrade-assistant-el6toel7 preupgrade-assistant-el6toel7-data-0 preupgrade-assistant-tools preupgrade-assistant-ui openscap
  4. Run the upgrade assessment
    • preupg
  5. Review the output and if the assessment reports any fail, needs_action, or needs_inspection issues, read the remediation instructions for these issues and perform any required actions before proceeding with the upgrade.
  6. Download the ISO file from Edelivery.oracle.com. Choose version OEL 7.5.
  7. Copy the ISO to the server
  8. Run the upgrade tool
    1. redhat-upgrade-tool-cli –iso=V975367-01.iso –debuglog=/tmp/upgrade.log –cleanup-post
  9. Reboot the system to start the upgrade

Enjoy !

CRS crash after upgrade to Oracle 12.2

vector illustration of Crazy man cartoon

Cai num bug “interessante” na última semana após a atualização para o Oracle 12.2

I have hit an interesting bug this week after upgrade to Oracle 12.2.

A nota Doc ID 2460394.1 tem os detalhes do bug mas basicamente sua base será reiniciada pelo CRS de tempos em tempos.

MOS note Bug 28298447: cluster crashed due to mellanox driver related issue (Doc ID 2460394.1) has the details but basically your cluster will bounce the DB from time to time.

Ou seja, se você tem um Exadata, atualize o kernel para a versão 4.1.12-94.8.5 antes de realizar o upgrade do banco para 12.2

So if you running an Exadata, upgrade the kernel to version 4.1.12-94.8.5 before upgrading db to 12.2.

Essa correção de kernel também está disponível nas versões 12.2.1.1.7 (ou superior) ou 18.1.5.0.0 (ou superior), ou seja, QFSDP abril/2018.

This kernel fix is also included on image version 12.2.1.1.7 (or higher) or 18.1.5.0.0 (or higher), or QFSDP from april/2018.

ODC Appreciation Day: LACP

odc

The ODC Appreciation Day was proposed by Tim Hall and you can find more information here: https://oracle-base.com/blog/2017/09/25/odc-appreciation-day-2017-thanksodc/

Link Aggregation Control Protocol (LACP), also known as 802.3ad is a methods of combining multiple physical network connections into one logical connection to increase throughput and provide redundancy in case one of the links should fail. The protocol requires both – the server and the switch(es) to have the same settings to allow LACP to work properly.

To configure LACP you need to change your bonding device configuration in /etc/sysconfig/network-scripts/ifcfg-bond0 similarly to this:

DEVICE=bond0
IPADDR=192.168.0.2
NETMASK=255.255.255.0
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
NM_CONTROLLED=no
MTU=9000
BONDING_OPTS="mode=4 miimon=100 downdelay=5000 updelay=5000 num_grat_arp=100 xmit_hash_policy=layer3+4 lacp_rate=1"

The key here is setting the MTU=9000 and establishing the transmit hash policy as layer3+4. This transmit policy is intended to more-evenly distribute the traffic across the physical links of the bonded device.

This is is the preferred mode of network bonding, but requires that the network is configured correctly on the switch.

You can then test the performance using iperf linux tool.

More information on https://wiki.linuxfoundation.org/networking/bonding

 

 

 

 

yum update error: Metadata file does not match checksum Trying other mirror.

If you get the error below:

http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/repodata/filelists.xml.gz: [Errno -1] Metadata file does not match checksum
Trying other mirror.
public_ol6_latest/primary                                |  39 MB     01:48
http://public-yum.oracle.com/repo/OracleLinux/OL6/latest/x86_64/repodata/primary.xml.gz: [Errno -1] Metadata file does not match checksum
Trying other mirror.
Error: failure: repodata/primary.xml.gz from public_ol6_latest: [Errno 256] No more mirrors to try.

Do not panic 🙂 Fix is simple, just:

1) Include http_caching=none in your /etc/yum.conf file
2) run yum clean metadata then
3) yum update