Rogerio Eguchi Blog

OCI – Cross-region volume backup copy

This feature has been available for a long time but has an interesting restriction that is fully documented here.

“When copying block volume backups across regions in your tenancy, you can copy up to five concurrent backups per tenancy at a time from a specific source region.”

Yes, that is right, it is only 5 parallel copies.

We recently moved a big customer to OCI and ended up reaching this limit.

This is of course a soft limit so I’ve tried to raise a request on MOS for them to increase it to 10.

Answer was no.

My workaround in this case was remove block volumes backups from DB servers from the backup policy. They are the bigger ones and frequently changed.

So from now on we have RMAN backups only.

FYI, this same limit is 20 on AWS.

OCI – ol-consolebaud.service

This service will be installed when you provision a VM on OCI.

But in some VM shapes this service will not start.

Additional information provided by Oracle Support about it:

“The ol-consolebaud service is a sanity check for the system console. Many OCI instances still use 9600 baud serial consoles, and this provides a check for it. The reason for the check is that slow system consoles can cause problems with diagnostics when a kernel bug occurs.”

You can list the failed services running:

systemctl list-units --state=failed

The output on my VM.Standard2.1 is:

[root@server ~]# systemctl list-units --state=failed
   UNIT                   LOAD   ACTIVE SUB    DESCRIPTION
 ● ol-consolebaud.service loaded failed failed Check console baud rate

 LOAD   = Reflects whether the unit definition was properly loaded.
 ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
 SUB    = The low-level unit activation state, values depend on unit type.

 1 loaded units listed. Pass --all to see loaded but inactive units, too.
 To show all installed unit files use 'systemctl list-unit-files'.

[root@server ~]# systemctl status ol-consolebaud.service
 ● ol-consolebaud.service - Check console baud rate
    Loaded: loaded (/usr/lib/systemd/system/ol-consolebaud.service; enabled; vendor preset: disabled)
    Active: failed (Result: exit-code) since Mon 2021-02-22 19:13:49 EST; 20h ago
  Main PID: 982 (code=exited, status=1/FAILURE)
 Feb 22 19:13:49 c0280579 systemd[1]: Starting Check console baud rate…
 Feb 22 19:13:49 c0280579 ol-consolebaud[982]: Serial console is set to the default of 9600 baud. This can
 Feb 22 19:13:49 c0280579 ol-consolebaud[982]: result in stalls or lockups in error conditions requiring a
 Feb 22 19:13:49 c0280579 ol-consolebaud[982]: large number of console system messages. Please increase the
 Feb 22 19:13:49 c0280579 systemd[1]: ol-consolebaud.service: main process exited, code=exited, status=1/FAILURE
 Feb 22 19:13:49 c0280579 ol-consolebaud[982]: rate to the highest your system will allow (for instance, 115200
 Feb 22 19:13:49 c0280579 systemd[1]: Failed to start Check console baud rate.
 Feb 22 19:13:49 c0280579 ol-consolebaud[982]: or 57600). See Oracle KM Note 2648582.1 for more information.
 Feb 22 19:13:49 c0280579 systemd[1]: Unit ol-consolebaud.service entered failed state.
 Feb 22 19:13:49 c0280579 systemd[1]: ol-consolebaud.service failed.

Fix in this case is simple:

1.- Modify the GRUB2 configuration file:

vim /etc/default/grub

2.- If the following lines are not in the file, please add them:

GRUB_SERIAL_COMMAND="serial --unit=0 --speed=115200"
GRUB_TERMINAL="serial console"

3.- Looking for console=ttyS0,9600 in “GRUB_CMDLINE_LINUX” line and change it for “console=ttyS0,115200”.

4.- Regenerate the GRUB2 configuration:

grub2-mkconfig -o /boot/efi/EFI/redhat/grub.cfg

It would be nice to have an error free image from OCI.

And I’m still trying to access MOS note 2648582.1 mentioned above 🙂

OCI – Oracle Cloud Agent Commands

Oracle Cloud Agent Commands is another useful feature recently launched on OCI.

This resource gives you the ability of remotely running scripts within the instance using the run command feature.

Let’s say you need to restart the sshd deamon to be able to login again.

Not a problem, just go to the console, hit create command and write a script to start it up again 🙂

After a few minutes you can check the output:

Return code was sucessfull but it actually failed with error below:

Failed to start sshd.service: Interactive authentication required.
See system logs and 'systemctl status sshd.service' for details.
Mon Dec 14 21:21:05 GMT 2020

For privileged commands to work you need to grant sudo privileges to ocarun user.

You can also check the logs at /var/log/oracle-cloud-agent/plugins/runcommand/runcommand.log file.

IAM policies are required for this feature to work so go ahead and read the documentation here.

AWS – EBS gp3

AWS announced the general availability of EBS gp3 which is 20% cheaper then gp2 and allows you to provision performance (IOPS and throughput) independent of storage capacity.

Baseline performance on a 1Tb gp2 volume is 3000 IOPS and 250 mb/s throughput.

Baseline performance on gp3 is 3000 IOPS and 125mb/s throughput, regardless of volume size.

Provisioning gp3 performance will cost you a fee.

Example:

Detailed pricing information available at https://aws.amazon.com/ebs/pricing/

So, from now on, instead of using gp2, just go with gp3 and use more disks so you can reach the same throughput as gp2 and save a few (or a lot) of money.

Check the instance type volume limits here so you can better understand how many volumes you need to reach instance max performance.

We have started moving some volumes from gp2 to gp3 and seeing some disk latency increase so I recommend you doing it on most idle periods.

Variable substitution using new_val

This is not new but I wasn’t aware of this cool feature !

You can assign a value to a variable using new_val on SQLPlus.

From the doc: “Specifies a variable to hold a column value”

Example:

column PROPERTY_VALUE new_val temporary_tablespace

select PROPERTY_VALUE from database_properties where PROPERTY_NAME = 'DEFAULT_TEMP_TABLESPACE';

This way I can easily assign the output from the select command to the variable temporary_tablespace.

You can check the variable’s value using define command:

define temporary_tablespace

Easy, right ?

OCI – ExaCS provisioning bug

If you are about to launch an ExaCS using the new resource model be aware of BUG 32104352 – Provisioning fails with minimum value for vcpu count is 4.

You should be able to launch an ExaCS with only one ocpu in each node but it will fail to provision due to this bug.

Fix is very simple, you just need to define a value of 4 or above for the OCPU.

After provisioning is completed, you can scale up/down the OCPU’s online.

Happy provisioning 🙂

Faster ASM Rebalance

Yes, it is possible to accomplish it but only if you have a flash storage.

Why ?

Because in this case you can skip the Compact Phase of ASM rebalance operation.

For 12c onwards, you can use the command below to skip the compact phase:

ALTER DISKGROUP <DG_NAME> SET ATTRIBUTE '_rebalance_compact'='FALSE';

But before running this command, please read the following articles so you can better understand what you are doing.

Rebalancing act – http://asmsupportguy.blogspot.com/2011/11/rebalancing-act.html

MOS Note 1902001.1 – What is ASM rebalance compact Phase and how it can be disabled

ASM Rebalance Too Slow? 3 Tips To Improve Rebalance Times – https://flashdba.com/2015/04/17/asm-rebalance-too-slow-3-tips-to-improve-rebalance-times/

Statspack batch install

If you don’t have Oracle Diagnostics pack or is running Oracle Standard Edition, just go for statspack !

In this blog post I will show you how to install it in batch mode.

It is actually very simple, you just need to define three variables:

connect / as sysdba

define default_tablespace='perfstat'
define temporary_tablespace='temp01'
define perfstat_password='YourComplexStatspackPassword'

then run

@?/rdbms/admin/spcreate.sql

Simple, isn’t ?

I would also recommend you to take a look at Franck’s way of improving statspack here.

OCI – iSCSI bug 30711156

If you run Oracle DB on OCI compute and leverage iSCSI as volume attachment, beware of bug 30711156 on iSCSI.

We hit this bug a while ago and as consequence we were not able to read/write to the block volume anymore.

Fix: kill all Oracle processes then remount the Filesystem.

If you see errors on /var/log/messages like the ones below, you mostly like hit the same issue:

Aug 10 00:29:30 host iscsid: iscsid: Kernel reported iSCSI connection 1:0 error (1020 - ISCSI_ERR_TCP_CONN_CLOSE: TCP connection closed) state (3) 
Aug 10 00:29:30 host iscsid: iscsid: re-opening session 1 (reopen_cnt 0) 
Aug 10 00:29:30 host iscsid: iscsid: disconnecting conn 0x563c2f155068, fd 7 
Aug 10 00:33:01 host kernel: session1: iscsi_eh_cmd_timed_out scsi cmd ffff9c3622aea948 timedout 
Aug 10 00:33:01 host kernel: session2: iscsi_eh_cmd_timed_out scsi cmd ffff9c3622ae8d48 timedout 
Aug 10 00:33:01 host kernel: session1: iscsi_eh_cmd_timed_out return timer reset 
Aug 10 00:33:01 host kernel: session2: iscsi_eh_cmd_timed_out return shutdown or nh 
Aug 10 00:33:01 host kernel: session1: iscsi_eh_cmd_timed_out scsi cmd ffff9c3622aec148 timedout 
Aug 10 00:33:01 host kernel: session1: iscsi_eh_cmd_timed_out return timer reset

Nice, right ?

A new iscsi-initiator-utils is available for download so go ahead and update your server.

Linux Errata available here.

Good patching !

AWS cli for rds reports

Another quick blog post on AWS stuff.

You can query your RDS metadata information using aws cli.

This is a very useful approach when you manage hundreds of servers and need to build a report.

Here is the command line I’ve got to retrieve the database name, license model, DB engine and DB version.

aws rds describe-db-instances --region us-east-1 --query "*[].{DBInstanceIdentifier:DBInstanceIdentifier,LicenseModel:LicenseModel,Engine:Engine,EngineVersion:EngineVersion}" --output table

You can find other rds cli options here.