iSCSI – target and initiators

iSCSI (internet Small Computer System Interface) is an IP based storage networking protocol that's designed for sharing block storage over the internet. iSCSI follows the Server-Client model. The Server (aka Target) makes storage available for Clients (aka Initiators) to use.

The Target makes the storage available in the form of a block device (e.g. /dev/sdb). As a result the Initiators views the remote storage as a locally attached block device, and therefore treats the remote block device like an ordinary block device, e.g. you can run command like fdisk, pvcreate, mkfs.ext4,...etc against it. This means that when Initiator successfully establish a session with a Target, one or more block devices in the /dev folder (e.g. /dev/sdc) will suddenly appear and be ready for use.

However, these remote block devices might not perform as well as the more traditional (locally attached) block devices. That's because remote block devices operates over a network so can suffer from data transfer lags.

We will walk through a hello-world example of setting up a target and initiator. The end result of this example will be that a new block device, /dev/sdb, will appear on the Initiator which we will then install a filesystem on and ensure that it persistantly mounts. We've created the a iSCSI demo vagrant project so that you can follow along with this example:

Setting up a Target

First we'll setup a target. To start with we install the target iSCSI software:

$ yum install -y targetcli

Then we need to find a block device to share. The block device can be in any of the following forms:

1. A whole physical block device, e.g. sda, sdb, xvdf...etc
2. A partition, e.g. sda3, sdb2, xvdf2...etc
3. A logical volume
4. Create a special iSCSI image file of the desired size, which is known as a fileio backstore - not recommended because it's performance is not that great.

In our example, we'll use the following block device:

[root@iscsi-target ~]# file /dev/sdb
/dev/sdb: block special

We start creating a target, by using the targetcli command, which starts it's own interactive session:

[root@iscsi-target ~]# targetcli
targetcli shell version 2.1.fb46
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/>

First off you can use the ls command to get an overall view:

/> ls
o- / ..................................................................... [...]
  o- backstores .......................................................... [...]
  | o- block .............................................. [Storage Objects: 0]
  | o- fileio ............................................. [Storage Objects: 0]
  | o- pscsi .............................................. [Storage Objects: 0]
  | o- ramdisk ............................................ [Storage Objects: 0]
  o- iscsi ........................................................ [Targets: 0]
  o- loopback ..................................................... [Targets: 0]
/>

Later on you'll find that there's a json file whose structure mirrors the above tree. This json file get's updated in the background with changes you make in this interactive session. You can traverse this tree using the cd command, and within each section you can run the 'help' command to get more contextual help.

Tip: use the tab+tab autocomplete technique extensively here to explore what commands are available in each section and more quickly construct your commands.

Here are the tasks that needs to be performed in the targetcli:

1. Register the block device
2. Create the iqn - this is something that gets published to initiators when queried
3. Create a LUN (Logical Unit Number) - this associates our block device with the iqn
4. Create name for initiators to use - this creates a private name that initiators can connect to
5. Set username and password (optional)

Step 1 - register the block device: register the /dev/sdb and tag it with a name that will be used to refer to it in the targetcli:

/> backstores/block create BD1 /dev/sdb
Created block storage object BD1 using /dev/sdb.



/> ls
o- / ......................................................................................... [...]
  o- backstores .............................................................................. [...]
  | o- block .................................................................. [Storage Objects: 1]
  | | o- BD1 ............................................ [/dev/sdb (2.0GiB) write-thru deactivated]
  | |   o- alua ................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................ [Targets: 0]
  o- loopback ......................................................................... [Targets: 0]
/>

You can name it anything you want, in my case I labeled it BD1, short for Block-Device-1

Heres another way to do the same thing:

/backstores> cd block
/backstores/block> create BD1 /dev/sdb
Created block storage object BD1 using /dev/sdb.
/backstores/block>

Step 2 - create the IQN: Next we need to create a iSCS Qualified Name (IQN), which is a fully qualified name that will be used to refer to our new backstore block. This IQN is provided to the client when the client scans the server (in discovery mode) for available devices.

So we create a new IQN with the name 'iqn.2018-02.net.cb.target:fqdn'. This is done in the iSCSI folder:

/backstores/block> cd /iscsi
/iscsi> create iqn.2018-04.net.cb.target:fqdn


Created target iqn.2018-04.net.cb.target:fqdn.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.


/iscsi> ls /
o- / ......................................................................................... [...]
  o- backstores .............................................................................. [...]
  | o- block .................................................................. [Storage Objects: 1]
  | | o- BD1 ............................................ [/dev/sdb (2.0GiB) write-thru deactivated]
  | |   o- alua ................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................ [Targets: 1]
  | o- iqn.2018-04.net.cb.target:fqdn .................................................... [TPGs: 1]
  |   o- tpg1 ............................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................... [ACLs: 0]
  |     o- luns .......................................................................... [LUNs: 0]
  |     o- portals .................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................... [OK]
  o- loopback ......................................................................... [Targets: 0]
/iscsi>

Note: the iqn must be unique and follow a specific format which is:

• The 'iqn.' string followed by
• YYYY-MM. - this is usually the year the box's domain is registered, but can be any month+year,
• machine's hostname written in reverse order (Note: our box's hostname is 'target.cb.net'. )
• optional string - this is to help with making the IQN unique. In our example we gave it the string 'fqdn'

Tip: you don't need to memorise this format. You can see an example of this format, when configuring the iscsi client, where an example is given in /etc/iscsi/initiatorname.iscsi, we'll cover this further down.

Notice something called a 'TPG1' has been created. This is short for 'Target Portal Group'. The TPG is something that groups various configurations together. We can take a deeper look like this:

Step 3 - Create a LUN: Next we need to create a LUN (Logical Unit Number). This is done inside the luns folder.

/iscsi> cd /iscsi/iqn.2018-02.net.cb.target:fqdn/tpg1/luns
....luns> create /backstores/block/BD1
Created LUN 0.



/iscsi/iqn.20...qdn/tpg1/luns> ls /
o- / ......................................................................................... [...]
  o- backstores .............................................................................. [...]
  | o- block .................................................................. [Storage Objects: 1]
  | | o- BD1 .............................................. [/dev/sdb (2.0GiB) write-thru activated]
  | |   o- alua ................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................ [Targets: 1]
  | o- iqn.2018-02.net.cb.target:fqdn .................................................... [TPGs: 1]
  |   o- tpg1 ............................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................... [ACLs: 0]
  |     o- luns .......................................................................... [LUNs: 1]
  |     | o- lun0 ........................................ [block/BD1 (/dev/sdb) (default_tg_pt_gp)]
  |     o- portals .................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................... [OK]
  o- loopback ......................................................................... [Targets: 0]

This essentially links our backstore block (BD1) device to our iqn, all grouped under our TPG.

Step 4 - create name for initiators to use: Next we need give our backstore block a unique name that initiators can reference in order to connect to it (note, we are providing the suffix string of 'client'). You can think of this initiator reference as a bit like a username, i.e. a client needs to know what this username is before hand in order to connect to it. Hence this reference is created under the:

/> cd /iscsi/iqn.2018-02.net.cb.target:fqdn/tpg1/acls
/iscsi/iqn.20...qdn/tpg1/acls> create iqn.2018-02.net.cb.target:client
Created Node ACL for iqn.2018-02.net.cb.target:client
Created mapped LUN 0.


/iscsi/iqn.20...qdn/tpg1/acls> ls /
o- / ......................................................................................... [...]
  o- backstores .............................................................................. [...]
  | o- block .................................................................. [Storage Objects: 1]
  | | o- BD1 .............................................. [/dev/sdb (2.0GiB) write-thru activated]
  | |   o- alua ................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................ [Targets: 1]
  | o- iqn.2018-02.net.cb.target:fqdn .................................................... [TPGs: 1]
  |   o- tpg1 ............................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................... [ACLs: 1]
  |     | o- iqn.2018-02.net.cb.target:client ..................................... [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ................................................... [lun0 block/BD1 (rw)]
  |     o- luns .......................................................................... [LUNs: 1]
  |     | o- lun0 ........................................ [block/BD1 (/dev/sdb) (default_tg_pt_gp)]
  |     o- portals .................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................... [OK]
  o- loopback ......................................................................... [Targets: 0]

At this stage, the target is now fully configured to be mounted.

Step 5 (optional) - set username and password: We might want to also password protect this storage. this is possible by running the following command:

/> cd /iscsi/iqn.2018-02.net.cb.target:fqdn/tpg1/acls/iqn.2018-02.net.cb.target:client/
/iscsi/iqn.20...target:client> set auth userid=codingbee
Parameter userid is now 'codingbee'.


/iscsi/iqn.20...target:client> set auth password=password
Parameter userid is now 'password'.

After all that's done, here is what it all finally looks like:

/> ls
o- / ......................................................................................... [...]
  o- backstores .............................................................................. [...]
  | o- block .................................................................. [Storage Objects: 1]
  | | o- BD1 .............................................. [/dev/sdb (2.0GiB) write-thru activated]
  | |   o- alua ................................................................... [ALUA Groups: 1]
  | |     o- default_tg_pt_gp ....................................... [ALUA state: Active/optimized]
  | o- fileio ................................................................. [Storage Objects: 0]
  | o- pscsi .................................................................. [Storage Objects: 0]
  | o- ramdisk ................................................................ [Storage Objects: 0]
  o- iscsi ............................................................................ [Targets: 1]
  | o- iqn.2018-02.net.cb.target:fqdn .................................................... [TPGs: 1]
  |   o- tpg1 ............................................................... [no-gen-acls, no-auth]
  |     o- acls .......................................................................... [ACLs: 1]
  |     | o- iqn.2018-02.net.cb.target:client ..................................... [Mapped LUNs: 1]
  |     |   o- mapped_lun0 ................................................... [lun0 block/BD1 (rw)]
  |     o- luns .......................................................................... [LUNs: 1]
  |     | o- lun0 ........................................ [block/BD1 (/dev/sdb) (default_tg_pt_gp)]
  |     o- portals .................................................................... [Portals: 1]
  |       o- 0.0.0.0:3260 ..................................................................... [OK]
  o- loopback ......................................................................... [Targets: 0]
/>

Now we can exit:

/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json

Notice the json file. This essentially stores everything we have configured using the targetcli command, which includes any usernames and passwords. It looks like this:

[root@target ~]# cat /etc/target/saveconfig.json
{
  "fabric_modules": [],
  "storage_objects": [
    {
      "alua_tpgs": [
        {
          "alua_access_state": 0,
.
.
.
...etc

Next we need to whitelist the iSCSI port in firewalld:

[root@iscsi-target ~]# firewall-cmd --permanent --add-service=iscsi-target
success
[root@iscsi-target ~]# systemctl restart firewalld

Then we enable the target service:

$ systemctl enable target
$ systemctl start target

Setting up an Initiator

Next we log into our initiator box, and install the following:

$ yum install -y iscsi-initiator-utils

Next there are 2 files that needs to be edited, they are both in the following directory:

[root@iscsi-initiator ~]# cd /etc/iscsi/
[root@iscsi-initiator iscsi]# ll
total 20
-rw-r--r--. 1 root root    50 Feb 26 19:44 initiatorname.iscsi
-rw-------. 1 root root 12329 Aug  7  2017 iscsid.conf

First the following file needs to be edited:

[root@iscsi-initiator iscsi]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.1994-05.com.redhat:41d9a65147c3

With the name we picked, so in our case this becomes:

[root@iscsi-initiator iscsi]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2018-05.net.codingbee.iscsi-target:clientmynewscsiblockdevice1

Now if the target is password protected, then you need edit the /etc/iscsi/iscsid.conf, in particular the following section:

.
.
.
# *************
# CHAP Settings
# *************

# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
#node.session.auth.authmethod = CHAP

# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
#node.session.auth.username = username
#node.session.auth.password = password
.
.
.

We need to uncomment these lines and add the credentials, in our example it ends up looking like this:

# *************
# CHAP Settings
# *************

# To enable CHAP authentication set node.session.auth.authmethod
# to CHAP. The default is None.
node.session.auth.authmethod = CHAP

# To set a CHAP username and password for initiator
# authentication by the target(s), uncomment the following lines:
node.session.auth.username = codingbee
node.session.auth.password = password123

Warning: There's another part in this file that looks similar to this segment, so need to make sure not to update the wrong part.

Next we start the iscsi service:

$ systemctl enable iscsi
$ systemctl start iscsi

Now we need to scan our target server for any available iSCSI storage, we do this by using the discovery mode:

[root@iscsi-initiator iscsi]# iscsiadm --mode discovery --type sendtargets --portal 192.168.14.100
192.168.14.100:3260,1 iqn.2018-05.net.codingbee.iscsi-target:mynewscsiblockdevice1

Here, we specified the target server's ip address. Here we sent a request to the target server to send us all available targets. It returns one result.

We are now ready to try to access this. But before we do, let's first run a couple of checks:

[root@iscsi-initiator ~]# lsblk --scsi
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sda  2:0:0:0    disk ATA      VBOX HARDDISK    1.0  sata

Notice only sda is listed here. Also the following doesn't exist yet:

[root@initiator iscsi]# iscsiadm --mode discovery --type sendtargets --portal 192.168.14.100
192.168.14.100:3260,1 iqn.2018-02.net.cb.target:fqdn

Note: always use an ip address for the portal setting. Using a DNS name will cause problems.

Now run the iscsiadm in node mode like this:

[root@initiator iscsi]# iscsiadm --mode node --targetname iqn.2018-02.net.cb.target:fqdn --portal 192.168.14.100 --login

Logging in to [iface: default, target: iqn.2018-02.net.cb.target:fqdn, portal: 192.168.14.100,3260] (multiple)
Login to [iface: default, target: iqn.2018-02.net.cb.target:fqdn, portal: 192.168.14.100,3260] successful.

To confirm that this has worked, run:

[root@initiator iscsi]# lsblk --scsi
NAME HCTL       TYPE VENDOR   MODEL             REV TRAN
sda  2:0:0:0    disk ATA      VBOX HARDDISK    1.0  sata
sdb  3:0:0:0    disk LIO-ORG  BD1              4.0  iscsi

Now you can install a filesystem on it:

[root@initiator ~]# mkfs.xfs -L iscsi_device /dev/sdb
meta-data=/dev/sdb               isize=512    agcount=4, agsize=131072 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=524288, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@initiator ~]# blkid | grep sdb
/dev/sdb: UUID="c00cfb33-2193-4d7d-80d4-f91af3468906" TYPE="xfs"

Note: we'll need this UUID value, see further down.

We then create a mount point for this new filesystem:

[root@initiator iscsi]# mkdir /mnt/remotedisk

We can test this mounts ok by running:

[root@initiator iscsi]# mount /dev/sdb /mnt/remotedisk

[root@initiator iscsi]# lsblk
NAME            MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda               8:0    0  64G  0 disk
├─sda1            8:1    0   1G  0 part /boot
└─sda2            8:2    0  63G  0 part
  ├─centos-root 253:0    0  41G  0 lvm  /
  ├─centos-swap 253:1    0   2G  0 lvm  [SWAP]
  └─centos-home 253:2    0  20G  0 lvm  /home
sdb               8:16   0   2G  0 disk /mnt/remotedisk


[root@initiator ~]# umount /dev/sdb

We unmounted it again, so that we can now mount persistantly using the /etc/fstab appraoch. So here's the line to append:

[root@initiator ~]# cat /etc/fstab

#
# /etc/fstab
# Created by anaconda on Fri Feb  2 05:24:03 2018
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
/dev/mapper/centos-root /                       xfs     defaults        0 0
UUID=9e33c3c1-98df-4fe1-a66f-13ffbaa5f154 /boot                   xfs     defaults        0 0
/dev/mapper/centos-home /home                   xfs     defaults        0 0
/dev/mapper/centos-swap swap                    swap    defaults        0 0
UUID=c00cfb33-2193-4d7d-80d4-f91af3468906 /mnt/remotedisk    xfs     _netdev        0 0

Notice the '_netdev'mount option, this is prevent the boot process hanging if there is any network related problems.

Now we test our fstab entry like this:

[root@initiator ~]# lsblk
NAME            MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda               8:0    0  64G  0 disk
├─sda1            8:1    0   1G  0 part /boot
└─sda2            8:2    0  63G  0 part
  ├─centos-root 253:0    0  41G  0 lvm  /
  ├─centos-swap 253:1    0   2G  0 lvm  [SWAP]
  └─centos-home 253:2    0  20G  0 lvm  /home
sdb               8:16   0   2G  0 disk

[root@initiator ~]# mount -a

[root@initiator ~]# lsblk
NAME            MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda               8:0    0  64G  0 disk
├─sda1            8:1    0   1G  0 part /boot
└─sda2            8:2    0  63G  0 part
  ├─centos-root 253:0    0  41G  0 lvm  /
  ├─centos-swap 253:1    0   2G  0 lvm  [SWAP]
  └─centos-home 253:2    0  20G  0 lvm  /home
sdb               8:16   0   2G  0 disk /mnt/remotedisk

So far so good. Finally we test our remote storage by creating a file in the new storage:

[root@initiator remotedisk]# touch testfile.txt
[root@initiator remotedisk]# ll
total 16
drwx------. 2 root root 16384 Feb 27 11:20 lost+found
-rw-r--r--. 1 root root     0 Feb 27 11:23 testfile.txt
[root@initiator remotedisk]#

The next step is to ensure this remote block device automounts when the machine reboots. This is done by adding a line to the /etc/fstab file.

To get statistics about our iSCSI mount, we can run:

[root@initiator remotedisk]# iscsiadm -m session -P 3
iSCSI Transport Class version 2.0-870
version 6.2.0.874-2
Target: iqn.2018-02.net.cb.target:fqdn (non-flash)
	Current Portal: 192.168.14.100:3260,1
	Persistent Portal: 192.168.14.100:3260,1
		**********
		Interface:
		**********
		Iface Name: default
		Iface Transport: tcp
		Iface Initiatorname: iqn.2018-02.net.cb.target:client
		Iface IPaddress: 192.168.14.101
		Iface HWaddress: 
		Iface Netdev: 
		SID: 1
		iSCSI Connection State: LOGGED IN
		iSCSI Session State: LOGGED_IN
		Internal iscsid Session State: NO CHANGE
		*********
		Timeouts:
		*********
		Recovery Timeout: 120
		Target Reset Timeout: 30
		LUN Reset Timeout: 30
		Abort Timeout: 15
		*****
		CHAP:
		*****
		username: codingbee
		password: ********
		username_in: 
		password_in: ********
		************************
		Negotiated iSCSI params:
		************************
		HeaderDigest: None
		DataDigest: None
		MaxRecvDataSegmentLength: 262144
		MaxXmitDataSegmentLength: 262144
		FirstBurstLength: 65536
		MaxBurstLength: 262144
		ImmediateData: Yes
		InitialR2T: Yes
		MaxOutstandingR2T: 1
		************************
		Attached SCSI devices:
		************************
		Host Number: 3	State: running
		scsi3 Channel 00 Id 0 Lun: 0
			Attached scsi disk sdb		State: running

Finally, you want to unmount this remote storage and stop the iSCSI service everytime your machine is shutdown:

$ umount /dev/sdb
$ systemctl stop iscsi

Also see: https://unix.stackexchange.com/questions/39226/how-to-run-a-script-with-systemd-right-before-shutdown

[post-content post_name=rhsca-quiz]

In the quiz we'll assume we have boxes with the following names:

iSCSI server with hostname target.cb.net
iSCSI client with hostname initiator.cb.net

Server side (target.cb.net) questions:

Client side (initiator.cb.net) questions: