Error Message

# hatsatus -sum

-- SYSTEM STATE
-- System               State                Frozen              

A  server101            RUNNING              0                    
A  server102            RUNNING              0                    

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State          
B  vxfen           server101            Y          N               OFFLINE|FAULTED
B  vxfen           server102            Y          N               OFFLINE|FAULTED

-- RESOURCES FAILED
-- Group           Type                 Resource             System              

D  vxfen           CoordPoint           coordpoint           server101           
D  vxfen           CoordPoint           coordpoint           server102           
 

# cpsadm -s 192.168.10.225 -a ping_cps

CPS ERROR V-97-1400-937 None of the SSL certificates found in directory /var/VRTSvxfen/security/certs/ could be used to successfully connect to 192.168.10.225 

Client SSL/TLS Certificates have expired. 

[root@server101 ~]# date
Mon Jun  3 14:36:04 PDT 2024

[root@server101 ~]# openssl x509 -in /var/VRTSvxfen/security/certs/client_192.168.10.225.crt -text -noout 
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = IN, L = Pune, OU = VCS, CN = CACERT
        Validity
            Not Before: May 25 21:36:10 2014 GMT
            Not After : May 22 21:36:10 2024 GMT
        Subject: C = IN, L = Pune, OU = VCS, CN = {c7c2d65e-058f-11e8-a32c-c094107f3b71}
 

Renew the Certificates for the clients along with the Certificate Authority (CA) and Server certificates if required. 

Each Coordinator point server provides a Certificate Authority (CA), Server Keys/Certificates, Client Keys/Certificates.  

Both the Keys and certificates for the Coordinator Point Server are located under /var/VRTScps/security. The location of the key/certificates is defined by /etc/vxcps_ssl.properties.

openSSL.server.privateKeyFile = /var/VRTScps/security/keys/server_private.key
openSSL.server.certificateFile = /var/VRTScps/security/certs/server.crt
openSSL.server.caConfig = /var/VRTScps/security/certs/ca.crt
 

Certificate Directory structure on Coordinator Point Server:

Server Keys/Certificates

/var/VRTScps/security/
├── certs
│   ├── ca.crt          <-- Certificate Authority (CA)
│   └── server.crt    <-- Server Certificate for use with the vxcpserv process. 
├── keys
     ├── ca.key         <-- Certifcate Authority (CA) Key
     └── server_private.key  <-- Server Key 
 

Client Keys/Certificates

/var/VRTSvxfen/security/
├── certs
│   ├── ca_cpserver205.crt
│   ├── client_cpserver205.crt
│   ├── client_drserver201_{c7c2d65e-058f-11e8-a32c-c094107f3b85}.crt
│   ├── client_drserver202_{c7c2d65e-058f-11e8-a32c-c094107f3b85}.crt
│   ├── client_{f5b1ced0-cab2-11ee-90a6-1533111db867}.crt
│   ├── client_server101_{c7c2d65e-058f-11e8-a32c-c094107f3b71}.crt
│   └── client_server102_{c7c2d65e-058f-11e8-a32c-c094107f3b71}.crt
└── keys
    ├── client_private_drserver201_{c7c2d65e-058f-11e8-a32c-c094107f3b85}.key
    ├── client_private_drserver202_{c7c2d65e-058f-11e8-a32c-c094107f3b85}.key
    ├── client_private_{f5b1ced0-cab2-11ee-90a6-1533111db867}.key
    ├── client_private.key
    ├── client_private_server101_{c7c2d65e-058f-11e8-a32c-c094107f3b71}.key
    └── client_private_server102_{c7c2d65e-058f-11e8-a32c-c094107f3b71}.key
 

Certificate Directory structure on Coordinator Point Clients:

/var/VRTSvxfen/security/

├── certs
│   ├── ca_192.168.10.225.crt <- CA Certificate
│   ├── ca_192.168.10.226.crt <- CA Certificate
│   ├── ca_192.168.10.227.crt <- CA Certificate
│   ├── client_192.168.10.225.crt <- Client Certificate
│   ├── client_192.168.10.226.crt <- Client Certificate
│   └── client_192.168.10.227.crt <- Client Certificate
└── keys
    └── client_private.key  <- Private Key

Ten years is the default expiration date for certificates issued by the  Coordinator point server. Once the Not After date is reached, the Coordinator Point Clients are no longer able to communicate with the Coordinator Point Servers. 

1.) Renew the Certificate Authority (CA) certificate used for signing the client certificates, if it is no longer valid.  

[root@cpserver205 ~]# date
Mon Jun  3 14:41:05 PDT 2024

[root@cpserver205 ~]# openssl x509 -in /var/VRTScps/security/certs/ca.crt -text -noout 
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            07:d0:57:34:1f:1d:00:a9:12:35:ca:ad:25:64:c2:a0:89:2f:b9:9c
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = IN, L = Pune, OU = VCS, CN = CACERT
        Validity
            Not Before: May 25 12:48:31 2014 GMT
            Not After : May 22 12:48:31 2024 GMT
        Subject: C = IN, L = Pune, OU = VCS, CN = CACERT

    Renewing the CA certificate:

    # /opt/VRTSperl/non-perl-libs/bin/openssl req -new -x509 -days 3650 -sha256 -key /var/VRTScps/security/keys/ca.key -subj '/C=IN/L=Pune/OU=VCS/CN=CACERT' -out /var/VRTScps/security/certs/ca.crt -config /opt/VRTSperl/non-perl-libs/bin/openssl.cnf 

  # openssl x509 -in /var/VRTScps/security/certs/ca.crt -text -noout 
  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number:
              a2:f4:4c:f3:56:86:19:c8
          Signature Algorithm: sha256WithRSAEncryption
          Issuer: C = IN, L = Pune, OU = VCS, CN = CACERT
          Validity
              Not Before: Jun  3 22:18:23 2024 GMT
              Not After : Jun  1 22:18:23 2034 GMT
          Subject: C = IN, L = Pune, OU = VCS, CN = CACERT
 

2.) Renew the server certificate using the updated CA.

# openssl x509 -in /var/VRTScps/security/certs/server.crt -text -noout
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 1 (0x1)
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = IN, L = Pune, OU = VCS, CN = CACERT
        Validity
            Not Before: May 25 12:59:16 2014 GMT
            Not After : May 22 12:59:16 2024 GMT
        Subject: C = IN, L = Pune, OU = VCS, CN = {f5b1ced0-cab2-11ee-90a6-1533111db867}
 

  Renewing the server certificate:

Create Configuration File /var/VRTScps/security/keys.httpsconfig using the IP, Hostnmame and fully qualified domain name of the CP Server. 

[ req ]distinguished_name     = req_distinguished_namereq_extensions         = v3_req[ req_distinguished_name ]countryName = Country Name (2 letter code)countryName_default = USlocalityName = Locality Name (eg, city)organizationalUnitName = Organizational Unit Name (eg, section)commonName = Common Name (eg, YOUR name)commonName_max = 64emailAddress = Email AddressemailAddress_max = 40[v3_req]keyUsage               = keyEncipherment, dataEnciphermentextendedKeyUsage       = serverAuthsubjectAltName         = @alt_names[alt_names]DNS.1  = 192.168.10.225DNS.2  = cpserver225 DNS.3  = cpserver225.domain.nameThe Cluster UUID of the CP Server must be set to the Common Name (CN) of the certificate. 

  # cat /etc/vx/.uuids/clusuuid

  {f5b1ced0-cab2-11ee-90a6-1533111db867}

Create the Certificate Sign Request (CSR)

  # /opt/VRTSperl/non-perl-libs/bin/openssl req -new -sha256 -key /var/VRTScps/security/keys/server_private.key -config /var/VRTScps/security/keys.httpsconfig -subj '/C=IN/L=Pune/OU=VCS/CN={f5b1ced0-cab2-11ee-90a6-1533111db867}' -out /var/VRTScps/security/certs/server.csr

Create the certificate 

  # /opt/VRTSperl/non-perl-libs/bin/openssl x509 -req -days 3650 -sha256 -in /var/VRTScps/security/certs/server.csr -CA /var/VRTScps/security/certs/ca.crt -CAkey /var/VRTScps/security/keys/ca.key -set_serial 01 -extensions v3_req -extfile /var/VRTScps/security/keys.httpsconfig -out /var/VRTScps/security/certs/server.crt
Signature ok
subject=/C=IN/L=Pune/OU=VCS/CN={f5b1ced0-cab2-11ee-90a6-1533111db867}
Getting CA Private Key

  # openssl x509 -in /var/VRTScps/security/certs/server.crt -text -noout

  Certificate:
      Data:
          Version: 3 (0x2)
          Serial Number: 1 (0x1)
          Signature Algorithm: sha256WithRSAEncryption
          Issuer: C = IN, L = Pune, OU = VCS, CN = CACERT
          Validity
              Not Before: Jun  3 22:22:44 2024 GMT
              Not After : Jun  1 22:22:44 2034 GMT
          Subject: C = IN, L = Pune, OU = VCS, CN = {f5b1ced0-cab2-11ee-90a6-1533111db867}

3.) Restart CPSSG service group to reload the new certificate. 

  # hagrp -offline CPSSG -sys `uname -n`;hagrp -wait CPSSG State OFFLINE -sys `uname -n`;hagrp -online CPSSG -sys `uname -n`;hagrp -wait CPSSG State ONLINE -sys `uname -n`

4.) Renew client certificates using the updated Certificate Authority (CA) certificate. 

Note: Make sure to use the UUID of the cluster when creating the client certificate 

cluster# cat /etc/vx/.uuids/clusuuid 
  {c7c2d65e-058f-11e8-a32c-c094107f3b71}

  # /opt/VRTSperl/non-perl-libs/bin/openssl req -new -sha256 -key /var/VRTSvxfen/security/keys/client_private_{c7c2d65e-058f-11e8-a32c-c094107f3b71}.key -subj '/C=IN/L=Pune/OU=VCS/CN={cff28f36-c747-11ee-bfba-e488cf18674f}' -out /var/VRTSvxfen/security/certs/client_{c7c2d65e-058f-11e8-a32c-c094107f3b71}.csr -config /opt/VRTSperl/non-perl-libs/bin/openssl.cnf

  # /opt/VRTSperl/non-perl-libs/bin/openssl x509 -req -days 3650 -sha256 -in /var/VRTSvxfen/security/certs/client_{c7c2d65e-058f-11e8-a32c-c094107f3b71}.csr -CA /var/VRTScps/security/certs/ca.crt -CAkey /var/VRTScps/security/keys/ca.key -set_serial 01 -out /var/VRTSvxfen/security/certs/client_{c7c2d65e-058f-11e8-a32c-c094107f3b71}.crt 

5.) Copy the updated CA and Client certificate from the CP Server to the CP Client. 

    Note: This can be done using scp or copy and paste. 

Example:

  #  scp client_server101_\{c7c2d65e-058f-11e8-a32c-c094107f3b71\}.crt server101:/var/VRTSvxfen/security/certs/client_192.168.10.225.crt

  # scp client_server102_\{c7c2d65e-058f-11e8-a32c-c094107f3b71\}.crt server102:/var/VRTSvxfen/security/certs/client_192.168.10.225.crt

  # scp /var/VRTScps/security/certs/ca.crt server101:/var/VRTSvxfen/security/certs/ca_192.168.10.225.crt

  # scp /var/VRTScps/security/certs/ca.crt server102:/var/VRTSvxfen/security/certs/ca_192.168.10.225.crt