Quantcast
Channel: UC Corner
Viewing all 54 articles
Browse latest View live

Network Automation with TCL

$
0
0
My company has a large (and not very well managed) network.  Different VLANs have different "ip helper-address" (DHCP) settings.  Some of them are pointing to development DHCP, some of them pointing to production DHCP, some of them are pointing to VDI environment.

For the production environment, there are two DHCP servers (A and B).  They are supposed to back up each other.  However some of the VLANs might be configured to use one of them but not both.

The DHCP administrator wanted to perform a DR test, which he would shut down one of the DHCP servers and see if the clients still be able to get IP address from the other server.

Before he can perform the DR test, he needs to make sure that DHCP server A and B are referenced in pairs, which means a interface either references both servers, or reference neither server.  If the interface references only one of the two, it'd be a problem.  We need to fix this problem before the DR test.

So the workflow is pretty straight forward:
1) SSH into a switch (where the  IP interfaces are configured).
2) Use "show ip interface brief | exclude unassigned" to display the interfaces with IP configured.
3) Use "show run interface xxx" command to review the configuration.  If one of the DHCP servers (A or B) was referenced but the other one of missing, we need to add the other one to the configuration.

* If both are present, it's fine.  If none of them present, that's fine.

This is a simple but tedious work.  Because we have a bunch of switches.  Some of the switches have more than 50 SVIs.  Visual inspection would be  time consuming and prone to human errors.

This is where automation should kick in.  You may use other program languages.  I chose TCL because it was built in on IOS.

In privilege (enable) mode, type "tclsh" to get into TCL shell.  Copy and paste the script into the command line.  It will create a procedure called "check".  Then type "check" (without quotation marks).  Below are some sample outputs:

Example 1: No problem found.

Example 2: Found some interfaces missing one of the helpers.











Script is as below.
proc check {} {

# Define the two commands we want to check
set str_helper1 "ip helper-address 10.50.23.27"
set str_helper2 "ip helper-address 10.202.32.30"

# Define missing_commands string
set str_missing_cmds ""

# List all the interfaces with IP address.
set str_sh_ip_int [exec sh ip int br | ex una]

# Break the output into individual lines and put them in a list.  First two lines are headers.
set list_int [split $str_sh_ip_int \n]

# Get the number of lines
set num_num_of_lines [llength $list_int]

# Skip headers, start from 1st interface
set num_current_line 2

# Process each each line (each IP interface).
while { $num_current_line < $num_num_of_lines } {

    # Get the interface name from each line.
    set str_int_name [lindex [split [lindex $list_int $num_current_line]] 0]

    # Do a "show run interface" againt the interface name.
    set str_sh_run_int [exec sh run int $str_int_name]

    # See if helper1 exists but not helper2
    if { [string match "*$str_helper1*" $str_sh_run_int] && ![string match "*$str_helper2*" $str_sh_run_int] } {
        append str_missing_cmds "interface $str_int_name\n $str_helper2\n"

    # See if helper2 exists but not helper1
    } elseif { [string match "*$str_helper2*" $str_sh_run_int] && ![string match "*$str_helper1*" $str_sh_run_int] } {
        append str_missing_cmds "interface $str_int_name\n $str_helper1\n"
    }

    # Move to next line (next IP interface)
    incr num_current_line
}

# Missing helper on some of the interfaces
if {[string length $str_missing_cmds]} {
    puts "\n\nIP Interface(s):"
    puts "----------------"
    puts $str_sh_ip_int
    puts "\n[expr $num_num_of_lines-2] interface(s) checked.  The following commands are missing:\n"
    puts $str_missing_cmds

# All interfaces are good
} else {
    puts "\n\nIP Interface(s):"
    puts "----------------"
    puts $str_sh_ip_int
    puts "\n[expr $num_num_of_lines-2] interface(s) checked.  No Problem Found."
}

}




Excel, AXL, and Cisco CUCM

$
0
0

Introduction

 Cisco CUCM (CallManager)'s clumsy web GUI has earned its reputation.  :)  It might be OK for a small-to-medium business to perform daily operation on the infamous CCMAdmin GUI.  But it's a nightmare for large organization, especially service providers that need to perform MACD(Move, Add, Change, Delete) on hundreds or thousands of entries.

DevOps is the trend.  No exception on CUCM.  All CUCM configuration is stored in database.  You could view or change the database if you know a little bit about SQL query language.  You either do it from CUCM CLI (command line), or via web calls (SOAP/AXL).

Though database is the most powerful and flexible way to view and change CUCM configuration, it requires you understand the database structure.  You'll have to read the "Database Dictionary" on cisco.com to figure out which table is for what function.  Some of the functions require more than one table, which makes things more complicated.

AXL (Administrative XML Web Service) is another option to manipulate CUCM configuration in a relatively easy way.  Instead of having to know database tables, fields, keys, etc., you may just tell AXL that "I want to list all users with first name Michael".  Then AXL will do the leg work to query database and return desired results.

Obviously, we need to have AXL service running on CUCM.  We also need a user account that has AXL privilege.  Using admin account is the lazy way, but you should create a dedicate AXL account.  To verify AXL service is running and the account has proper privilege, you may put the following URL into a web browser, where "cucm.domain-name.com" is the FQDN of your CallManager server.

https://cucm.domain-name.com:8443/axl/

You should see the following displayed on the web page:

I'm not going to turn this article into an AXL tutorial.  For details, please refer to https://developer.cisco.com/docs/axl/

Excel and VBA

"Why VBA?  It's so old-school!  You should be using Python!"

I'm not a big fan of VBA.  However, Excel's dominance makes it the most common tool across industries.  Office clerks, field engineers, end users, almost everyone has Excel installed on their computers.  Almost everyone knows how to use Excel with no (or very little) training.  In network integration or migration projects, we're still seeing a lot of data being stored in Excel (or CSV) format.

Thus we need to use the data already there to program network gears including CUCM.  VBA is only built-in script language for MS office suite.  Which means, end users don't have to install any additional software to run the script.

Unfortunately, VBA is not as popular as Python or PHP from developer perspective, especially when it comes to network programming.  Thus very little information online to show you how to get things done (concerning network programming).  That doesn't mean it can't be done.  You'll just have to spend more time on research and test.

I did quite a lot of search online.  Couldn't find a complete example of how to make AXL calls to Cisco CUCM from Excel spreadsheet.  I'm sharing my script here so you don't have to reinvent the wheel.

AXL, SOAP, and SoapUI

I assume you know how to code with VBA scripting.  I assume you spent your time reading AXL, SOAP (basically XML).  And of course, I assume you know how to do administrative work on CUCM Admin GUI.  We're not going to cover those topics here.

You may think AXL as programming API.  Before using the API, you need to know what functions are available in the API.  And you also want an easy way to test the API (without involving language-specific coding).  I recommend you download a free version of SoapUI (https://www.soapui.org/downloads/latest-release.html).  Use SoapUI to open the AXLAPI.wsdl downloaded from CUCM (https://developer.cisco.com/docs/axl/#!download-the-axl-wsdl/download-the-axl-wsdl).  Then you may test the AXL calls and examine the SOAP/XML being sent and received.



In the screen above, the XML code on left hand side is the request, the XML code on right is the response from CUCM.  It is a example of querying a Calling Search Space (CSS) and get all partitions in that CSS.

Excel and VBA Script

Now we build an Excel spreadsheet like below:


There are a couple cells in the spreadsheet need to be filled out:
1) The FQDN of CUCM publisher (AXL server)
2) Username
3) Password
4) Calling Search Space (CSS) name

When click on the 'Execute' button, the script will reach out to CUCM and retrieve all partitions in that CSS.  Then it'll fill the partition names in cell B8, B9, B10, ... so on so forth.

You see another benefit of using Excel is that:
1) You have a place to store the data input
2) You have a place to store the data output

This is what it looks like on CCMAdmin:


This is what it looks like after clicking the 'Execute' button in spreadsheet:


This is the script:



By the way, in order to use the XML objects, you need to enable the reference to "Microsoft XML v6.0".






Network Virtualization - OVA or QCOW2?

$
0
0
20 years ago, when I was working on my CCIE R/S, I had to borrow a Cisco 2501 router from a friend's company during the weekend and return it before Monday.  Routers were too expensive for me (and they still are) to build a lab.

Virtualization changes everything.  IOU(IOS on Unix) was a well-known secret on Sun Sparc platform.  And then Cisco 7200/3600 on dynamics.  Then IOL(IOS on Linux) with various front ends including famous GNS3 and EVE-NG.  Then Cisco's official network simulator VIRL.

More and more manufacturers offer virtual appliances of their network products.  Virtual appliances are offered in various formats to support VMware, KVM, MS Hypervisor, etc.  Two of the popular formats are ova (for VMware) and qcow2 (for KVM).

It's pretty straight forward if you plan to run the virtual appliance on physical host.  For example, if you have VMware Workstation/ESXi you'll choose ova.  If you have KVM, you'll choose qcow2.  However, it's not so straight forward if you plan to use them with GNS3.

GNS3, along the time has evolved a lot.  GNS3 is a client/server application suite.  The best practice is to use GNS3 VM, which is a Linux virtual machine with GNS3 server preinstalled.  With GNS3 VM, you may have all kinds of software images (IOS, IOS-XR, NX-OS, Viptela, etc.) contained in a VM.  Makes it easier to package, distribute and share your labs.

If you already have VMware (either Workstation or ESXi), you probably want the GSN3 VM run on VMware.  How about those Cisco appliances?  Shall you download the OVA file so that you may run them on VMware as well?  Not exactly.  In order to take advantage of GNS3's user friendly GUI front end, You'll have to run Cisco appliances inside GNS3 VM.  You'll have to use qcow2 (KVM format).

In theory, you COULD use OVA format.  If you do so, you'll run Cisco appliances outside of the GNS3 VM.  You could still connect those appliances to your GNS3 virtual routers.  But it would be very cumbersome.

In short, if you plan to use GNS3, always download the qcow2 format.


Cisco SD-WAN with GNS3

$
0
0
To avoid turning this blog into a book, I assume the following:
  • You are already using GNS3 and know how to build basic routing/switching lab.
  • You understand the concept of GNS3 appliance.
  • You know how to download software from Cisco (sorry I cannot provide any Cisco software images or licenses)

This blog is based on the following software:
  • Windows 10
  • VMware Workstation Pro for Windows
  • GNS3 2.2.7 for Windows (with GNS3 VM)

Cisco software images (download from cisco.com):
  • vManage: viptela-vmanage-19.2.2-genericx86-64.qcow2
  • vSmart: viptela-smart-19.2.2-genericx86-64.qcow2
  • vEdge: viptela-edge-19.2.2-genericx86-64.qcow2
  • vBond (same image as vEdge)
  • CSRv with SD-WAN: csr1000v-universalk9.17.02.01r-serial.qcow2.  This file is optional.  It is a Cisco version of vEdge (a.k.a. cEdge).  Some customers prefer using Cisco routers as SD-WAN edge because they already have them purchased.  In your lab, you may use vEdge, cEdge or both.

GNS3 Appliance Templates (download from gns3.com):
  • vManage: viptela-vmanage-genericx86-64.gns3a
  • vSmart: viptela-smart-genericx86-64.gns3a
  • vEdge: viptela-edge-genericx86-64.gns3a
  • vBond: Make a copy of the vEdge file above, name it viptela-bond-genericx86-64.gns3a, change line 2, 4 and 8 to indicate a vBond template.  It's just cosmetic.  Example as below:
{
    "name": "vBond",
    "category": "router",
    "description": "vBond",
    "vendor_name": "Cisco",
    "vendor_url": "https://www.cisco.com",
    "documentation_url": "http://www.cisco.com/",
    "product_name": "VIPtela Bond",
  • empty30G.qcow2

License file (generate from cisco.com.  See https://codingpackets.com/blog/cisco-sdwan-self-hosted-lab-part-1/ for details):
  • serialFile.viptela

SD-WAN icons (png format, download from below):
  • vManage 
  • vSmart
  • vBond
  • vEdge

Steps to import SD-WAN appliances into GNS3

1. From GNS3 > File > Import Appliance.

2. Choose the GNS3 Appliance file (.gns3a) from your hard drive.  e.g viptela-smart-genericx86-64.gns3a

3. Install the appliance on GNS3 VM






4. QENU V3.1.0 is chosen by default


5. Cisco software image we use here is version 19.2.2, which is not on the list.  We may choose "Create a new version".







6. Type in 19.2.2.  Again, this is cosmetic only.  You may upload 19.2.2 image under the label 19.2.0 and it still works.  But it is a good idea always be accurate.

7. Highlight the missing file under the newly created 19.2.2 version, and click "import"

8. Locate the Cisco SD-WAN image file (.qcow2 file) on your hard drive.  It will start uploading to GNS3 VM once you choose the file.  For some appliances (such as vManage), you will need a secondary qcow2 file (empty30G.qcow2).  Repeat the same steps to import the second file.


9.  Once it finish uploading, you will see "ready to install" message.


10.  Click "Next" to continue install.  Click "Yes" to confirm.

11. Here is the completion screen.

12.  SD-WAN appliances are put into "Router" category in GNS3.  By default, router appliances are given the dark green hockey puck icon.  Optionally, you may change that to more distinguishable SD-WAN icons.  Right-click on the newly added template, then choose "Configure Template".


13. In "General Settings" tab, click "Browse" button on the "Symbol" line.


14. You may either choose from a set of pre-installed symbols(icons), or choose to upload a custom one.  Here we choose to upload a PNG file.  You should already downloaded the SD-WAN icons in previous steps.  Use "Browse" button to locate the PNG file on your hard drive.

15. Now you have the beautiful light blue SD-WAN icons for each appliance templates.


cEdge - Single Image Switch to Controller Mode

$
0
0
cEdge is a Cisco router acting as SD WAN Edge.  Cisco's road map is to replace vEdge products with cEdge.  cEdge can be on ISR, ASR and CSR, as well as their corresponding virtual variants (such as ISR 1000v, CSR 1000v, etc.)

When the above devices operating in "regular mode", it's called "Autonomous Mode", which is your good old IOS XE command system.  When they operate in "SD WAN mode", it's called "Controller Mode", which is Viptela-like command system.

In pre-17.2 versions, you'll have to load different software on the router to support different modes.  Since version 17.2.1r, one single image can support two different modes.  Please see https://community.cisco.com/t5/networking-blogs/ios-xe-17-2-1r-single-again-and-ready-to-mingle/bc-p/4091398

I put CSR 1000v 17.2.1r in my GNS3 lab.  It boots into autonomous mode by default.  I tried to switch it to controller mode with no luck.  I was referring to https://www.cisco.com/c/en/us/td/docs/routers/sdwan/configuration/sdwan-xe-gs-book/install-upgrade-17-2-later.html#d17982e2074a1635.  Also https://codingpackets.com/blog/cisco-sdwan-self-hosted-lab-part-2/

I did quite a lot research.  However, this version is too new to yield any helpful resource online.  I finally figured it out.  It's a documentation issue.

First, let's take a look at Cisco documentation below:


Per above documentation, the cfg file (in router bootflash:) will trigger the mode change.  That is NOT TRUE!  At least not in CRS 1000v 17.2.1r.  The cfg file won't be used UNTIL the router is switched to controller mode.

To switch from autonomous mode to controller mode, you use CLI command "controller-mode enable".  Router will warn you that all configuration will be lost.  After conformation, router will reboot into controller mode.

The first sign of controller mode is - you'll be prompted to enter username/password, even if you don't have it set up previously.  Default username/password is admin/admin, which aligns with Viptela defaults.


After login, you may use the following commands to double confirm it's in controller mode:


Revisit "Urgent Priority"

$
0
0

Cisco CUCM (CallManager) has "urgent priority" option for translation patterns and route patterns.  At the first glance, it is pretty straight forward.  It is usually used with emergency patterns like "911".  The purpose is to eliminate potential inter-digit timeout.  For instance, when user dialed 911, CUCM will route the call immediately, even if there are potential matches (like 911XXX).

But what happens if we use urgent priority on variable length pattern (like "!")?  Since wildcard ! means "one or more digits", shouldn't the system wait for more digit anyway?  Would interdigit timeout happen or not?

The short answer is "No".  If you have urgent priority on pattern "!", and you are dialing digit by digit, the system will start routing the call after the first digit is pressed.  Because that matches the definition "one or more digits".  That seems pretty useless.  Why would people do that?

It is not totally useless.  You may still dial multiple digits with the following options:

Option 1: Bloc-Dial

Keep the phone on hook (do not get a dial tone), enter all the digits you want, then hit the "Dial" button.  This is called bloc-dial.  You may pass all digits to ! pattern with urgent priority with bloc-dial.

Option 2: From previous hop

In large-scale dial plan design, we usually expose translation patterns(TPs) to phones, but not route patterns(RPs).  The intend is to use TPs to do all kinds of digit manipulation and class of control.  Then pass the manipulated digits to RPs.  In this case, TP has no problem passing all digits to RP (even if the RP has urgent priority).

Let say, you have a two-tier dial plan design (TP/RP).  You have emergency TPs with urgent priority.  When those TPs pass digits to RPs, it may or may not experience interdigit timeout depending on your RP setup.  Interdigit timeout is evaluated at each hop.  If you use RP ! to catch all digits passed by TPs, you might want to enable urgent priority on that ! pattern.  Or as an alternative, you may enable "Do Not Wait for Interdigit Timeout On Subsequent Hops" on the TP.


Another interesting topic is the interaction of urgent priority and "longest match".  Take a look at the following patterns:

  • 7XXX
  • 700XX

When you dial 7, 0, 0, 1, 2 digit-by-digit, which one will be matched?  At the first glance, 700XX seems to be the best candidate because it matches more digits.  Enable urgent priority on 7XXX seems harmless for this dialing string, right?  Actually not.  When urgent priority is enabled on 7XXX, you won't be able to enter the fifth digit.  Once you entered the 4th digit, system immediately routes the calls.  The only way to work around that is to use bloc-dialing.

In summary:

  1. Interdigit timeout applies to digit-by-digit dialing.  It does not apply to bloc-dialing.
  2. Digits passed by previous hop does NOT equal to bloc-dialing.  For instance, urgent priority on TP level does not necessarily immune to interdigit timeout at RP level.
  3. Be careful of overlapped patterns.  Using urgent priority might have side effects.


CUCM Security Tips

$
0
0

CUCM Security Tips



  • Secure Conference Bridge has special requirement.  Though configuration for secure conference bridge (CFB) is pretty much the same as secure transcoder(TRA) and secure media termination point(MTP), secure CFB requires the register name to match the hostname portion of the subject name (in the certificate/trust-point).  For instance,
    • You router's FQDN is R1.MyCompany.com.
    • Most likely you would enroll a certificate for the router with a subject name "R1.MyCompany.com".
    • On the router, you created a trust point R1-Cert to hold the above certificate.
    • You would think you could use R1-Cert for secure SIP-trunk, secure media resource, etc.  Until it comes to the point that you try to register a secure conference bridge.
      • On CUCM, you saw the conference bridge status is "rejected", yet secure transcoder and MTP are registered.  So it's unlikely certificate problem.
      • On CUCM, you named them R1-CFB, R1-TRA, and R1-MTP

The problem is with the name "R1-CFB".  It has to match with the hostname portion in the certificate (which is "R1" in this case).  OK, no big deal, I'll just change the name from "R1-CFB" to "R1".  Well... you cannot do that.  Because IOS won't accept a conference bridge name shorter than 6 characters or longer than 15 characters.  So you either change the router name (make it between 6 and 15 characters), re-enroll the certificate; or keep the router name/keep the router certificate, and enroll another certificate for conference bridge (yes, for conference bridge only).  For instance,

    • Your router certificate is R1.MyCompany.com
    • Your conference bridge certificate is R1-CFB.MyCompany.com

No, you don't need DNS entry created for those names.  You may use IP addresses and TLS handshake will still succeed.

Use IOS router as CA (Certificate Authority) Server

If you have a router handy (especially with GNS3), IOS CA is probably the most convenient way to sign certificates.  Though you may set up IOS CA for online enrollment, I strong recommend you practice terminal method.  In most cases, terminal is the quickest (and probably the only) way to get your job done.

To set up CA server on IOS, you just need a couple commands:

conf t
crypto key generate rsa general-keys exportable label myCA modulus 2048
crypto key export rsa myCA pem url nvram: des myCAkey
ip http server
crypto pki server myDB
 database level minimum
 database url nvram:
 issuer-name cn=myCA, l=Dallas, c=US
 lifetime certificate 7305
 grant auto

 no shut

From a router (say, R1) you want to enroll certificate, do the following:

crypto key generate rsa label MyRSAkey exportable modulus 2048
crypto pki trustpoint R1-Cert
 serial-number none
 fqdn none
 ip-address none
 subject-name cn=R1.mycompany.com
 revocation-check none
 rsakeypair MyRSAkey
 enrollment terminal
crypto pki enroll R1-Cert

R1 will generate CSR and print it on the terminal.  You're going to copy/paste the CSR to the CA server we created above.  An example output is like below:

R1(config)#crypto pki enroll R1-cert
% Start certificate enrollment .. 
% The subject name in the certificate will include: cn=R1.mycompany.com
% The fully-qualified domain name will not be included in the certificate
Display Certificate Request to terminal? [yes/no]: yes
Certificate Request follows: 
MIIBfzCB6QIBADAfMR0wGwYDVQQDExRteUNVQkUuZXhhbXBsZS5sb2NhbDCBnzAN
BgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAqvvg3/mSs86PF/4EFGmLt+hbmj0YmBM8
JOfHLJ0lC1uEEYMxe/8+4D+J5vTrR3TgyMy2xkW2eUmZHWnFbWiGDhVE9QrnBGHV
YcS6eFL1WUMlk1y7PWICX8sBWmig6t3D28pMdvlLQTaLAyn9hiJvm6VPEhK3Pao7
+kwTpMPP5AkCAwEAAaAhMB8GCSqGSIb3DQEJDjESMBAwDgYDVR0PAQH/BAQDAgWg
MA0GCSqGSIb3DQEBBQUAA4GBAJ7VyBjaiu2t8IbTeKKBGzPgVqaja4NBTDkl5bHX
1OUyBJ0Ih02NM9Cq07HtdwaiWXiAcPdqPYOVfLHNM50FuC1e1aag0QgGWW126Na5
buyLyg3Daf67wymMhAyrKhFTkhGlIO1gJ739c9yPpVf2TUVtjMWNl+Fz6Je52qSF
97iD 
---End - This line not part of the certificate request--- 
Redisplay enrollment request? [yes/no]: no
R1(config)#

On CA server, use global command "crypto pki server myDB request PKCS10 terminal" to sign a CSR.  An example output is like below:

CA#crypto pki server myDB request PKCS10 terminal
PKCS10 request in base64 or pem 
% Enter Base64 encoded or PEM formatted PKCS10 enrollment request.
% End with a blank line or "quit" on a line by itself.
MIIBfzCB6QIBADAfMR0wGwYDVQQDExRteUNVQkUuZXhhbXBsZS5sb2NhbDCBnzAN
BgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAqvvg3/mSs86PF/4EFGmLt+hbmj0YmBM8
JOfHLJ0lC1uEEYMxe/8+4D+J5vTrR3TgyMy2xkW2eUmZHWnFbWiGDhVE9QrnBGHV
YcS6eFL1WUMlk1y7PWICX8sBWmig6t3D28pMdvlLQTaLAyn9hiJvm6VPEhK3Pao7
+kwTpMPP5AkCAwEAAaAhMB8GCSqGSIb3DQEJDjESMBAwDgYDVR0PAQH/BAQDAgWg
MA0GCSqGSIb3DQEBBQUAA4GBAJ7VyBjaiu2t8IbTeKKBGzPgVqaja4NBTDkl5bHX
1OUyBJ0Ih02NM9Cq07HtdwaiWXiAcPdqPYOVfLHNM50FuC1e1aag0QgGWW126Na5
buyLyg3Daf67wymMhAyrKhFTkhGlIO1gJ739c9yPpVf2TUVtjMWNl+Fz6Je52qSF
97iD 
% Granted certificate:
MIICFDCCAX2gAwIBAgIBBDANBgkqhkiG9w0BAQQFADAtMQswCQYDVQQGEwJVUzEP
MA0GA1UEBxMGRGFsbGFzMQ0wCwYDVQQDEwRteUNBMB4XDTIwMTAxNjEzNTI0M1oX
DTIzMTAxNjAwMjAzMFowHzEdMBsGA1UEAxMUbXlDVUJFLmV4YW1wbGUubG9jYWww
gZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJAoGBAKr74N/5krPOjxf+BBRpi7foW5o9
GJgTPCTnxyydJQtbhBGDMXv/PuA/ieb060d04MjMtsZFtnlJmR1pxW1ohg4VRPUK
5wRh1WHEunhS9VlDJZNcuz1iAl/LAVpooOrdw9vKTHb5S0E2iwMp/YYib5ulTxIS
tz2qO/pME6TDz+QJAgMBAAGjUjBQMA4GA1UdDwEB/wQEAwIFoDAfBgNVHSMEGDAW
gBTp6GtJM5PvbI3sgyhm0JTXwyp87TAdBgNVHQ4EFgQU8wuU1JkqJ5iCnGNsFDPW
eo/WSE0wDQYJKoZIhvcNAQEEBQADgYEAOup/WXtvcJ7piHuWP8kA9IyYaGMrAsMr
dDlIsQ2BqexzrQiWKRNVVzUl/33FFD6eIi+UN7ZnQG/CspL2xTI2FRQ5KWBDJC0r
Znga3d2ITEWHwpn7yhscnI/B9MFABZgBAfXVFjUkZmV6Mlmjz/E+GDX+G1adnTcA
HbA6IdsOoFE= 
CA#

The yellow portion above is our input.  The green portion the the signed certificate (for R1).  We're going to copy/paste it to R1.  This is called "import" a signed certificate into R1.  But before importing a signed certificate, we need to import the signer (CA) certificate first.  On CA, use config command "crypto pki export myDB pem terminal" to export CA certificate.  An example is as below:

CA(config)#crypto pki export myDB pem terminal
% The specified trustpoint is not enrolled (myDB).
% Only export the CA certificate in PEM format.
% CA certificate:
-----BEGIN CERTIFICATE-----
MIICMzCCAZygAwIBAgIBATANBgkqhkiG9w0BAQQFADAtMQswCQYDVQQGEwJVUzEP
MA0GA1UEBxMGRGFsbGFzMQ0wCwYDVQQDEwRteUNBMB4XDTIwMTAxNjAwMjAzMFoX
DTIzMTAxNjAwMjAzMFowLTELMAkGA1UEBhMCVVMxDzANBgNVBAcTBkRhbGxhczEN
MAsGA1UEAxMEbXlDQTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkCgYEAgn+6fx3p
Km9VI/kLO32o5SPM184fbWBGm5OWq47B1PP8FfiyrboftDjClAl+AYXrV3mp79rt
jifQujen4vukCet/UNC+1U5txG8Y27wf+Zz4LQKlQWfXnXDUyQZ6DBbHCQrYs5Wm
9QqpDmq6JSfKaymAfY3aYBqp0SJjlR8+f/kCAwEAAaNjMGEwDwYDVR0TAQH/BAUw
AwEB/zAOBgNVHQ8BAf8EBAMCAYYwHwYDVR0jBBgwFoAU6ehrSTOT72yN7IMoZtCU
18MqfO0wHQYDVR0OBBYEFOnoa0kzk+9sjeyDKGbQlNfDKnztMA0GCSqGSIb3DQEB
BAUAA4GBAEMKnOCtDz9UMP48liWPXBLzImwrr81ARfYnjUk5b9h+3r2xB7t4welr
y+bRNQGcBT7tnWI16KMFUmfijqD4NsGFsgxObGJT4eF3IZeLgiRCTmz8HEhP3chb
aTRkht8WRNchU+YO1Nds/V/j/vD1x/eaeL9Myqb8Zh4JgTHaB+/k
-----END CERTIFICATE----- 
CA(config)#

Now on router R1, do the following:

  1. Import CA certificate with config command "crypto pki authenticate R1-Cert".  R1-Cert is the trust point name, which is just a placeholder for a cert and its signer.  Paste the CA cert (the 2nd green block above).
  2. Import R1 cert with config command "crypto pki import R1-Cert cert".  Paste the signed R1 cert (the 1st green block above).

Now you have successfully installed certificate on R1.


Guest Shell on CSR1000v 17.3.2

$
0
0

Software used:

  • GNS3 2.2.17
  • VMware Workstation for Windows 16.1.0
  • Windows 10 x64 Version 20H2 (Build 19042.685)
  • Cisco CSR 1000v (csr1000v-universalk9.17.03.02-serial.qcow2)


In VMware Virtual Network Editor, a NAT network was created with subnet address 192.168.28.0/24. (Your subnet might be different.  But the NAT network should have been created when you install VMware).

When I created GNS3 VM, I told it to use NAT network.  As shown in the picture below, it got a DHCP IP 192.168.21.128.


In GNS3, create a CSR instance and connect to GNS3-VM cloud, so that the CSR can have Internet access (to download software).  If your GNS3-VM has only one NIC, it'd be eth0.  My GNS3-VM has two NICs and the eth1 is connected to the NAT network.  When connecting CSR's Gi1 to the GNS3-VM eth1, we actually put the CSR Gi1 into the NAT network.


When CSR boots up, it shall get a DHCP IP from NAT network.  It also gets the DNS IP from DHCP.

To verify Internet is working, try to ping www.google.com.

Guest Shell is like a service module in the router (like the RSM in Catalyst 5500 switch, or the CUE module in Cisco 2800 router).  A Virtual PortGroup (VPG) is needed to be the gateway between Guest Shell and the physical interfaces.  In Cisco's document, VPG and Guest Shell are configured with private IP and  NAT.  See diagram below.


In home lab, you may do it in a different way.  You may configure VPG with "ip unnumber Gi1".  VGP will use the IP of Gi1.  Then configure the Guest Shell interface in the same subnet as Gi1 (but a different IP).  The advantage is - one less subnet in the network.  No NAT is needed.  The disadvantage is - you need to allocate an IP in the same subnet as Gi1. (this shouldn't be a problem in home lab though).  This option is illustrated in the diagram below:

Enable IOX:

Configure VirtualPortGroup0:
interface VirtualPortGroup0
 ip unnumbered GigabitEthernet1

Check the VPG IP:

Configure Guest Shell parameters.  In the example below, 192.168.28.130 is the VPG IP.  192.168.28.127 is an arbitrary (available) IP in the same subnet.  8.8.8.8 is a DNS server.
app-hosting appid guestshell
 app-vnic gateway0 virtualportgroup 0 guest-interface 0
  guest-ipaddress 192.168.28.127 netmask 255.255.255.0
 app-default-gateway 192.168.28.130 guest-interface 0
 app-resource profile custom
  cpu 1500
  memory 512
 name-server0 8.8.8.8
end

VERY IMPORTANT: the router needs to know how to send the traffic to guest shell:
ip route 192.168.28.127 255.255.255.255 VirtualPortGroup 0

Enable Guest Shell:

Enter Guest Shell.  Optionally, sudo:
CSR1#guestshell
[guestshell@guestshell ~]$ 
[guestshell@guestshell ~]$ sudo su -
Last login: Fri Dec 25 20:45:49 UTC 2020 on pts/4
[root@guestshell ~]# 

DNS resolution within Guest Shell is independent of host platform itself. The name-server configured in "Guest Shell parameters" will automatically get injected into the /etc/resolv.conf file on the CSR1000v. For NX-OS you must explicitly configure the /etc/resolv.conf entry.

[root@guestshell ~]# cat /etc/resolv.conf
nameserver 8.8.8.8

Verify Guest Shell can ping Internet host by DNS name.

Check versions:
[root@guestshell ~]# cat /etc/*-release
CentOS Linux release 8.1.1911 (Core) 
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

CentOS Linux release 8.1.1911 (Core) 
CentOS Linux release 8.1.1911 (Core) 
[root@guestshell ~]#             
[root@guestshell ~]# hostnamectl
   Static hostname: guestshell
         Icon name: computer-container
           Chassis: container
        Machine ID: d1eabe2de31449ccbbc0bae3567b0b83
           Boot ID: 222a6b054eda4e3f8bb93705a9bb7a44
    Virtualization: lxc-libvirt
  Operating System: CentOS Linux 8 (Core)
       CPE OS Name: cpe:/o:centos:centos:8
            Kernel: Linux 4.19.106
      Architecture: x86-64
[root@guestshell ~]# 
[root@guestshell ~]# uname -a
Linux guestshell 4.19.106 #1 SMP Fri Oct 2 17:55:01 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@guestshell ~]# 
[root@guestshell ~]# uname -mrs
Linux 4.19.106 x86_64
[root@guestshell ~]# 
[root@guestshell ~]# cat /proc/version
Linux version 4.19.106 (oe-user@oe-host) (gcc version 8.2.0 (GCC)) #1 SMP Fri Oct 2 17:55:01 UTC 2020
[root@guestshell ~]# 

Python3:
[root@guestshell ~]# python3
Python 3.6.8 (default, Nov 21 2019, 19:31:34) 
[GCC 8.3.1 20190507 (Red Hat 8.3.1-4)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 


References:


Cisco Expressway (MRA) and AT&T Wireless Interoperability Issue

$
0
0

When deploying a Cisco Expressway MRA(Mobile Remote Access) solution, I ran into a weird interoperability issue with AT&T wireless.  The symptom was: MRA calls to AT&T Wireless numbers went straight to voicemail without ringing the cell phone at all.  The same MRA call doesn't seem to have problem with other carriers like Verizon, T-Mobile or even AT&T wired phones.


At the first glance, this seems to be a carrier issue and there is not much we can do unless the carrier tells us what's wrong.  The demarcation point is at the CUBE.  I don't have any visibility beyond the CUBE.

I decided to do some troubleshooting within my scope.  I noticed that non-MRA calls didn't seem to have this problem.


If it's only the MRA calls having the problem, it is unfair to point the finger to the carrier.  But on the other hand, this only happens with one carrier.  It must be an interoperability between MRA and that particular carrier.

Both MRA calls and non-MRA calls go through the same CUBE.  I looked at the INVITEs sent from CUBE to carrier.  They are very similar except that the MRA calls have "Max-Forwards: 12" in the SIP messages while non-MRA calls have "Max-Forwards: 69".

I'm not sure if that's the root cause of the problem but that is the only thing sticks out.  By looking at Cisco documentations, Expressway has default Max-Forwards of 15 and CUCM has default of 70.  These values are very close to 12 and 69 from the CUBE logs.

Max-Forwards tag was designed to prevent infinite loops in call routing, similar to the TTL in IP packets or hop-count in routing protocols.  The value will be decreased by 1 on each hop along the path.  If one of the hops has a different value on Max-Forwards, the lower value takes precedence.  The diagram below explains why the MRA calls have a value of 12 while the non-MRA calls have a value of 69.


Without seeing the AT&T Wireless logs, I cannot tell what happened within the cellular network.  But imagine there are 12 or more hops in the cellular network before the call reaches the wireless endpoints (cell phones).  What would happen?

When the Max-Forwards value decreased to 0 on the way, the call will be dropped.  If that happens, the call controller within the cellular network will think the cell phone is unreachable (like when the cell phone is powered off or out of signal).  The call controller will send a REFER (redirect) SIP message back to the originator.  The call will be redirected to the cell phone's voicemail.  This is exactly what happens when the cell phone is "unreachable".

  • If it take less hops for the CUBE to reach the voicemail server (less than 12 hops), the call will be established.  The caller will hear voicemail greetings.
  • If it takes 12 hops or more for the CUBE to reach the voicemail server, the caller will hear reorder tone (fast busy) or the carrier's error announcement.  Because the call will fail for the same reason (Max-Forwards decreased to 0).

In my case, it is the prior.  Again, all these are just my guess, but an educated guess.  Is there a way we can fix this problem without carrier involved?  Of course.

The solution is to change Expressway default value from 15 to 70.  It doesn't necessarily have to be 70.  It just needs to be a value large enough so that the SIP message can survive the number of hops before the Max-Forwards decreased to 0.  Since CUCM has a default value of 70 and it seems to work, I decided to set Expressway to 70 as well.  If you are one of those OCD (Obsessive-compulsive disorder) persons, you may set Expressway to 72.  Then both MRA and non-MRA calls will leave the CUBE with the same value of 69, making it "consistent" from carrier point of view.


After the change, MRA calls to AT&T wireless numbers work as expected.

Minimalist's SIP Lab

$
0
0

SIP has become dominant in UC(Unified Communications) now.  If you're interested in learning/practicing SIP, CUBE, SBC, etc., here are some ideas.

Software lab is always better than hardware lab, because of cost, power consumption, noise, portability, etc.   Unless you want to test transcoding (which requires hardware DPS, like PVDM), software is sufficient for most cases.

If you are reading this, you are probably already familiar with lab software like GNS3, EVE-NG, Virl(CML), etc.  I'll use GNS3 as an example.

There are different virtual images on GNS3.  Each one has its pros and cons.  I personally like IOL (a.k.a. IOU) because it's lightweight (as little as 256MB per instance) and fast boot time.  Most of the L3 IOL image comes with CME(CallManager Express) function.  You may use CME as a IP PBX and register SIP phones to it.  Codec is limited to G.711 only.

! Global configuration for VoIP, with SIP sub-section
voice service voip
 allow-connections sip to sip
 sip
  bind control source-interface eth0/0
  bind media source-interface eth0/0
  registrar server expires max 1200 min 300
!
voice class codec 1
 codec preference 1 g711ulaw
!
! Global config for SIP registration(CME)
voice register global
 mode  cme
 source-address 192.168.28.10 port 5060
 max-dn 20
 max-pool 10
 authenticate register
!
! Create DNs to be used in later config
voice register dn  1
 number 2001
!
! Each register pool is a phone.  MAC address doesn’t matter
voice register pool  1
 id mac 0000.0000.0001
 number 1 dn 1
 dtmf-relay rtp-nte sip-notify
 voice-class codec 1
 username user1 password pass1

Then you may download SIP softphone apps, such as X-Lite(now known as "Bria", MicroSIP, etc.  Register SIP softphone to CME should be straightforward.  Make calls between two or three SIP phones and use debug commands (such as "debug ccsip message") to view the SIP messages is a good starting point.

Most of softphone allows single instance only.  You may work around that by using Sandboxie or virtual machines (such as VMware, VirtualBox, etc.)


If you want to explore the features of CUBE virtually, you may use CSR100v virtual router, which can also be run on GNS3.  There is no license needed.

If you want to play with codecs (such as g.729) or media resources (such as transcoding), you'll need a hardware router with DSPs.  You may get a Cisco ISR4K from eBay for about $200. And get a PVDM4-32 for about $100.  I'd get the routers comes with license (UC/K9 or VSEC/K9), though Cisco didn't seem to enforce CUBE license (yet).



Minimalist's SIP Lab (Part 2)

$
0
0

With may vendors going virtualization, it is possible to run virtual labs where it requires physical equipment before(such as Cisco ISR routers).  CML, VIRL, GNS3, EVE-NG, etc. we all heard about them.  But how can we run a meaningful lab with minimal resource?  That way, we don't have to fire up the lousy UCS servers?

Take the following diagram as an example.  I was working with a group of developers on a voice product, which require a lot of custom tagging in the SIP messages.  My job is to configure the routers to copy/modify/manipulate the SIP message.  (Yes, a lot of SIP profiles and regex).

It'd be great if I can run the lab on my laptop.  Did I mention that I have 2-3 production VMs (Virtual Machines) running on my laptop already?  Adding GNS3 and some virtual routers will definitely have impact on CPU and memory usage.


I achieved the goal with IOL (I86BI_LINUX-ADVENTERPRISEK9-M, Version 15.7(3)M2), which requires 384MB memory per each router.  The IOL image boots much faster than the CSR1000V or the C8000V virtual routers (which requires 4GB memory each).


I created 3 router instances.  Two of them are acting as CME, with MicroSIP softphone registered.  One of them is acting as a CUBE.  You don't actually need the CUBE license, nor the "mode border-element" command.  To test SIP profiles and SIP header manipulation, you just configure regular voip dial-peers.

As mentioned in the previous blog, Sandboxie comes handy when you need to run multiple softphones on the same computer.  The footprint is much smaller than launching multiple VMs.

With this lab, I can make test calls from CME1 to CME2 via the CUBE.  I can do all kinds of manipulations.



SFTP Server

$
0
0

Network and UC engineers often need a SFTP server.  I've been using FreeFTPd for years due to it's small size and free.  However FreeFTPd's encryption algorithm was outdated and not supported by some version of Cisco UC appliances.  I have been looking for an alternative for quite some time with no luck.  Solarwinds' SFTP server is "Free" but with 4GB limit.  Other products are either too bukly or not free at all.

It turned out that Windows (Server or Desktop) has an "OpenSSH" option which works pretty well.


Go to Apps > Add optional features > Search for OpenSSH Server.  Once installed, there will be two services.  Start these two services.

By default, user has access to C:\, which is also the root directory of the SFTP server.  However, Cisco UC appliance cannot handle that (with the SFTP root as "/C:/").  To change the SFTP root, you may edit the %programdata%\ssh\sshd_config file.  Say, you want to make "C:\SFTP" as the SFTP root directory, add the following line to the file:

ChrootDirectory "C:\SFTP"

Restart the OpenSSH services to take effect.




NFP sponsorship - Microsoft vs. Google

$
0
0

TLDR: Microsoft is much better than Google on NFP(Not for Profit) sponsorship program.

I was helping my friend to get sponsorship for his NFP (Not for Profit) organization.  Many technology companies offer free plans and discounted (paid) plans for NFPs.  This article is not to compare their plans or features.  This article is to compare the processes of application and initial setup.

Most NFPs have limited budget and (human) resource.  They don't normally have strong IT knowledge or experience.  So, the sponsorship application process should be as simple as possible.  Otherwise, it would dissuade NFPs from applying.

Microsoft's process is very straight forward, in two steps:

  1. You submit the NFP sponsorship(grant) application on Microsoft website.  You upload supporting documents, such as business registration, tax ID, etc.  The verification will be handled by a 3rd-party company.
  2. Once the verification is completed successfully, you will be given access to Microsoft free and discounted plans.  It is up to you which plan to choose.

In my case, there was no human intervene at all.  Everything was processed automatically and smoothly.  It was a totally different experience with Google.  As a matter of fact, the experience is like day and night.

  • Go to https://www.google.com/nonprofits/ and click "Get Started".  However, to really get started, you need to log in with a Google account.  (in contrast, Microsoft doesn't require you to have a "Microsoft account").  No big deal, I just use my personal Google account.
  • Google asked me to search for your NFP organization.  This is hit and miss, depending on if the NFP had already been 'verified' with Google's 3rd-party partner (named "Percent").  Sounds like a chick-and-egg problem.  My first attempt (search by EIN#) didn't come up with anything.  I had to upload the supporting documents.  After that, I was asked to enter the following:
    • The organization website.  This is a mandatory field.  Cannot be left blank or skipped.  A NFP without a website will probably be out of luck here.  In contrast, Microsoft does not require a NFP website in application.
      • The organization's name and mission statement.  This is asked even if you found your NFP by search.  Why asking for the mission statement again if the NFP is already verified by Percent (Google's partner)?  The mission statement is part of Percent's verification process.
        • The contact person's name and email.  This is also confusing - what email should I use?  In order to start the application, I had to log in with my Google account first.  What email address I should put here?  In fact, this is the biggest caveat.  I'll explain later.
          • Submit.

        Within a few hours, I got a rejection email.  The wording is very vague - "Unfortunately, XXX(the NFP's name) isn't eligible because either the organization or your affiliation with it couldn't be verified based on the information provided."

        This is weird.  The NFP has been established as 501c3 organization for 9 years and filed tax every year.  It definitely meets Google's eligibility requirements.  I reattempted a few times.  Got the same email rejection every time.  But there is no further explanation why it was not eligible.  After researching, someone said the success rate would be higher if the NFP was registered/verified by Percent first.

        OK, I gave it a try.  I went straight to Percent website https://causes.poweredbypercent.com/, opened a ticket, uploaded all the supporting documents.  After a few days, I received email from Percent saying the NFP has been verified.  Please note, this "verification" is for Percent only.  It is not for Google.  As Percent partners with other companies to offer sponsorship program.

        I went back to Google website to try again.  This time, when I searched for EIN, the NFP came up right away, even with the custom logo I uploaded to the Percent website.  This proves that Google taps into Percent database to retrieve the data.  I thought this time should work.

        I was wrong.  I got the exact same rejection email.  At this point, I could rule out the "eligibility" part as the NFP had been verified by Percent in their own database.  So it must be the "affiliation" part.  It looks like the system could not verify if the applicant was affiliated with the NFP or not.  But how could it verify the affiliation with the information provided?  Does it verify by the contact person's email?  Luckily, this NFP already have a business email domain set up (like <business name>.org).  I tried again.  This time I entered the business email in the "contact person's email" field.

        Just to clarify:

        1. To start the application, you need a Google account.  In my case, I have no other options but to use a personal Google account.  Because the NFP does not have any business account with Google yet.
        2. In the application process, I have the option to enter the contact person's email.  It could be any email address, not necessarily Google email.  To be more legit, I entered the NFP's business email address.

        In step 2 above, whatever email address you entered, Google will send a verification request to that email.  You'll have to click the "Verify" button in the email to confirm that you received the email.

        However, I got the same rejection email again, even with the business email address in the contact filed (and confirmed).  This was the sixth time of failure.  I just couldn't figure out how the system would be able to verify the affiliation with the information it asked.  As an IT veteran, there are only 3 ways to prove you are whom you claimed to be:

        • You know something (e.g. password).
        • You own something (e.g. an access badge)
        • You have some unique things that cannot be easily removed from you (e.g. fingerprints)

        Google's verification process didn't ask any of the things above.  Well, business email address could be a potential one, but only if Google can verify that email domain is associated with the business.

        The only 'help' is to post a question on Google's community.  But as many users pointed out - No one from Google is actually monitoring the community, especially the NFP community.  99% of the questions were not answered.  I had no choice, but to open a ticket with Percent.  Ironically, this is NOT mentioned in the application process.  I figured it out myself.  After uploading all documents (voided check, drivers license, official documents) and waited for days.  Percent said it was verified and asked me to wait for 24 hours to reapply.  It is worth noting, this is the 2nd time I verified by Percent:

        • First time is to verify the NFP's eligibility (501c3)
        • Second time is to verify the applicant's 'affiliation' with the NFP

        I waited 24 hours, re-applied.  Guess what?  I got the same rejection email again!  I sent email to Percent.  They replied with prescript language saying I had to contact Google.  Creatively, I made a Zoom appointment with Percent to "introduce my NFP".  7:30AM in the morning, I had a video call with a lady in London, UK.  After introducing my NFP and show some interest in Percent's products/services, I casually mentioned "by the way, I was trying to get verified with the Google program...".  The lady was kind enough to look into it and said she would get the right team to help me.  She also admitted that the process could have been better.  When I asked how does the process verify the applicant's affiliation with the NFP because it is logically impossible (without human intervene).  She declined to disclose the information.

        After a few hours, I finally got an email below (in my personal Gmail):

        At this point, you would think the most painful part has passed.  All you need is to choose a subscription plan (Google Workspace).  Below are the free or discounted plans for NFPs.  Most of the small NFPs will choose the free plan.

        There are also some confusions on the free plan.  The picture above indicate the free plan has 100TB storage shared across all users.  However the picture below indicate it is 30GB per user for the free plan.

        Regardless of 30GB or 100TB, let's just get started by subscribing to the free plan (official name "Google Workspace for Nonprofits").  It turned into another (unpleasant) journey.

        First of all, you cannot subscribe the free plan directly.  You'll have to activate a paid plan with trial period first.  I followed the onscreen prompt, entered the NFP size (2-9 persons).  Google recommend the plan "Google Workspace Business Plus".  Again, free plan is not an option here.  OK, I had no choice but to choose the "Business Plus" plan.  During the process, it asked me to do the following:

        1. Purchase a DNS domain or verify(transfer) an existing domain the NFP owns.  This is not very friendly for the NFPs who do not own an domain.  In my case, the NFP has its business domain on Microsoft Azure and Office 365.  The NFP decided to purchase a new domain just to test drive the Google Workspace features.  For privacy, let's say the new domain is foobar.org.
        2. Create an admin account, with the domain (e.g. admin@foobar.org).

        Here comes another confusion.  When applying for the NFP sponsorship with Google, the Google account (my personal Gmail account) is designated as the admin account.  However, when activate the product (Google Workspace plan), I cannot use my personal account.  I'll have to use a business domain account (e.g. admin@foobar.org, instead of jsmith@gmail.com).  So, there are two admins:

        • The admin who manages the sponsorship, activate the products.
        • The admin who manages the Workspace features, such as creating users, manage plans and storage, etc.

        But in Google's online help, it doesn't (clearly) indicate when to use which admin.  It is a trial-and-err process.

        I activated the "Business Plus" plan (with regular price).  Then I had to go back to the sponsorship page to request for the NFP discount (either a free plan or a discounted paid plan).  Why so complicated?  Why can't I just choose a NFP discounted plan directly?

        I submitted the request.  But it didn't give me any option to choose a discounted plan.  It just said "We received your NFP discount plan request.  It is under review.  We will get back to you".  Seriously?  I spent almost a week, verified with Percent twice to get to this point.  You sent me an email to invite me activate Nonprofit products.  Now not only I had to activate a regular price product first, but you'd have to review again (for the 3rd time) to validate I'm eligible for NFP price?

        After a day or two, I received another email saying my eligibility was verified.  Now I can use NPF discounts. 

        However, depending on your situation, you have to take different routes:

        • If you're happy with your current plan (which was activated before with regular price) but you want a NFP discount, you'll have to "Contact Google Workspace support and let them know you are looking to enable the nonprofit discount for your existing Google Workspace edition. Support will walk you through next steps".  Seriously?  For a software giant like Google, this cannot be done with a few clicks like scenario #2 below?  Customers have to contact support that requires human intervene?
        • If you want to switch to another plan, you need to log in as Workspace admin and change the plan.  This is my case because I wanted the NFP free plan instead of the discounted "Business Plus" plan.
        However, when I logged into Workspace admin console, NFP free plan is not an option.  Only (discounted) paid plans are available.

        I had no choice, but to contact Workspace support.  Here are the interesting (yet frustrating) facts:

        • I had to switch from "Business Plus" plan to "Business Starter" plan first.  Why?  I don't know.  As a programmer for years, the only explanation I can think of is - instead of rewrite the code to improve customer experience, let's just ask customers to take the trouble and deal with it.
        • Support said he couldn't do it for me.  I had to do it myself.  OK, I switched from Business Plus plan to Business Starter plan (with annual pay).
        • Then support said it couldn't be annual pay, it had to be flexible pay (monthly pay).  Hmm... why didn't you tell me earlier?  Again, I don't understand why it matters since my ultimate goal is NFP free plan.
        • Then in the verification process, support said my Workspace admin is not on the list of the sponsorship admin.  He asked me to add the Workspace admin to the sponsorship admin list.  I don't know why but I did that to satisfy him.  But this should not be the best practice.  The Workforce admin is a technical role that handles the functions and features of Google Workspace.  A technical role should not be given the organizational admin rights to change sponsorship (more of business relationship).
        • After all these, the support said the free NFP plan was handled by a different team.  He would have to transfer the request to that team to handle.  They'd get back to me when necessary.
        What else I can say?  Don't get me wrong.  Google is a great company and I'm very satisfied with their personal products, such as Gmail, Google Voice, Google Photos, etc.  But this NFP sponsorship program, even though exists for year, was poorly designed.  There is a lot of work to do, both internally and externally to be on par with other competitors (like Microsoft).  As a matter of fact, some NFP customer with 800+ users was talking about shifting to Microsoft 365 because of this.

        Just to give you an idea, I'm a IT professional with many technical certifications: Cisco CCIE, Microsoft MCSE, Oracle DBA, VMware VCP, etc.  My specialty is to set up, test, demonstrate, integrate, troubleshoot complex systems, like global data/voice networks, large scale applications, database, middleware, etc.  If something seems 'complicated' to me, it is not just complicated.  It is VERY complicated and doesn't make sense at all.


        Build a $30 Wireless Lab

        $
        0
        0
        One of the recent project has quite a lot wireless LAN stuff.  So I feel the urge to build a home lab.

        To build a wireless LAN lab, you need at least two things - a WLC (Wireless LAN Controller) and some compatible APs (Access Points).

        WLC was easy since you may download the virtual WLC (vWLC) software from cisco.com and throw it on VMware.

        It's not that easy when it comes to AP.  There are so many different models from Cisco.  I want the one that I can test most (if not all) the features with, while not costing me a fortune.  After some research (both on cisco.com and eBay.com), I decided 1242AG is the one.  This is a not-so-old AP that has 802.11a/b/g frequency and support many enterprise WLAN features (such as FlexConnect).  Most importantly, it's pretty affordable.  I got two for $30 (free shipping) from eBay.  I ordered two in case I need to test the "roaming" feature.

        It looks like this:



        Two things to be aware of:
        1) Make sure to order one with antennas.  Otherwise it'll cost you some extra bucks.
        2) They are mostly POE.  So you'll need a POE switch or power adapter.  You may get a cheap POE switch for less than $20.  But those switch won't support VLAN trunking, just FYI.

        Luckily I still have my 3750G POE switch sitting around (from my CCIE voice lab).  Now I have to design the network.

        In case you don't know, in real-life enterprise WLAN, they usually use DHCP option 43 to deliver the WLC IP address to APs.  I'd like to do the same in my lab.

        But my Linksys router doesn't have the capability to configure DHCP options.  Thus I need to set up a another DHCP server.  How may I set up a secondary DHCP server while not interfering with the primary one?  The answer is to put them into different VLAN/subnets.

        Here's my network design:


        My Linksys home router connects to 3750 switch VLAN 1.  The two APs connect to 3750 switch VLAN 3.

        3750 configuration:
        ip dhcp excluded-address 192.168.3.1 192.168.3.10
        !
        ip dhcp pool Wireless-Lab
           network 192.168.3.0 255.255.255.0
           default-router 192.168.3.1
           option 43 hex f104.c0a8.0216
        !
        interface Vlan1
         ip address 192.168.2.1 255.255.255.0
        !
        interface Vlan3
         ip address 192.168.3.1 255.255.255.0
        !
        ip route 0.0.0.0 0.0.0.0 192.168.2.100
        !
        interface GigabitEthernet1/0/1
         description Linksys Router
        !
        interface GigabitEthernet1/0/2
         description AP-1
         switchport access vlan 3
        !
        interface GigabitEthernet1/0/3
         description AP-2
         switchport access vlan 3
        Linksys configuraiton:

        Now you should be able to ping from home PC (VLAN1) to VLAN 3 and vice versa.

        On the vWLC virtual machine, I set the NIC to bridge network so I can configure a static IP in my home network segment (I used 192.168.2.22).

        Now you should be able to open a web page to the vWLC management portal.  Also, you should be able to ping from the vWLC (192.168.2.22) to VLAN3 (192.168.3.1) and vice versa.

        In theory, when I plug the APs to the switch, they should:
        1) Power up
        2) Get their IP address and the vWLC's IP address (via option 43 from DHCP)
        3) Join the WLC

        Well, not surprisingly, they didn't work as desired.  (if they did, there will be not much value for CCIEs)

        As a WLAN newbie, I went for documents, turned on debug, capture error messages, post questions on Cisco support forum.  After spent quite some time on troubleshooting, I was advised to upgrade the IOS (does that sound familiar?)

        There are many different software, tools and procedures regarding AP upgrade:
        • Autonomous vs. Lightwight vs. Recovery
        • TFTP vs. Upgrade Tool
        • etc.
        After many trial and err, here are my conclusions:
        1) Upgrade to the latest IOS version before you troubleshoot
        2) All you need is a TFTP server.  Don't use "upgrade tool"

        High-level recovery(upgrade) process:
        1) When the AP boots into recovery mode, it'll set its own IP address to 10.0.0.1 and search for TFTP server in the range of 10.0.0.2 - 10.0.0.30.
        2) If it found one, it'll try to download the "default" image.  File name of the "default" image depends on the AP model.  For 1242AG, the default image file name is "c1240-k9w7-tar.default".
        3) If the above file is found on TFTP, AP will download and install it.  Then reboot with that image.

        Now you have a high-level view, let's talk about the details and catchas.

        1) How to put a AP into recovery mode
        Power off the AP.  Hold the "mode" button.  Plug in the power (POE or Power Adapter).  Now the status LED will be orange.  Keep holding the button for about 30 seconds.  You'll see the status LED turned purple.  That means the AP is in recovery mode.  You may release the button.

        2) What TFTP server to use
        You need a TFTP server that can customize the timeout threshold.  Cisco recommends 30 seconds timeout.  I set it to 60 just in case.

        3) What IP address to configure for the TFTP server
        You may use any IP in the range of 10.0.0.2 - 10.0.0.30.  I normally use 10.0.0.2.  If you got a "IP Conflict" message, just pick another one.

        4) What IOS image I should download
        There are three different IOS images you can download:
        Autonomous Image (e.g. c1240-k9w7-tar.124-25d.JA2.tar)
        Lightweight Image (e.g. c1240-k9w8-tar.124-25e.JAO3.tar)
        Recovery Image (e.g. c1240-rcvk9w8-tar.124-25e.JAO3.tar)

        You'll ultimate goal is to upgrade to the latest lightweight image (that's the image who can work with a WLC).  But you might need to flash the AP with other images first in some situations (e.g. when your AP has a very very old firmware).

        When AP joins a WLC, it'll compare its IOS version and the ones on the WLC.  If there's any discrepancy, it'll download and use the one from WLC.  This is similar to IP phones download firmware from CallManager during registration.

        Because of that, it's recommended to put the recovery image on AP in recovery mode.  The recovery image is a small footprint image that boot up the AP, provide network function so the AP can download the latest IOS from WLC.

        5) How do I make the AP take the image I specified?

        Remember that AP will only take a "default" image with specific file name in recovery mode.  If you want AP to take the image, you'll need to rename it to the specific file name.  See this link for naming conventions: http://www.cisco.com/c/en/us/td/docs/wireless/access_point/conversion/lwapp/upgrade/guide/lwapnote.html#wp160918

        Be aware that Windows normally hide the file extensions.  You need to configure Windows Explorer to show file extension so you can name the file correct.

        For example, you want to rename c1240-rcvk9w8-tar.124-25e.JAO3.tar to c1240-k9w7-tar.default.  By default, Windows explorer will display "c1240-rcvk9w8-tar.124-25e.JAO3" as the file name.  If you rename it to "c1240-k9w7-tar.default" in Windows Explorer, the file name actually becomes "c1240-k9w7-tar.default.tar", which is NOT correct.

        If AP successfully joined a WLC, you'll see something like this:

        For troubleshooting, take a look at http://www.cisco.com/c/en/us/support/docs/wireless/4400-series-wireless-lan-controllers/99948-lap-notjoin-wlc-tshoot.html

        Enjoy your $30 wireless lab.  :)
        Viewing all 54 articles
        Browse latest View live