[sip-router] / src / modules / htable / README

HTable Module

Elena-Ramona Modroiu

   asipto.com
   <ramona@rosdev.ro>

Edited by

Elena-Ramona Modroiu

   <ramona@rosdev.ro>

Edited by

Alex Balashov

   <abalashov@evaristesys.com>

Edited by

Ovidiu Sas

   <osas@voipembedded.com>

   Copyright © 2008-2011 http://www.asipto.com
     __________________________________________________________________

   Table of Contents

   1. Admin Guide

        1. Overview
        2. Dependencies

              2.1. Kamailio Modules
              2.2. External Libraries or Applications
              2.3. Loading from database

        3. Parameters

              3.1. htable (str)
              3.2. db_url (str)
              3.3. key_name_column (str)
              3.4. key_type_column (str)
              3.5. value_type_column (str)
              3.6. key_value_column (str)
              3.7. expires_column (str)
              3.8. array_size_suffix (str)
              3.9. fetch_rows (integer)
              3.10. timer_interval (integer)
              3.11. db_expires (integer)
              3.12. enable_dmq (integer)
              3.13. timer_procs (integer)
              3.14. event_callback (str)

        4. Functions

              4.1. sht_print()
              4.2. sht_rm_name_re(htable=>regexp)
              4.3. sht_rm_value_re(htable=>regexp)
              4.4. sht_reset(htable)
              4.5. sht_lock(htable=>key)
              4.6. sht_unlock(htable=>key)
              4.7. sht_iterator_start(iname, hname)
              4.8. sht_iterator_end(iname)
              4.9. sht_iterator_next(iname)

        5. Exported pseudo-variables
        6. RPC Commands

              6.1. htable.get htable key
              6.2. htable.delete htable key
              6.3. htable.sets htable key value
              6.4. htable.seti htable key value
              6.5. htable.dump htable
              6.6. htable.reload htable
              6.7. htable.listTables
              6.8. htable.stats

        7. Event routes

              7.1. htable:mod-init
              7.2. htable:expired:<table>

   List of Examples

   1.1. Accessing $sht(htname=>key)
   1.2. Dictionary attack limitation
   1.3. Storing array values
   1.4. Set hash_size parameter
   1.5. Set db_url parameter
   1.6. Set key_name_column parameter
   1.7. Set key_type_column parameter
   1.8. Set value_type_column parameter
   1.9. Set key_value_column parameter
   1.10. Set expires_column parameter
   1.11. Set array_size_suffix parameter
   1.12. Set fetch_rows parameter
   1.13. Set timer_interval parameter
   1.14. Set db_expires parameter
   1.15. Set enable_dmq parameter
   1.16. Set timer_procs parameter
   1.17. Set event_callback parameter
   1.18. sht_print usage
   1.19. sht_rm_name_re usage
   1.20. sht_rm_value_re usage
   1.21. sht_reset usage
   1.22. sht_lock usage
   1.23. sht_unlock usage
   1.24. sht_iterator_start usage
   1.25. sht_iterator_end usage
   1.26. sht_iterator_next usage

Chapter 1. Admin Guide

   Table of Contents

   1. Overview
   2. Dependencies

        2.1. Kamailio Modules
        2.2. External Libraries or Applications
        2.3. Loading from database

   3. Parameters

        3.1. htable (str)
        3.2. db_url (str)
        3.3. key_name_column (str)
        3.4. key_type_column (str)
        3.5. value_type_column (str)
        3.6. key_value_column (str)
        3.7. expires_column (str)
        3.8. array_size_suffix (str)
        3.9. fetch_rows (integer)
        3.10. timer_interval (integer)
        3.11. db_expires (integer)
        3.12. enable_dmq (integer)
        3.13. timer_procs (integer)
        3.14. event_callback (str)

   4. Functions

        4.1. sht_print()
        4.2. sht_rm_name_re(htable=>regexp)
        4.3. sht_rm_value_re(htable=>regexp)
        4.4. sht_reset(htable)
        4.5. sht_lock(htable=>key)
        4.6. sht_unlock(htable=>key)
        4.7. sht_iterator_start(iname, hname)
        4.8. sht_iterator_end(iname)
        4.9. sht_iterator_next(iname)

   5. Exported pseudo-variables
   6. RPC Commands

        6.1. htable.get htable key
        6.2. htable.delete htable key
        6.3. htable.sets htable key value
        6.4. htable.seti htable key value
        6.5. htable.dump htable
        6.6. htable.reload htable
        6.7. htable.listTables
        6.8. htable.stats

   7. Event routes

        7.1. htable:mod-init
        7.2. htable:expired:<table>

1. Overview

   The module adds a hash table container to the configuration language.
   The hash table is stored in shared memory and the access to it can be
   done via pseudo-variables: $sht(htname=>name). The module supports
   definition of many hash tables and can load values at startup from a
   database table.

   A typical use case for the SIP server is to implement a cache system in
   configuration file - if a value is not found in hash table, load it
   from database and store it in hash table so next time the access to it
   is very fast. In the definition of the table you can define the default
   expiration time of cached items. The expiration time can be adjusted
   per item via assignment operation at runtime.

   Replication between multiple servers is performed automatically (if
   enabled) via the DMQ module.

   You can read more about hash tables at:
   http://en.wikipedia.org/wiki/Hash_table.

   The “name” can be a static string or can include pseudo- variables that
   will be replaced at runtime.

   Example 1.1. Accessing $sht(htname=>key)
...
modparam("htable", "htable", "a=>size=8;")
...
$sht(a=>test) = 1;
$sht(a=>$ci::srcip) = $si;
...

   The next example shows a way to protect against dictionary attacks. If
   someone fails to authenticate 3 times, it is forbidden for 15 minutes.
   Authenticatiion against database is expensive as it does a select on
   the “subscriber” table. By disabling the DB auth for 15 minutes,
   resources on the server are saved and time to discover the password is
   increased substantially. Additional alerting can be done by writing a
   message to syslog or sending email, etc.

   To implement the logic, two hash table variables are used: one counting
   the failed authentications per user and one for storing the time of the
   last authentication attempt. To ensure a unique name per user, the hash
   table uses a combination of authentication username and text
   “::auth_count” and “::last_auth”.

   Example 1.2. Dictionary attack limitation
...
modparam("htable", "htable", "a=>size=8;")
...
if(is_present_hf("Authorization"))
{
    if($sht(a=>$au::auth_count)==3)
    {
                $var(exp) = $Ts - 900;
        if($sht(a=>$au::last_auth) > $var(exp))
        {
            sl_send_reply("403", "Try later");
            exit;
        } else {
            $sht(a=>$au::auth_count) = 0;
        }
    }
    if(!www_authenticate("$td", "subscriber"))
    {
        switch ($retcode) {
            case -1:
                sl_send_reply("403", "Forbidden");
            exit;
            case -2:
                if($sht(a=>$au::auth_count) == $null)
                    $sht(a=>$au::auth_count) = 0;
                $sht(a=>$au::auth_count) = $sht(a=>$au::auth_count) + 1;
                if($sht(a=>$au::auth_count) == 3)
                    xlog("auth failed 3rd time - src ip: $si\n");
                $sht(a=>$au::last_auth) = $Ts;
            break;
        }
        www_challenge("$td"/*realm*/,"0"/*qop*/);
        exit;
    }
    $sht(a=>$au::auth_count) = 0;
} else {
    www_challenge("$td","0");
    exit;
}
...

   The module also provides a way to store multiple values for a single
   key. This is emulated by storing individual keys as 'key_name[n]',
   where n is incremented for each key. The total number of keys is stored
   in a dedicated key, by default: 'key_name::size'.

   The array is built when the table is loaded in memory and afterwards
   all the keys are treated as individual keys. If a particular entry in
   the array is deleted, it is the administrator's responsibility to
   update the size of the array and any other elements (if required).

   Example 1.3. Storing array values
# Example of dbtext with multiple keys
$ cat /usr/local/etc/kamailio/dbtext/htable
1:key:1:0:value3:0
2:key:1:0:value2:0
3:key:1:0:value1:0

# The array key will be loaded in memory in the following format:
$ kamcmd htable.dump htable
{
        entry: 35
        size: 1
        slot: {
                item: {
                        name: key[0]
                        value: value1
                }
        }
}
{
        entry: 50
        size: 1
        slot: {
                item: {
                        name: key::size
                        value: 3
                }
        }
}
{
        entry: 67
        size: 1
        slot: {
                item: {
                        name: key[1]
                        value: value2
                }
        }
}
{
        entry: 227
        size: 1
        slot: {
                item: {
                        name: key[2]
                        value: value3
                }
        }
}

# Now let's delete a particular entry in the array: key[0].
$ kamcmd htable.delete htable key[0]

# The array key will look like this after a key was deleted:
$ kamcmd htable.dump htable
{
        entry: 50
        size: 1
        slot: {
                item: {
                        name: key::size
                        value: 3
                }
        }
}
{
        entry: 67
        size: 1
        slot: {
                item: {
                        name: key[1]
                        value: value2
                }
        }
}
{
        entry: 227
        size: 1
        slot: {
                item: {
                        name: key[2]
                        value: value3
                }
        }
}

2. Dependencies

   2.1. Kamailio Modules
   2.2. External Libraries or Applications
   2.3. Loading from database

2.1. Kamailio Modules

   The following modules must be loaded before this module:
     * If DMQ replication is enabled, the DMQ module must be loaded
       first..

2.2. External Libraries or Applications

   The following libraries or applications must be installed before
   running Kamailio with this module loaded:
     * None.

2.3. Loading from database

   The module is able to load values in a hash table at startup upon
   providing a DB URL and table name.

   The structure of the table must contain:
     * key name - string containing the name of the key.
     * key type - the type of the key
          + 0 - simple key - the key is added as 'key_name'.
          + 1 - array key - the key is added as 'key_name[n]' - n is
            incremented for each key with this name to build an array in
            hash table. In addition, an additional key is built to hold
            the total number of key in the array, by default
            key_name::size (see array_size_suffix parameter).
     * value type - the type of the key value
          + 0 - value is string.
          + 1 - value is integer.
     * key value - string containing the value of the key.

3. Parameters

   3.1. htable (str)
   3.2. db_url (str)
   3.3. key_name_column (str)
   3.4. key_type_column (str)
   3.5. value_type_column (str)
   3.6. key_value_column (str)
   3.7. expires_column (str)
   3.8. array_size_suffix (str)
   3.9. fetch_rows (integer)
   3.10. timer_interval (integer)
   3.11. db_expires (integer)
   3.12. enable_dmq (integer)
   3.13. timer_procs (integer)
   3.14. event_callback (str)

3.1. htable (str)

   The definition of a hash table. The value of the parameter may have the
   following format:
     * "htname=>size=_number_;autoexpire=_number_;dbtable=_string_"

   The parameter can be set multiple times to get more hash tables in same
   configuration file.
     * htname - string specifying the name of the hash table. This string
       is used by $sht(...) to refer to the hash table.
     * size - number specifying the size of hash table. Larger value means
       less collisions. The number of entries (aka slots or buckets) in
       the table is 2^size. The possible range for this value is from 2 to
       31, smaller or larger values will be increased to 3 (8 slots) or
       decreased to 14 (16384 slots).
     * autoexpire -time in seconds to delete an item from a hash table if
       no update was done to it. If is missing or set to 0, the items
       won't expire.
     * dbtable - name of database to be loaded at startup in hash table.
       If empty or missing, no data will be loaded.
     * cols - the column names of the database table. They must be
       enclosed in quotes in order to form a valid SIP parameter value and
       be separated by comma. The first column corresponds to key_name.
       When specified, there must be at least two columns. If this
       attribute is not specified, then the global module parameters for
       column names are used. If more than one value columns are
       specified, the hash table will pack the column values in a comma
       separated string, which will be associated with the key (string
       transformation {s.select,...} can be used in configuration file to
       extract a specific column value). When cols attribute is present,
       writing back to database table is disabled.
     * dbmode - if set to 1, the content of hash table is written to
       database table when the SIP server is stopped (i.e., ensure
       persistency over restarts). Default value is 0 (no write back to db
       table).
     * initval - the integer value to be returned instead of $null when a
       requested key is not set.
     * updateexpire - if set to 1 (default), the time until expiration of
       an item is reset when that item is updated. Certain uses of htable
       may dictate that updates should not reset the expiration timeout,
       however, in which case this attribute can be set to 0.
     * dmqreplicate - if set to 1, any actions (set, update, delete etc.)
       performed upon entries in this table will be replicated to other
       nodes (htable peers). Please note, module parameter “enable_dmq”
       must also be set in order for this to apply (see below). Default is
       0 (no replication).

   Default value is NULL.

   Example 1.4. Set hash_size parameter
...
modparam("htable", "htable", "a=>size=4;autoexpire=7200;dbtable=htable_a;")
modparam("htable", "htable", "b=>size=5;")
modparam("htable", "htable", "c=>size=4;autoexpire=7200;initval=1;dmqreplicate=1
;")
...

3.2. db_url (str)

   The URL to connect to database for loading values in hash table at
   start up.

   Default value is NULL (do not connect).

   Example 1.5. Set db_url parameter
...
modparam("htable", "db_url", "mysql://kamailio:kamailiorw@localhost/kamailio")
...

3.3. key_name_column (str)

   The name of the column containing the hash table key name.

   Default value is 'key_name'.

   Example 1.6. Set key_name_column parameter
...
modparam("htable", "key_name_column", "kname")
...

3.4. key_type_column (str)

   The name of the column containing the hash table key type.

   Default value is 'key_type'.

   Example 1.7. Set key_type_column parameter
...
modparam("htable", "key_type_column", "ktype")
...

3.5. value_type_column (str)

   The name of the column containing the hash table value type.

   Default value is 'value_type'.

   Example 1.8. Set value_type_column parameter
...
modparam("htable", "value_type_column", "vtype")
...

3.6. key_value_column (str)

   The name of the column containing hash table key value.

   Default value is 'key_value'.

   Example 1.9. Set key_value_column parameter
...
modparam("htable", "key_value_column", "kvalue")
...

3.7. expires_column (str)

   The name of the column containing the expires value.

   Default value is 'expires'.

   Example 1.10. Set expires_column parameter
...
modparam("htable", "expires_column", "expiry")
...

3.8. array_size_suffix (str)

   The suffix to be added to store the number of items in an array (see
   key type).

   Default value is '::size'.

   Example 1.11. Set array_size_suffix parameter
...
modparam("htable", "array_size_suffix", "-count")
...

3.9. fetch_rows (integer)

   How many rows to fetch at once from database.

   Default value is 100.

   Example 1.12. Set fetch_rows parameter
...
modparam("htable", "fetch_rows", 1000)
...

3.10. timer_interval (integer)

   Interval in seconds to check for expired htable values.

   Default value is 20.

   Example 1.13. Set timer_interval parameter
...
modparam("htable", "timer_interval", 10)
...

3.11. db_expires (integer)

   If set to 1, the module will load/save the expires values of the items
   in hash table from/to database. It applies only to hash tables that
   have the auto-expires attribute defined.

   Default value is 0.

   Example 1.14. Set db_expires parameter
...
modparam("htable", "db_expires", 1)
...

3.12. enable_dmq (integer)

   If set to 1, will enable DMQ replication of actions performed upon
   entries in all tables having "dmqreplicate" parameter set. Any update
   action performed via pseudo-variables and RPC commands will be repeated
   on all other nodes. Therefore, it is important to ensure the table
   definition (size, autoexpire etc.) is identical across all instances.

   Important: If this parameter is enabled, the DMQ module must be loaded
   first - otherwise, startup will fail.

   Currently, values are not replicated on load from DB as it is expected
   that in these cases, all servers will load their values from the same
   DB.

   Default value is 0.

   Example 1.15. Set enable_dmq parameter
...
modparam("htable", "enable_dmq", 1)
...

3.13. timer_procs (integer)

   If set to 1 or greater, the module will create its own timer processes
   to scan for expired items in hash tables. If set to zero, it will use
   the core timer for this task. Set it to 1 if you store a lot of items
   with autoexpire property.

   Default value is 0.

   Example 1.16. Set timer_procs parameter
...
modparam("htable", "timer_procs", 4)
...

3.14. event_callback (str)

   The name of the function in the kemi configuration file (embedded
   scripting language such as Lua, Python, ...) to be executed instead of
   event_route[...] blocks.

   The function receives a string parameter with the name of the event,
   the values can be: 'htable:mod-init', 'htable:expired:htname' ('htname'
   being the name of hash table).

   Default value is 'empty' (no function is executed for events).

   Example 1.17. Set event_callback parameter
...
modparam("htable", "event_callback", "ksr_htable_event")
...
-- event callback function implemented in Lua
function ksr_htable_event(evname)
        KSR.info("===== htable module triggered event: " .. evname .. "\n");
        return 1;
end
...

4. Functions

   4.1. sht_print()
   4.2. sht_rm_name_re(htable=>regexp)
   4.3. sht_rm_value_re(htable=>regexp)
   4.4. sht_reset(htable)
   4.5. sht_lock(htable=>key)
   4.6. sht_unlock(htable=>key)
   4.7. sht_iterator_start(iname, hname)
   4.8. sht_iterator_end(iname)
   4.9. sht_iterator_next(iname)

4.1.  sht_print()

   Dump content of hash table to L_ERR log level. Intended for debug
   purposes.

   This function can be used from REQUEST_ROUTE, FAILURE_ROUTE,
   ONREPLY_ROUTE, BRANCH_ROUTE.

   Example 1.18. sht_print usage
...
sht_print();
...

4.2.  sht_rm_name_re(htable=>regexp)

   Delete all entries in the htable that match the name against regular
   expression.

   This function can be used from REQUEST_ROUTE, FAILURE_ROUTE,
   ONREPLY_ROUTE, BRANCH_ROUTE.

   Example 1.19. sht_rm_name_re usage
...
sht_rm_name_re("ha=>.*");
...

4.3.  sht_rm_value_re(htable=>regexp)

   Delete all entries in the htable that match the value against regular
   expression.

   This function can be used from REQUEST_ROUTE, FAILURE_ROUTE,
   ONREPLY_ROUTE, BRANCH_ROUTE.

   Example 1.20. sht_rm_value_re usage
...
sht_rm_value_re("ha=>.*");
...

4.4.  sht_reset(htable)

   Delete all entries in the htable. The name of the hash table can be a
   dynamic string with variables.

   This function can be used from ANY_ROUTE.

   Example 1.21. sht_reset usage
...
sht_reset("ha$var(x)");
...

4.5.  sht_lock(htable=>key)

   Lock the slot in htable corresponding to the key item. Note that the
   locking is re-entrant for the process, therefore the lock and unlock
   should be done by the same process.

   This function can be used from ANY_ROUTE.

   Example 1.22. sht_lock usage
...
sht_lock("ha=>test");
...

4.6.  sht_unlock(htable=>key)

   Unlock the slot in htable corespoding to the key item. Note that the
   locking is re-entrant for the process, therefore the lock and unlock
   should be done by the same process.

   This function can be used from ANY_ROUTE.

   Example 1.23. sht_unlock usage
...
sht_lock("ha=>test");
$sht(ha=>test) = $sht(ha=>test) + 10;
sht_unlock("ha=>test");
...

4.7.  sht_iterator_start(iname, hname)

   Start an iterator for hash table named by the value of parameter hname.
   The parameter iname is used to identify the iterator. There can be up
   to 4 iterators at the same time, with different name.

   Both parameters can be dynamic strings with variables.

   IMPORTANT: the slot of the hash table is left locked when retrieving in
   item. Therefore be sure you do not update the content of the hash table
   in between sht_iterator_start() and sht_iterator_end(), because it may
   end up in dead lock.

   This function can be used from ANY_ROUTE.

   Example 1.24. sht_iterator_start usage
...
sht_iterator_start("i1", "h1");
...

4.8.  sht_iterator_end(iname)

   Close the iterator identified by iname parameter and release the hash
   table slot aquired by the iterator. The iname value must be the same
   used for sht_iterator_start().

   The parameter can be dynamic string with variables.

   This function can be used from ANY_ROUTE.

   Example 1.25. sht_iterator_end usage
...
sht_iterator_end("i1");
...

4.9.  sht_iterator_next(iname)

   Move the iterator to the next item in hash table. It must be called
   also after sht_iterator_start() to get the first item in the hash
   table. Items are returned as they are found in the hash table slot,
   starting with the first slot.

   The return code is false when there is no (more) item in the hash
   table.

   The item name and value are accessible via variables: $shtitkey(iname)
   and $shtitval(iname).

   The parameter can be dynamic string with variables.

   This function can be used from ANY_ROUTE.

   Example 1.26. sht_iterator_next usage
...
    sht_iterator_start("i1", "h1");
    while(sht_iterator_next("i1")) {
        xlog("h1[$shtitkey(i1)] is: $shtitval(i1)\n");
    }
    sht_iterator_end("i1");
...

5. Exported pseudo-variables

     * $sht(htable=>key)
     * $shtex(htable=>key)
     * $shtcn(htable=>key)
     * $shtcv(htable=>key)
     * $shtinc(htable=>key)
     * $shtitkey(iname)
     * $shtitval(iname)
     * $shtrecord(attribute)

   Exported pseudo-variables are documented at
   https://www.kamailio.org/wiki/.

6. RPC Commands

   6.1. htable.get htable key
   6.2. htable.delete htable key
   6.3. htable.sets htable key value
   6.4. htable.seti htable key value
   6.5. htable.dump htable
   6.6. htable.reload htable
   6.7. htable.listTables
   6.8. htable.stats

6.1.  htable.get htable key

   Lists one value in a hash table

   Name: htable.get

   Parameters:
     * htable : Name of the hash table to dump
     * key : Key name of the hash table value to dump

   Example:
...
# Dump $sht(students=>alice)
kamcmd htable.get students alice

# Dump first entry in array key course $sht(students=>course[0])
kamcmd htable.get students course[0]
...

6.2.  htable.delete htable key

   Delete one value in a hash table

   Name: htable.delete

   Parameters:
     * htable : Name of the hash table to delete
     * key : Key name of the hash table value to delete

   Example:
...
# Delete $sht(students=>alice)
kamcmd htable.delete students alice

# Delete first entry in array key course $sht(students=>course[0])
kamcmd htable.delete students course[0]
...

6.3.  htable.sets htable key value

   Set an item in hash table to string value.

   Name: htable.sets

   Parameters:
     * htable : Name of the hash table
     * key : Key name in the hash table
     * Value : String value for the item

   Example:
...
# Set $sht(test=>x) as string
kamcmd htable.sets test x abc

# Set firsti entry in array key x $sht(test=>x[0]) as string
kamcmd htable.sets test x[0] abc
...

6.4.  htable.seti htable key value

   Set an item in hash table to integer value.

   Name: htable.seti

   Parameters:
     * htable : Name of the hash table
     * key : Key name in the hash table
     * Value : Integer value for the item

   Example:
...
# Set $sht(test=>x) as integer
kamcmd htable.seti test x 123

# Set firsti entry in array key x $sht(test=>x[0]) as integer
kamcmd htable.sets test x[0] 123
...

6.5.  htable.dump htable

   Lists all the values in a hash table

   Name: dhtable.dump

   Parameters:
     * htable : Name of the hash table to dump

   Example:
...
kamcmd htable.dump ipban
...

6.6.  htable.reload htable

   Reload hash table from database.

   Name: dhtable.reload

   Parameters:
     * htable : Name of the hash table to reload

   Example:
...
kamcmd htable.reload ipban
...

6.7.  htable.listTables

   Lists all defined tables

   Name: htable.listTables

   Parameters:
     * None

   Example:
...
kamcmd htable.listTables
...

6.8.  htable.stats

   Get statistics for hash tables - name, number of slots, number of
   items, max number of items per slot, min number of items per slot.

   Name: htable.stats

   Parameters:
     * None

   Example:
...
kamcmd htable.stats
...

7. Event routes

   7.1. htable:mod-init
   7.2. htable:expired:<table>

7.1.  htable:mod-init

   When defined, the module calls event_route[htable:mod-init] after all
   modules have been initialized. A typical use case is to initialise
   items in hash tables. The event route is executed only once, after core
   and module initialization, but before Kamailio forks any child
   processes.
...
event_route[htable:mod-init] {
    $sht(a=>x) = 1;
}
...

7.2.  htable:expired:<table>

   When defined, the module calls event_route[htable:expired:<table>] when
   an entry in the given table expires. In this event route, the key and
   value of the expired record are available with the $shtrecord(key) and
   $shtrecord(value) pseudo-variables.
...

event_route[htable:expired:mytable] {
    xlog("mytable record expired $shtrecord(key) => $shtrecord(value)\n");
}
...