CDRTool rating engine --------------------- CDRTool provides on-the-fly rating of CDRs from multiple data-sources like OpenSIPS, Asterisk or Cisco gateways based on easy to build rating plan, which can be provisioned in a web interface or imported from csv files. A Call Detail Record (CDR) is one record from the radius radacct table. The CDR contains all information related to a session, its duration, the calling and called party and media information. The rating engine calculates the price of the session. The calculation is done once and its results are saved in the CDR table for later use. It is possible to re-calculate the prices at a later time for example when having to change the tariffs. Based on exceptions, different rates may be applied per caller party identified by source IP, domain or subscriber. Rating is applied only after the call has ended. CDRTool considers that a call has ended when there is a stop time. In case of missing BYEs, CDRTool relies on the fact that MediaProxy will update the CDR with the proper stop time information based on the last moment the media stream passed through the media proxy. This functionality is not available when the end-points have negotiated using ICE another candidate than MediaProxy. The rates are linked with profiles corresponding with different time of the day, day of the week or holidays. For rating calls, which span multiple profiles, the right rate is selected and applied for the call duration within each profile. Each customer may be assigned its own dedicated rating plans destination id and names. Chained profiles are possible to enable exception based rating. Multiple customers may share a common rate list, while some destinations may be rated differently, only the differences must be provisioned. Multiple time zones are supported for multiple billing parties hosted on the same platform. Rating logic ------------ Different subscribers can have different rating plans. A rating plan is a unique combination of holidays, day of week, time of day, destination ids, and associated costs. The following steps are performed to rate a CDR: 1. Determination of the billing party 2. Determination of the destination id 3. Determination of the costs The steps are described in detail below. 1. Determination of the billing party ------------------------------------- To be able to calculate the Price for a call the rating engine must determine whose rating plan to use. The rating engine does this by performing a match against entries in the billing_customers table for the the BillingPartyId field of the the CDR (radacct.UserName for radius based datasources) in the following order: a. SIP account user@domain b. Username@SourceIP (Trusted peer account) c. SIP domain of the SIP account d. SourceIP of the session (Trusted peer IP) e. Default (when none of the above matches) You must have a maximum of one column (either subscriber, domain or gateway) set in every row of billing_customers table for this to work. The first match is considered to be the billing party for which the rating plan is determined. The rating plan is further derived from the profiles associated with the entry found in the billing_customers table. Note The billing_customers table field that matches the Source IP is called gateway. "gateway" or "trusted peer" terms are used interchangeably in this document. They both relate to the source IP address that generated the SIP session. If you want to use the multi-tenant features for OpenSIPS accounting you must create an integer reseller_id column in the trusted and domain tables. Traffic generated by IP addresses from the trusted peers table and local domains will be marked as belonging to their correspondent reseller_id. The reseller_id is then later used to lookup destinations and rates belonging to the reseller_id. 2. Determination of the destination id -------------------------------------- The rating engine identifies the 'destination id', which has associated rates depending on day of week and time of day. The 'destination id' is for example a country prefix like '31' for the Netherlands. This 'destination id' has prices associated with it so all calls to the Netherlands will have prices associated with prefix '31'. The 'destination id' is derived from the logical destination the SIP session has been routed to. In SIP headers and Radius records there are multiple places that contain information related to the destination. Some of them are generated by the SIP User-Agents (hence cannot be trusted and must not be used for accounting purposes) and others are generated by the SIP Proxy configured by the operator and are suitable for accounting purposes. The rating engine considers the destination to be the first non-empty CDR field in this order: a. CanonicalURI (the destination after all lookups inside the SIP Proxy) b. SipTranslatedRequestURI (the Request URI as presented by the SIP UA) c. CalledStationId (the content of the To header, used as a last resort) The CanonicalURI is the preferred because is reliable information generated by the operator based on the logic configured in the SIP Proxy and the subscriber cannot control it. OpenSIPS must be configured to send the CanonicalURI Radius attribute when creating the radius accounting START record. Instructions for how to set this up are available in INSTALL.txt file. The 'destination id' is then calculated based on the longest match of this destination field in the billing_destinations table. The longest match for the chosen destination field is performed by the E164_class, which by default uses E164_Europe that defines an European numbering plan. It assumes the destinations start with a zero for a national call and with double zero for an international call. See cdr_generic.php for the actual logic. If your dialing plan is different, you must use other provided class like E164_US or create a custom class and point to it in global.inc for each datasource as follows: 'E164_class'=>'MyE164Class', For example the pre-defined E164_US class from cdr_generic.php matches the American dialing plan. Destinations are uploaded into destination table. Each reseller_id can have its own list of destinations. If the reseller_id of trusted peers and domain tables have been set, then the rating engine will apply the rating information belonging to them. Each destination has also a set of properties that can be used to alter the calculation of the price. These properties are: - increment: the call duration is rounded to the next increment in seconds, this is only used for rating, the actual CDR duration is not modified - min_duration: set the duration to min_duration if the CDR duration is smaller than min_duration, this is only used for rating, the actual CDR duration is not modified - max_duration: limit the duration to max_duration if CDR duration is bigger than max_duration, this is only used for rating, the actual CDR duration is not modified - max_price: the price is set to max_price per call if the calculated price is bigger than the max_price The min_duration and increment values from billing_customers override the values found in the destinations table. 3. Determination of the costs ----------------------------- The following steps are taken to determine the cost for the calls based on the 'destination id' and the billing party determined at the previous steps. 1. Lookup the billing profiles in cdrtool.billing_customers table in the following order: subscriber,domain,gateway (based on $this->dayofweek): (as explained in "Determination of the billing party" above) - profile_name1 matches week days [1-5] - profile_name1_alt matches week days [1-5] if no rates for profile_name1 are found - profile_name2 matches week-ends [6-0] - profile_name2_alt matches week-ends [6-0] if no rates for profile_name2 are found - profile_name2 matches also holidays from billing_holidays table The week starts with 0 (Sunday) and ends with 6 (Saturday) This step determines which rates should be applied based on the day of the week when the call started. 2. Using the profile_name found, lookup the rate_name based on $this->hourofday in cdrtool.billing_profiles table If no rate_name is found for the given profiles a second set of profiles are used, profile_name1_alt and profile_name2_alt. - the day may be split in maximum 4 periods - the days starts with hour 0 and ends with hour 24 - rate_name1 defines the first interval after hour 0 - rate_name2 defines the first interval after rate_name1 - rate_name3 defines the first interval after rate_name2 - rate_name4 defines the first interval after rate_name3 When the hour matches an interval use the rate_nameX found to lookup the rate in billing_rates, if no record is found use the rate called 'default' This step determines which rate should be applied for the time of day when the call started. 3. Lookup in the cdrtool.billing_rates table the record having same name found at point 2 having billing_rates.destination = 'destination id' and billing_rates.application = application type found in the steps above. - return an array with all the rating values and the duration rated No rate will be returned if no 'destination id' is found. Make sure each possible destination has a 'corresponding id' and name in the destinations table. The 'audio' application may optionally be specified with a subtype in order to refine billing rates. If a subtype is specified, it is appended to the application name, separated by a dot (i.e. 'audio.incoming'). For example, ('1', 'audio') can have a durationRate of 280, where ('1', 'audio.incoming') can have a durationRate of 0. If the application isn't specified in the CDR, it defaults to 'audio'. This step determines the costs within the current time span associated with the time of day and destination id. If the call duration exceeds this time span (that is a new interval for which another rate applies is reached), step 4 is performed. 4. If the duration rated at point 3 is less than total call duration, apply point 3 again for the remaining call duration in the next profile. A maximum of 10 spans (different rates depending of time of day, day of the week) can be calculated using this mechanism. After 10 spans, the engine bails out to avoid loops caused by invalid tables provisioning. 5. Calculate the total call Price based on its duration and connection fees. In global.inc there are several variables that affect how the price is calculated. These settings are global per CDRTool installation but some can be overwritten with per customer values in the billing_customers table. $RatingEngine=array( "priceDenominator" => 10000, // Rates units (global setting) "priceDecimalDigits" => 4, // Decimal information (global setting) "minimumDurationCharged" => 0, // Rate a minimum of X seconds (per customer) "minimumDuration" => 0, // Minimum duration to rate, if call duration is shorter the price is zero (per customer) "durationPeriodRated" => 60 // Rate is per 60 seconds (global setting) "trafficSizeRated" => 1024, // Default we rate per 1 MB (global setting) "reportMissingRates" => 0 // Send emails to administrator in case of missing rates ); Pricing formula --------------- if min_duration then minimumDurationCharged = min_duration else if minimumDurationCharged set in global inc use minimumDurationCharged from global.inc else minimumDurationCharged = call duration if increment then durationForRating = round to the next increment else durationForRating = call duration if durationForRating >= minimumDurationCharged then Price = connectCost/priceDenominator+ durationRate*durationForRating/durationPeriodRated/priceDenominator else Price = 0 ENUM discounts -------------- The rating engine can apply a discount associated with the ENUM top level domain that returned the final destination. Price = Price - Price * ENUM discount / 100 To apply ENUM based discounts, the ENUM TLD must be saved with each CDR and the TLDs with their corespondent discounts must be provisioned in the Rating tables section. See ENUM TLD discounts section for more information. Purchasing price ---------------- A second price called 'Price in' is calculated using the same formula but based on connectCostIn and durationRateIn values. It can be used to match the purchasing price and calculate the margin between purchasing and selling prices. The information about both prices is stored in the RateInfo field of the CDR. The values for connectCostIn and durationRateIn must be provisioned in the billing_rates and billing_rates_history tables using the web interface or by importing csv files. 6. Save the calculated Price, billing party and 'destination id' for each call in the CDR table. Having the price stored in the database, it is possible to build statistics to display consolidated revenues per destination id, gateway, domain or subscriber. Testing the rating engine ------------------------- You can test the rating engine by telneting to the IP and port configured in global.inc. Type 'help' once connected to see the available commands. Use 'ShowPrice' command to simulate the rating of one session, for example: adigeo@w1:/var/www/CDRTool/doc$ telnet ws1 9024 Trying 10.0.0.1 ... Connected to 10.0.0.1. Escape character is '^]'. ShowPrice From=sip:123@example.com To=sip:0031650222333@example.com Gateway=10.0.0.1 Duration=59 0.2023 Duration: 59 s App: audio Destination: 31650 Customer: domain=example.com Connect: 0.0450 StartTime: 2009-01-03 14:29:10 -- Span: 1 Duration: 59 s ProfileId: 442 / weekend RateId: 442 / 0-24h Rate: 0.1600 / 60 s Price: 0.1573 Below is a description of the fields that must be separated by one or more spaces: * From - must contain the caller party * Gateway - must contain the source IP of the session * To - must contain the full CanonicalURI destination, its format must be synced with the E164 class logic used to determine the 'destination id' * Duration - the duration of the session in seconds Check the syslog for any errors, most of the configuration errors like missing rates are logged to the syslog. Importing and exporting of rating files --------------------------------------- It is important to set the impersonate field correctly for the login accounts. To access in the CDRTool Rating web page all the rates table for all resellers in the system you must set the impersonate field of the login account to 0.0 There are different data files needed for rating. The data files are imported into their corresponding MySQL tables. The files must be uploaded to /var/spool/cdrtool directory. To load the files into the database run the following command: /var/www/CDRTool/scripts/importRatingTables.php Data partitioning for multiple resellers The reseller_id column present in all rating tables is used to filter access based on Login account impersonate field. You may create numerical sub-directories under /var/spool/cdrtool directory. When importing a file from such numerical directory, the reseller_id field in all imported records is set to the directory name. The import script knows to import the files only once so you may dump several files over time with the same name and safely run the import script from cron. The import script detects whether each file has already been imported by building a unique key out of the filename and the hash of the file content. So you may use the same filenames as long as the content differs and viceversa. If the import file has changed any records, the rating engine is automatically instructed to reload the changes. Sample csv files are found in the setup directory. The CSV field order is described in setup/*.csv sample files. The first element on each line specifies the operation will be performed with the current record. The operation can be 2 (update/insert), 1 (insert) or 3 (delete). The updates are performed based on a unique key present in each table: billing_customers - cust_idx (reseller,gateway,domain,subscriber) destinations - cust_dest_idx (reseller,gateway,domain,subscriber,dest_id) billing_profiles - profile_idx (reseller,name) billing_rates - rate_idx (reseller,name,destination,application) billing_rates_history - rate_idx (reseller,name,destination,application,startDate,endDate) The content of the rating tables can be exported in the Rating tables page. The import script detects the type of file to import based on its filename. The filename must comply with the following naming convention: 1. Must start with the name of the table without the billing_ prefix 2. May optionally contain extra characters after the name 3. Must end with .csv extension Examples: - rates.csv or rates20061201.cvs will be loaded into the rates table - profiles.csv or profiles20061201.cvs will be loaded into the profiles table - destinations200601.csv will be loaded in the destinations table - ratesHistory200801.csv will be loaded in the rates_history table Do not use 'billing_' prefix in front of the file name. It is advisable to name the files in a consistent manner like tableYYYYMMDD.csv The results of the import operation is logged in the database and can be viewed in the Log section of the web interface and the syslog. MySQL schema ------------ To see the rating tables and their structures connect to the cdrtool database using mysql client. Run 'show tables' and 'describe table_name': billing_customers +-------------------+ | Field | +-------------------+ | id | | reseller_id | | gateway | | domain | | subscriber | | profile_name1 | | profile_name1_alt | | profile_name2 | | profile_name2_alt | | timezone | | increment | | min_duration | +-------------------+ billing_profiles +------------+ | Field | +------------+ | id | | reseller_id| | name | | rate_name1 | | hour1 | | rate_name2 | | hour2 | | rate_name3 | | hour3 | | rate_name4 | | hour4 | +------------+ billing_rates +-----------------+ | Field | +-----------------+ | id | | reseller_id | | name | | destination | | application | | connectCost | | durationRate | | connectCostIn | | durationRateIn | +-----------------+ billing_rates_history +-----------------+ | Field | +-----------------+ | id | | reseller_id | | name | | destination | | application | | connectCost | | durationRate | | connectCostIn | | durationRateIn | | startDate | | endDate | +-----------------+ destinations +--------------+ | Field | +--------------+ | id | | reseller_id | | gateway | | domain | | subscriber | | dest_id | | region | | dest_name | | increment | | min_duration | | max_duration | | max_price | +--------------+ billing_holidays +-------+ | Field | +-------+ | day | +-------+ prepaid +------------------+ | Field | +------------------+ | id | | reseller_id | | account | | domain | | balance | | change_date | | active_sessions | | session_counter | +------------------+ prepaid_history +-------------+ | Field | +-------------+ | id | | reseller_id | | username | | domain | | action | | number | | value | | balance | | date | +-------------+ billing_enum_tlds +-------------+ | Field | +-------------+ | id | | reseller_id | | gateway | | domain | | subscriber | | enum_tld | | e164_regexp | | discount | +-------------+ Web based rating tables management ---------------------------------- The rating tables can be edited from the web, click on Rating tables link. One may insert/update/delete records or apply the changes on selections. For example it is possible to increase with XX units the rate for a specific destination. Numeric fields support mathematical operators [+-*/], one may update using absolute or relative values the fields in the rating tables. The rates may be copied in bulk and start quickly working with a fresh new rating table. Select in the Rates PSTN table by filtering on rate name. A new button appears which allows the copy of all selected rates into a new set. The rates are copied under the old rate id with _N suffix where N is the next available number for which same rate id does not exist. The content of the rating tables can be exported into comma separated files. The CSV format has the same structure as the import file, is fairly easy to modify an exported batch file into an external application and load it back into CDRTool. Note When $RatingEngine['split_rating_table'] is true, after changing the rates in the web interface or by importing them, you must run the script scripts/splitRatingTables.php to split the central billing_rates table into individual tables for each rate id. You do not need this feature if you have less than 100K rates in your system. ENUM TLD discounts ------------------ To apply discounts based on ENUM certain conditions must be met. 1. The username part of the result of the ENUM lookup must be numeric and contain a fully qualified E164 number, optional with a numeric prefix. Example: The user dialed 020800001, the SIP Proxy has normalized the destination based on local policy by stripping 0 and adding country code 31 to obtain the fully qualified E164 number 3120800001, than it performed an ENUM lookup under top level domain e164.example.com for +3120800001. The ENUM server responsable for e164.example.com returned a response with the destination sip:01131208000011@gateway.example.com In the CDR, ENUMtld is stored as e164.example.com The rating logic checks if the TLD exists in the billing_enum_tlds table. If it does, the rating engine tries to match the regexp field against the username part of the destination from the ENUM response, which has been saved in the Canonical URI. The match must return a fully qualified E164 number otherwise the call is considered to have ended to a no E164 destination and the call will be free of charge. 2. In the ENUM tld table you must provision (for the example above): - TLD: e164.example.com - Regexp: 011([1-9][0-9]{7,}) - Discount: 25 The parenthesis of the Regexp field indicate the E164 number returned by the match and discount is a percentage that will be subtracted form the total price of the call. The formula is described in the PSTN rating section. Reloading rating tables ----------------------- The rating engine loads some of the rating tables in the memory, when the tables change a reload is needed. Reload of rating tables is possible without stopping the daemon by connecting to it and issuing the reload command. The init.d script can also be used for reloading the rating engine with the current values from the rating database. There are 3 ways of reloading the rating tables: a. Each change executed in the WEB interface for rate management may update the rating tables. If there is a change made to the database that requires a reload the link 'Reload rating tables' appears in red color on web page. Click on the link to execute the reload. b. Telnet to the IP address and port number specified for the Rating engine in global.inc. Type help to see the list of commands available. Locate the reload rates command and execute it followed by \n. You may see the result of the command in syslog. The results displayed by syslog will show how many entries have been reloaded from the rating table. c. Run /etc/init.d/cdrtool reload command Troubleshooting --------------- To examine the rate information for a rated call click on the Id field on the leftmost column. (Java script support in browser is required). A blue area will open under the CDR line containing more information about the SIP session. If you see no price in the CDR or no rating information appears in the call details it means that either no destination was found in the destinations table or no rate has been associated with that destination. Make sure that for each entry in the destinations table there is a corespondent entry in the rates table. CDRTool rating engine can send warning emails if it finds missing entries in the rating tables if the system where CDRTool runs is properly configured to send emails and the e-mail notification addresses are set in global.inc: $CDRTool['provider']['toEmail'] = "support@example.com"; $CDRTool['provider']['bccEmail'] = "cdrtool@example.com"; To log to syslog about missing rates or incorrect setup of the rating tables enable 'reportMissingRates' in $CDRTool['rating'] section of global.inc Renormalizing CDRs and historical rating ---------------------------------------- Sometime is useful to be able to change the rates for calls that have been already normalized and rated, for example after changing the rating tables you wish to apply the changes for the previous month for a customer. To re-rate the CDRs do the following: 1. Change the current rates by using cvs files/WEB interface or add rates valid for specific dates/destinations in the rates_history table 2. Re-normalize the calls to be re-rated by either selecting ReNormalize check-box in the search screen or by changing the Normalized field in the CDR MySQL table (e.g. radacct): Examples: a) Re-rate calls for this month (2004-12) SIP domain example.com: UPDATE radacct set Normalized = '0' where Realm = 'example.com' and AcctStartTime >= '2004-12-01' b) Re-rate calls for SIP subscriber sip01@example.com: UPDATE radacct set Normalized = '0' where UserName = 'sip01@example.com' 3. Apply rating again using command: /var/www/CDRTool/scripts/normalize.php Notes Renormalization process can take long time during which your database (radacct table) will be intermitently locked. Perform this operation only during low traffic periods. It is advisable to re-rate only the CDRs for destinations that have different rates. To do this, select a filter in the CDR search screen, if the selection is right re-run the query by selecting Re-normalize button. After renormalization, the monthly usage information used by the quota system will be out of date. At the next run of the quotaCheck script, a full table scan will be performed. See QuotaSystem.txt for more information about quota. Holidays must be added as individual days YYYY-MM-DD in table billing_holidays. The profile applied for holidays is the same as for week-ends. Holidays are global and cannot be specified per customer. Renormalization process does not affect the balance of prepaid users. Prepaid is a real time un-reversible process, it goes in one direction. The prepaid balance is changed only by placing a call or adding credit to it. There are several reasons for this: - The balance before and after each CDR is not known to be able to roll it back at a later time - Re-rating correctly is mathematically not possible for prepaid users that have calls in progress - If the prices are higher than previously debited and end up with a negative balance, the software cannot force the user to pay more retroactively Re-normalization for the purpose of re-rating is useful only for postpaid accounts where you send an invoice at the end of the month and your can change things back and forth. If you need to perform manual credit/debit operations to some prepaid users because of faulty pricing, you can edit in CDRTool in the prepaid table the balance by using + or -. Known limitations ----------------- The rating engine does not calculate prices based on the outbound carriers or outbound gateways, the rating plan is is assigned by the calling party and not by called party. Price discounts (except those based on ENUM tld) must be applied outside CDRTool, in the billing system that prints the actual invoices. CDRTool has no possibility to rate only calls after X minutes per month for subscriber Y, all calls are rated uniformly. Performance ----------- Rating is part of the normalization process that happens every time a query is executed in the web interface or when the rating engine is contacted by the SIP Proxy or by the User Agent that performs the prepaid application. The following tests have been performed between two machines with 3 GHz CPU and 1 GB memory located on the same LAN having a round trip time of 0.2 ms. The rating tables have been populated durring the tests with: Aug 11 11:25:43 sip03 CDRTool[4945]: Loaded 8135 destinations Aug 11 11:25:43 sip03 CDRTool[4945]: Loaded 6 profilesPSTN Aug 11 11:25:43 sip03 CDRTool[4945]: Loaded 4 holidays Aug 11 11:25:44 sip03 CDRTool[4945]: Loaded 7273 prepaid accounts Tests results for the postpaid application: Clients Rating command Execution speed ---------------------------------------------------------------- 1 ShowPrice 390/s per client 5 ShowPrice 100/s per client 10 ShowPrice 60/s per client Tests results for the prepaid application Clients Rating command Execution speed ---------------------------------------------------------------- 1 MaxSessionTime/DebitBalance 250/s per client 5 MaxSessionTime/DebitBalance 80/s per client 10 MaxSessionTime/DebitBalance 40/s per client Client means either a SIP Proxy entity or a CDRTool server, which performs the normalization process. The rates are read directly from MySQL from version 6.1 instead of being cached as they are found based on a known index. To monitor the connections to the rating engin server telnet to the rating engine port and issue the ShowClients command. Example output: ShowClients Clients: 1. 91.20.228.143:32837 2. 85.1.86.71:57945 3. 91.20.228.146:35098 4. 91.20.228.150:34285 5. 91.20.228.129:55090 6. 91.20.228.164:34147 Requests: 12 requests from 91.20.228.129 11 requests from 91.20.228.150 8 requests from 85.11.86.71 2 requests from 91.20.228.164 Statistics: Total requests: 33 Uptime: 169 seconds Load: 0.20/s