Open with table of contents

Import Lexical Data

By using the Lexical Data Import page you can import lexical data that your Gateway can use to detect specifically defined lexical expressions. For example, you can configure your security policy to search for a specific set of account numbers, listed words or phrases (lexical data).

You prepare your lexical data by using the ffcreate utility to create data files on an upload server or file hosting site. You then use the Lexical Data Import page to upload to your Gateway.

Tell me about...

• The Lexical Data Import page

  You use this page to configure how and when you want the Gateway to import your data. The page consists of three sections:

  • Import Schedule - create an automatic schedule for importing your data.
  • Server Settings - configure your upload server details.
  • Import Files - select the paths required to locate your input files.
• Lexical Data

  Lexical Data is held in Data Sets (*.leq) which can be used as references by lexical expression qualifiers. Each qualifier looks up a Data Set and a Key which identifies the appropriate column within the data set.

  Data must be converted from a tab-separated input file to a data set before it can be imported to the Gateway. Only exact matches will trigger the security policy.

• The ffcreate utility

  You use the ffcreate utility to convert tab-separated input files into data sets which are compatible with the Gateway. Data sets use Key fields to reference useful data from your source. The utility must be run from the command line. You need to download this utility from the Clearswift support site.

  You will need to install the following package to run the ffcreate utility in a Windows environment. https://www.microsoft.com/en-us/download/details.aspx?id=40784

How do I...

• Download the ffcreate utility?

  1. Download ffcreate480.zip from the Clearswift Support Portal and save to an appropriate location.

     If you are using an older version of the Gateway, you will need to download ffcreate.zip from the Support Portal.

  2. Extract the zip file and save the ffcreate directory.

     Do not remove the contents of the ffcreate directory from the folder.

• Prepare my data?

  Input files must:

  • be a flat tab-delimited file
  • be encoded in UTF-8
  • not contain column headers
  • consist of entries on separate rows
  • not contain header properties

  • have CRLF (Windows) line termination

• Convert an input file to a data set (.leq)?

  1. Prepare your input file as tab-separated data.
  2. Save the file in a suitable directory.

  3. Run the ffcreate command line utility in the same directory. Configure the key fields and the structure of your output file using the command line:

     ffcreate -k <key_nameThe reference the  will look for. Each key must be composed of alphanumeric characters a-z, A-Z or 0-9 and must match one reference from the schema.> -s <schemareference,type;reference,type... index of the columns available in your input file. Type must be 'Simple', 'SimpleNC', or 'CCNUMBER'. If you are using Linux, enclose the schema in quotation marks (").> -i <input_fileThe name of the file containing your source data> -o <output_fileThe required name of the output file (.leq)>

     You must include the file extension of the input file, and append the output file with the extension .leq to ensure it can be accessed by the Gateway. Key names must consist of alphanumeric characters only: a-z, A-Z and 0-9.

     Schema is a semicolon-separated list of “name,type” pairs, where “Name” is the name you want to give the field and “Type” is one of:

     • Simple - for any field (case-sensitive match)

     • SimpleNC - for any field (case-insensitive match)

     • CCNUMBER - for a credit card number

     Example: Customer account details

     The file Client_Info.txt contains data which is arranged in tab separated columns as follows:

     First Name, Second Name, Contact Number, Email Address, Credit Card Number, Payment Number.

     

     • Run the ffcreate utility to generate your Data Set and Key Fields:

       ffcreate -k FirstName -k SecondName -k CCNum -s FirstName,SimpleNC;SecondName,SimpleNC;ContactNum,Simple;Email,Simple;CCNum,CCNUMBER;PaymentNumber,CCNUMBER -i client_info.txt -o ClientInfo.leq

     This generates a data set called ClientInfo.leq with key fields corresponding to the data in your input file.

     Data Set	Key	Corresponds to column containing...
     ClientInfo	FirstName	First names
     ClientInfo	SecondName	Second names
     ClientInfo	CCNum	Credit card numbers

     Each Key field must match a unique reference in the schema. You cannot create keys or schema references with identical names.

  4. Import the data set. You can configure a schedule to import data automatically or Retrieve data sets now.

  5. You can update client_info.txt at any time but you must re-configure and re-import the Data Set using the ffcreate utility if:

     • The file is renamed
     • The file content changes (for example, new rows are added)
     • The order of the columns changes
• Configure my upload server?

  1. Click System > Gateway Settings > Lexical Data Import.

  2. In the Server settings section, click Click here to change these settings.
  3. Enter your server settings:

     Setting	Description
     Server type	Type of connection: HTTPS, FTPS or SFTP
     Address	IP address or hostname of the upload server or domain name of the hosting site
     Port	The port connecting the Gateway to your upload server
     Use authentication	Select if server requires authentication. Do not select if you are using anonymous authentication on an FTPS server.
     User name	Authentication user name
     Password	Authentication password

     Select Use untrusted certificate to ensure that any self-signed certificates from a file hosting site can be imported automatically. If this check box is cleared, you will only be able to import untrusted certificates by using the console.

  4. Use the Import Files section to add the relative File Path of your data set. Click New and enter the path. Click OK.

     File paths must include the name of the file with directories separated by / characters. For example: /cust_data/cc_numbers.leq.

  5. Click Save.

     Use the Test server settings option in the task pane to check the status of your server.

• Import a security certificate to allow import from a secure server?

  To import a security certificate (cert.mycompany.org), use the following command:

  keytool -import -noprompt -trustcacerts -alias cert.mycompany.org -file cert.mycompany.org -keystore /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.65.x86_64/jre/lib/security/cacerts -storepass changeit 

• Import lexical data now?

  We recommend that you select a time when there is minimal activity on your Gateway for the following reasons: Importing large data sets from the upload server can affect Gateway performance. Importing lexical data file will restart the Gateway engine.

  1. Click System > Gateway Settings > Lexical Data Import.
  2. From the task pane, click Retrieve data sets now.
  3. Click Continue.
• Create or edit an import schedule?

  We recommend that you select a time when there is minimal activity on your Gateway for the following reasons: Importing large data sets from the upload server can affect Gateway performance. Importing lexical data file will restart the Gateway engine.

  1. Click System > Gateway Settings > Lexical Data Import.

  2. In the Import Schedule section, click Click here to change these settings.
  3. Select Enable lexical data import.
  4. Use the time grid to specify your schedule. You can configure an import once per day.
     Example:

     I want lexical data to be imported at 02:00 on weekdays.

     

  5. Click Save.
• Check that the import has been successful?

  Import success or failure is recorded in the Infrastructure log. See Clearswift Gateway Logs for more information.

• Use lexical data on conjoined Gateways?

  You can share lexical data and lexical expression qualifiers across conjoined Gateways. Each peer type should be configured to include the Import Schedule, Server Settings and File Path in the Systems Center. For example, if your peer group contains a Web Gateway and an Email Gateway, you should configure the Lexical Data Import settings in the System Centers of both Gateways.

• Remove a data set?

  We strongly recommended that you check that the data sets you want to remove are not used by your configured security policy. You will not be able to remove qualifiers which are currently used by lexical expression lists.

  1. Remove the location reference of the data set from the Import Files section on the Lexical Data Import page.
  2. Check lexical expression lists for expressions which use qualifiers.
  3. Remove any lexical expression qualifiers which use the data sets. Click Policy > Lexical Expressions > Lexical Expression Qualifiers
  4. Apply the configuration.

  5. Log into the Gateway using SSH, remove the data sets and summary files from their location using the rm command.

     /var/cs-gateway/lexical_data_import/data_file.leq

     /var/cs-gateway/lexical_data_import/summary.xml

  If you remove an active data set from the Gateway, any content rules which use this data will be ignored by the policy.

See also...

• Lexical Expression Qualifiers
• Working with Lexical Expressions

© 1995–2018 Clearswift Ltd.