ContentConnector FAQs
General
Can ContentConnector send an email if it has not succeeded?
Yes. You can configure configuration\log4j.xml to have an SMTPAppender that only responds to FATAL log messages.
Can I run ContentConnector from a different directory?
ContentConnector configuration uses relative paths by default. This makes it easier to deal with the variety of installation locations and systems that our customers use. The recommended method is to have your scheduler use ContentConnector root directory as the starting or working directory. The next easiest method is to run ContentConnector in a batch/script file. The scripts first task should be to change to ContentConnector root directoy. The last method is to alter the configuration files to use absolute paths instead of relative paths.
Can I run the ContentConnector on the MariaDB platform?
The ContentConnector is compatible with the MariaDB. You would need to enable the MySQL settings in the contentconnector.xml file as follows:
<Schema definition="configuration/DataSource/DataSource-MySQL.xml"/>
<!-- MySQL -->
<Databases Type="MySQL4.0CompatibilityMode">
<Driver>com.mysql.jdbc.Driver</Driver>
<ServerURL>jdbc:mysql://SERVER:3306/DATABASE</ServerURL>
<ProductDataDatabase name="DATABASE" repositoryTablePrefix="" stagingTablePrefix="in_"/>
<Login User="USERID" Password="PASSWORD"/>
<TableType>MyISAM</TableType>
<Loader>Local</Loader>
</Databases>
Can I run the ContentConnector on the Microsoft Azure platform?
The ContentConnector is compatible with the Microsoft Azure platform. You would need to make the following changes to the contentconnector.xml file.
• <loader>BCP</loader> should be used instead of <loader>Native</loader>.
• <option>ConnectorStatus</option> should be commented out.
Can I specify a download.zip to use?
Normally, importnodownload (and legacy commands fullnodownload or incrementnodownload) trigger loading the files from the LocalUnzippedPath directory. However, you can tell ContentConnector to use any copy of download.zip. ContentConnector will unzip that file into the LocalUnzippedPath directory and process it normally. To do so, run ContentConnector as follows: java -jar dataSource-2.2.x.jar datasource configuration\ContentConnector.xml importnodownload path\to\my\otherdownload.zip
Can I use SFTP with the ContentConnector?
Yes, you can use SFTP with the ContentConnector.
The 1st step is to enable the following option in the contiguration.xml file.
<!--Option>Secure FTP</Option>
The 2nd step is to add the CNET SFTP server to your known_host file.
Unix : by connecting manually using the sftp command line tool: sftp sftp.cnetcontentsolutions.com
- The default known hosts file location is ~/.ssh/known_hosts, if it does not exist you can either get it created by the above command or create it in the current folder by other means
Windows: installing SSH from OpenSSH for Windows running in a new DOS window:
- cd <connector install location>
- ssh-keyscan -t rsa sftp.cnetcontentsolutions.com > known_hosts
How do I configure a SOCKS proxy that I have?
The ContentConnector has been tested to successfully work with SOCKS proxy 4 and 5. In the ContentConnector.xml the following line should be set:
<Proxy Host="MYPROXYHOST" Port="MYPORT" Username="" Password=""/>
Replace MYPROXYHOST with the existing SOCKS proxy address and replace MYPORT with the SOCKS proxy port. Currently, the SOCKS username and password are unused and therefore, should remain blank. This line should also be placed after the DigitalContent sections, but before the Thumbnails section.
How do I configure ContentConnector to send an email when it exits with the “FATAL” error?
Contact your local network administrator and request the following:
- Mail account which ContentConnector can use to send emails.
- This information will be used for the ContentConnector mail address, ContentConnector login and ContentConnector password.
- Mail server host name and port.
- This information will be used for SMTP Server Host and SMTP Server Port
- Email address.
- It will be used as notification email address.
When the information is received, please follow the next steps to configure the notification emails:
- Open log4j.xml file (located in the directory "configuration") in a text editor.
- Uncomment the section starting with <appender name="MAIL">
- Update the parameter value with SMTP Server Host in <param name="SMTPHost" value="smtp.host.net" />
- Update the parameter value with SMTP Server Port in <param name="SMTPPort" value="25" />
- Update the value smtpusername with the ContentConnector login in <param name="SMTPUsername" value="smtpusername" />
- Update the value smtppassword with the ContentConnector password in <param name="SMTPPassword" value="smtppassword" />
- Update the parameter value with the ContentConnector mail address in <param name="From" value="connector@yourdomain.com" />
- Update the parameter value with the notification email address in <param name="To" value="customer@yourdomain.com" />
- Uncomment <appender-ref ref="MAIL"/> located in the “root” section.
- Save log4j.xml file.
Then the notification email will be sent to the address specified in the parameter "To" when the ContentConnector has a “FATAL” error.
How do I configure ContentConnector.xml file
When using with DataSource, ContentConnector requires two configuration files: ConnectConnector.xml and DataSource\DataSource-DATABASE.xml. The DataSource-DATABASE.xml file contains platform dependent SQL instructions, and is currently offered in four versions: DataSource-SQLServer.xml, DataSource-MySQL, DataSource-Oracle and DataSource-PostGreSql.xml. The configuration\ContentConnector.xml file contains the setup information for ContentConnector. Locate and edit the ContentConnector.xml file and make the edits to align the behavior of the application to your needs. More details can be found in the integration guide
How do I fix my thumbnails from appearing jagged or with defects?
An option is to install the native JAI library. Install the JRE version of the JAI library for your platform.
How do I know what has changed from one version of ContentConnector to the next?
You may see what has been added, in each version of ContentConnector, by reviewing the file "changes.txt" which is in the same folder of ContentConnector itself.
How do I prepare ContentConnector to use an absolute path?
You need to edit the ContentConnector.xml file. The data file locations, the digital content then the upload file locations must all be changed to absolute paths. Next, the log4j.xml file must be edited. The locations of datasourceLog.txt and datasourceLog.html must be changed to absolute paths. Finally, when the ContentConnector is run, both the JAR file and configuration file must be specified in terms of absolute paths.
How do I run the ContentConnector in debug mode?
To run the ContentConnector in debug mode, make the following update:
- In the configuration\log4j.xml file, replace <priority value="info"/> with <priority value="debug"/>.
With this option enabled, the ContentConnector will log additional debug information and load it into the logs\log.txt file.
How often is ContentConnector upgraded?
Each time 1WorldSync adds a new feature (alternative categorization, lifecycle...), the connector may also require an update to integrate the new tables. It may also be updated due to various changes within the different operating systems.
How often should I run ContentConnector and at what time?
We are not able to guarantee/determine an exact delivery time for the files. This will vary based on that cycle's load. Only one download.zip can be delivered during one distribution cycle and a cycle may normally last one to two days. Therefore, running it once a day should be fine. You may also choose to run the processing of the download.zip several times and at different hours, in a day, if you want it to be processed on the same day it is received.
How to upgrade to the latest ContentConnector version?
Here are the upgrade steps:
For version 2.2.3.4 and higher
- Download the latest ContentConnector version here.
- Copy the JAR file, the schema file (configuration\DataSource) and the config file (ContentConnector.xml) from the new version and replace these files in the old version.
- Modify the config file (ContentConnector.xml) according to your account settings and enable the new options if needed.
- Run the Connector as usual
For versions before 2.2.3.4
- Download the latest ContentConnector version here.
- Copy the JAR file, the schema file (configuration\DataSource) and the config file (ContentConnector.xml) from the new version and replace these files in the old version.
- Uncomment CompatibleSchema option in the config file (ContentConnector.xml)
- Modify the config file (ContentConnector.xml) according to your account settings and enable the new options if needed.
- Run ContentConnector with create command
- Run the ContentConnector as usual
I requested a full file. Do I need to make any adjustments to ContentConnector?
If you are running ContentConnector 2.2.3.4 or later then you do not need any adjustments. ContentConnector detects the file type automatically and processes it accordingly.
If your version is 2.2.3.3 or below, make sure that the option “CheckIsFullDownload” is enabled in the config file (ContentConnector.xml). ContentConnector
will then detect the file type automatically and process it as a full file.
If you are on a version 1.6 or below please contact your Customer Support Team for upgrade.
I set my HTTP proxy settings on the command line, why is it still not working?
On some systems, the Java applications don't seem to pick up the proxy settings from the command line. Instead, find the "net.properties" file on your system and edit the settings from there.
Why am I having to constantly reset table permissions?
This may be a problem if you assign users table level permissions on Oracle or SQL Server. For performance reasons, the ContentConnector sometimes does table swapping. Instead of updating tables, it will build a new table and replace the old table with the new table. This means that the user permissions are not set for the new table. The solution is to set all permissions at the database level. Assign users to a role, and then allow the role to determine what permissions the users have (select only, owner, write access, etc.). This way, the same permissions will apply to any new tables.
Why does the log state that "An increment has been missed"?
The ContentConnector (starting from the version 1.6) detects missed increments. The config file (ContentCoonnector.xml) has the option "CheckMissedIncrement" automatically enabled. It does this by comparing staging cds_Catalog and both staging and production cds_Prod. All products found in staging cds_Catalog must be found in either the current production cds_Prod or the staging cds_Prod.
If you get this error, you would either need to manually download the missed download.zip file and all other download files delivered after that from the Archive directory and run the Connector in nodownload mode or you may request the full file delivery via the Extranet or by sending the request to Customer Support team.
Why don't accents and other special characters show up correctly?
This is a result of the default character set running on most Linux environments. Usually, the default character set is UTF-8. However, most of the DataSource
data files are ASCII, not Unicode. Therefore it should work in Windows, which has a default character set of Windows-1252. On Linux, the most similar
character set is ISO-8859-1. Java provides an easy way to specify an alternate character set. Run ContentConnector with the following command:
java -Dfile.encoding=ISO-8859-1 -jar ...
SQL Server
Can I use integrated security on SQL Server?
Yes, however it will only work with Windows. Add ";integratedSecurity=true" to the end of the database connection URL in the config file. You must also add the path to the correct sqljdbc_auth.dll found in lib\auth\xxx to your PATH. You can do this by editing your system path or using the Java parameter -Djava.library.path.
How can we stop rebuilding the index (DBCC DBREINDEX) after every import process?
In DataSource-SQLServer.xml there is an update policy for rebuilding indexes. It is possible to remove or comment it out and DBCC DBREINDEX will not run:
<!--UpdatePolicyMethod tableName="cds_Prod" methodName="rebuildIndexes" description="*** <p>REBUILD INDEXES"> <UpdatePolicyArg name="tableSet" value="DataSource"/> <br></UpdatePolicyMethod-->
If I don't want to give ContentConnector sysadmin access, how can the tables still be owned by dbo?
ContentConnector login does not need to be in the sysadmin role, however it should own the database. The easiest way to do this is to assign ContentConnector login as the owner when you create the database. If the database has already been created, the command "exec sp_changedbowner 'LOGIN'" will change ownership of the database to the ContentConnector user. Now, when ContentConnector creates tables, it will be done as dbo.
Why do I get Foreign Key error "The ALTER TABLE statement conflicted with the FOREIGN KEY constraint "S_FK_CCTPROD_2_CCTCAT"..."
ContentConnector does swap table operations during the update process. In doing this, it creates copies of the existing table. It could happen that ContentConnector encounters Foreign Key conflict between the existing and swap tables. ContentConnector version 2.2.0.x fixes this issue. Please contact your Customer Support to get the latest version of ContentConnector.
MySQL
Are there any known character set issues?
Most modern versions of MySQL greater than 5.0.26 support the CHARACTER SET syntax. However, a few later versions, including 5.0.32 still do not support the CHARACTER SET syntax. The work around for this is, in the config file, enable the NoMySQLCharacterSetSupport option.
Can I use InnoDB for MySQL?
ContentConnector will work with InnoDB. To enable InnoDB, the TableType must be changed to InnoDB in the ContentConnector.xml file. ContentConnector is only affected in create mode during table creation.
Does MySQL support multiple character sets?
MySQL only supports the CHARACTER SET syntax in versions 5.0.27 and above. This becomes an issue with loading files that do not use the latin1 character set. Currently, this appears in use cases for Polish and Czech languages (1250 character set).
How can I increase the duration of the timeout?
You can use SHOW VARIABLES like wait_timeout%; to see the value of wait_timeout. By default, it is 28800, which is 8 hours. To change the value, you can do type the command set @@wait_timeout=28800;
MySQL is loading slowly. How can I improve performance?
By default, MySQL loads data in local mode. This means that the JDBC driver streams the data files over the network to the database server. This is ideal in an environment where the database server is not on the same machine as ContentConnector or the database user has limited permissions. MySQL also has the option to have the server directly access the data files if the data files are on the same machine as the database server. To do this, the database user must first be granted the FILE permission. Then the following line must be added to the DataSourceConfig-MySQL.xml file immediately after the TableType element in the Databases section: <Loader>NoLocal</Loader>
Why do I get this error: "Could not connect to database: 0 - Unknown initial character set index '192' ..."?
This is a known bug in version 5.0 of MySQL (URL: http://bugs.mysql.com/bug.php?id=22456). The following workaround is avalable: Open ContentConnector.xml. In the databases section, add "?characterEncoding=UTF8" to the end of the ServerURL.
Oracle
Can I run ContentConnector on the same database instance yet use different schemas?
Yes - you can specify the Oracle schema name in the <ProductDataDatabase> element by adding a “schema” attribute. Note that you will still need to provide a user login/password.
My locale uses commas and some values in cds Atr.NNV that are not being imported into Oracle, why?
This is due to locale issues. DataSource uses a decimal for float values. But some European locales use commas instead. The solution is to set the NLS_NUMERIC_CHARCTERS value to ".,". This can be done is a Unix script or Windows bat file.
Why do pipes and degree symbols appear as an upside-down question mark?
Check the environment. The environment may not be set correctly for the Windows character set. The following variable should be in both your shell environment and your automated scripts. NLS_LANG=AMERICAN_AMERICA.WE8MSWIN1252