QuerySurge and Apache Kafka - Lenses (version 3.x and above)

Versions: 8.0+

Apache Kafka is an open source data streaming platform that provides handling for real-time data feeds. The Lenses offering is a "wrapper" for Kafka (among other technologies) that provides access to stream data via the Lenses SQL engine. This article outlines the installation and setup of a Lenses docker container, deploying the JDBC Driver and dependencies to your QuerySurge Agent and setting up a connection from QuerySurge to the L-SQL engine within your Lenses instance.

  Note: RTTS, the vendor of QuerySurge, does not have distribution rights for Lenses, and has no relationship with the vendor. If you download Lenses, your license is with the vendor and not with RTTS. Support for this driver is not provided by RTTS.

  Note: This article applies to Lenses versions 3.x and above. 

1. Lenses Installation and Setup

If you already have a Lenses instance running, proceed to step 2. You will need docker installed to download and run the container. Docker installers for Windows 10 and Linux distributions can be found here. Windows 7 users must use the legacy solution docker toolbox. Once docker is installed you can pull down the latest version of the Lenses box with the following command:

docker pull landoop/kafka-lenses-dev

To run the Lenses container, run the following command:

docker run --rm -p 3030 --name=lenses-dev --net=host -e EULA="https://dl.lenses.stream/d/?id=<LICENSE_KEY>" landoop/kafka-lenses-dev

Note: You will need a license key to run Lenses. You can obtain a free trial key from here. Replace <LICENSE KEY> with your license key in the command above.

This will run the container on the docker machine. Once services have started the Lenses console will be visible in your browser at the address:

<docker machine IP>:3030

To find your docker machines IP use the following command:

docker-machine ip

If you would like to make this instance accessible to your local network you can add the following flag when invoking the docker run command

-e ADV_HOST="<Local Machine IP>"

Note: You may need to add port forwarding rules for port 3030 on the docker VM for it to be accessible on your local network.

At this point you can log into the Lenses console (at <hostname>:3030) using the default credentials admin/admin.

2. Deploying the JDBC Driver

Your QuerySurge Agent requires the appropriate driver and dependencies to connect to the L-SQL engine. A fat jar driver can be download from this archive. For full instructions on driver deployment, consult this article for Windows and this article for Linux.

Note: The fat jar linked above is not compatible with all Lenses versions. If you cannot find a fat jar matching your Lenses version in the archive, follow the instructions below to download the latest jdbc driver and its dependencies.

For Linux-installed Agents, navigate to the Agent JDBC folder:

cd /<QuerySurge Install Dir>/agent/jdbc

Download the Lenses JDBC driver and the POM file. On Linux, use the following commands:

wget https://repo.maven.apache.org/maven2/com/landoop/lenses-jdbc/<version>/lenses-jdbc-<version>.jar
wget https://repo.maven.apache.org/maven2/com/landoop/lenses-jdbc/<version>/lenses-jdbc-<version>.pom

Note: Replace <version> with the version of the Lenses box currently running. 

Rename the POM file and download the dependencies using Maven.

mv lenses-jdbc-<version>.pom pom.xml
mvn install dependency:copy-dependecies

On Windows, you can download your driver (choosing the proper version) from:

https://repo.maven.apache.org/maven2/com/landoop/lenses-jdbc/

You'll also need to download all dependencies for this jar (this can be done via maven commands or via web tools such as https://jar-download.com/).

Once you have downloaded the Lenses JDBC driver jar and all its dependencies, make sure to copy the full set of jars to the Agent jdbc directory (your Agent may be installed on a drive other than the C:\ drive):

C:\<QuerySurge Install Dir>\agent\jdbc

Note:   The driver jar and all dependency jars must be copied to the Agent jdbc directory; this may be several tens of jar files, depending on the version.

To register the driver and the dependencies with your QuerySurge Agent, the Agent service must be restarted.

3. Creating a Connection in QuerySurge

To create a connection to the L-SQL engine we will use the Connection Extensibility feature. For more details on this set up see this article. Key points specific for this driver are:

• Driver Class: io.lenses.jdbc4.LensesDriver
• JDBC URL: jdbc:lenses:kafka:http://<hostname>:3030
• Username/Password: A login for the Lenses console. The default is admin/admin.

Note: The port shown above (3030) is the default port; you may have used a different port.

You can enter a test connection here to ensure connectivity between the L-SQL engine and QuerySurge. A generic SQL statement to try is SHOW TABLES;

Once the connection has been created, you can query data from your Kafka topics in QueryPairs using standard SQL.

In this article we have shown how to:

• set up the Lenses box within docker
• deploy drivers to your QuerySurge Agent(s)
• set up a connection from QuerySurge to Lenses (3.x and above)
• query your topic data.