Apache Kafka is an open source data streaming platform that provides handling for real-time data feeds. The Landoop Lenses offering is a "wrapper" for Kafka (among other technologies) that provides access to stream data via the Lenses SQL engine. This article outlines the installation and setup of a Lenses docker container, deploying the JDBC Driver and dependencies to your QuerySurge Agent and setting up a connection from QuerySurge to the L-SQL engine within your Lenses instance.
1. Lenses Installation and Setup
If you already have a Lenses instance running, proceed to step 2. You will need docker installed to download and run the container. Docker installers for Windows 10 and Linux distributions can be found here. Windows 7 users must use the legacy solution docker toolbox. Once docker is installed you can pull down the latest version of the Lenses box with the following command:
docker pull landoop/kafka-lenses-dev
To run the Lenses container, run the following command:
docker run --rm -p 3030 --name=lenses-dev --net=host -e EULA="https://dl.lenses.stream/d/?id=<LICENSE_KEY>" landoop/kafka-lenses-dev
Note: You will need a license key to run Lenses. You can obtain a free trial key from here. Replace <LICENSE KEY> with your license key in the command above.
This will run the container on the docker machine. Once services have started the Lenses console will be visible in your browser at the address:
<docker machine IP>:3030
To find your docker machines IP use the following command:
If you would like to make this instance accessible to your local network you can add the following flag when invoking the docker run command
-e ADV_HOST="<Local Machine IP>"
Note: You may need to add port forwarding rules for port 3030 on the docker VM for it to be accessible on your local network.
At this point you can log into the Lenses console (at <hostname>:3030) using the default credentials admin/admin.
2. Deploying the JDBC Driver
Your QuerySurge Agent require the appropriate driver and dependencies to connect to the L-SQL engine. A fat jar driver can be download from this archive. For full instructions on driver deployment, consult this article for Windows and this article for Linux.
Note: The fat jar linked above is not compatible with all Lenses versions. If you cannot find a fat jar matching your Lenses version in the archive, follow the instructions below to download the latest jdbc driver and its dependencies.
On the system you have a QuerySurge agent installed, navigate to the JDBC folder:
cd <Agent Install Dir>\agent\jdbc
Download the Lenses JDBC driver and the POM file.
Note: Replace <version> with the version of the Lenses box currently running.
Rename the POM file and download the dependencies using Maven.
mv lenses-jdbc-<version>.pom pom.xml
mvn install dependency:copy-dependecies
To register the driver and the dependencies with your QuerySurge agent, the agent service must be restarted.
3. Creating a Connection in QuerySurge
To create a connection to the L-SQL engine we will use the Connection Extensibility feature. For more details on this set up see this article. Key points specific for this driver are:
- Driver Class: com.landoop.jdbc4.LsqlDriver
- JDBC URL: jdbc:lsql:kafka:http://<hostname>:3030
- Username/Password: A login for the Lenses console. The default is admin/admin.
You can enter a test connection here to ensure connectivity between the L-SQL engine and QuerySurge. A generic SQL statement to try is SHOW TABLES;
Once the connection has been created, you can query data from your Kafka topics in QueryPairs using standard SQL.
In this article we have shown how to set up the Lenses box within docker, deploy drivers to your QuerySurge Agent(s), and set up a connection from QuerySurge to Lenses and query your topic data.