Tuesday, May 11, 2021

How to get headless chromium to launch from a pipeline using ubi8 image

My goal was to be able to run unit tests from a pipeline for the Angular's "Tour of Heroes" tutorial application which use Karma Jasmine framework. In this case the pipeline was built using Tekton to deploy to a Red Hat OpenShift cluster. 

Step 1: Change Karma Browser Configuration to use Headless Chromium

Launching the unit tests locally was not an issue because I had Chrome installed locally, but in order to get this to work in a pipeline we need to switch to a Headless browser configuration. This can be accomplished by updating karma.conf.js as follows. This configuration prunes all interactivity, not all of these settings are necessary, but these will work both locally and from the pipeline task:

   reporters: ['progress'],

    port: 9876,

    colors: true,

    logLevel: config.LOG_INFO,

    autoWatch: false,

    browserNoActivityTimeout: 30000,

    browsers: ['ChromiumHeadlessNoSandbox'],

    customLaunchers: {

      ChromiumHeadlessNoSandbox: {

          base: 'ChromiumHeadless',

          flags: ['--no-sandbox', '--headless', '--disable-gpu', '--disable-translate', '--disable-extensions', '--remote-debugging-port=9223']

      }

    },    

    singleRun: true,

    restartOnFileChange: false


Step 2: Add Puppeteer to the project

With these changes the local tests were running headless successfully, however they were still failing because they still need a browser binary downloaded. Our pipeline was using the ubi8/nodejs-12 image to run npm tasks. All of the npm tasks (e.g. install, lint, build) were running successfully, except for the test task. One easy solution would have been to install chromium into a modified image using a Dockerfile with ubi8/nodejs-12 as the base. Unfortunately the chrome download url was not whitelisted from the pipeline server and could not go that route. 

This led us to using puppeteer. Puppeteer is a node library which allows you to test using headless chromium. One of the features that we needed is that it downloads the chromium binary.

Step 2.1: Add puppeteer as a dev dependency

Puppeteer can be easily added as a development dependency by adding it to package.json within devDependencies section:

   "puppeteer": "^9.1.0",

Step 2.2: Export download host for puppeteer

This may not be an issue locally, but in case npm install gets stuck downloading puppeteer, this can be addressed by adding this download host environment variable prior to running npm install (ensure this is added to the pipeline task as well):


      export PUPPETEER_DOWNLOAD_HOST=https://npm.taobao.org/mirrors 


Step 2.3: Update karma configuration to use puppeteer

Update karma.conf.js with the following changes:

  • Require puppeteer
  • Export the path to the Chromium executable
  • Wait for chromium to launch (i.e. to ensure it is downloaded if applicable)

 const puppeteer = require('puppeteer');

  process.env.CHROMIUM_BIN = puppeteer.executablePath();

  (async () => {

    const browser = await puppeteer.launch({

      args: ['--no-sandbox', '--disable-setuid-sandbox'],

    });

  })();

Step 3: Add required Linux libraries to ubi8 image

At this point npm test should execute successfully with puppeteer and headless chromium. However tests were failing in our pipeline due to missing Linux libraries in the ubi8 required to run chromium:

06 05 2021 20:58:47.098:ERROR [launcher]: Cannot start ChromiumHeadless /workspace/source/node_modules/puppeteer/.local-chromium/linux-869685/chrome-linux/chrome: error while loading shared libraries: libnss3.so: cannot open shared object file: No such file or directory

To troubleshoot this error the following commands were instrumental:

Command #1 (from troubleshooting puppeteer)

This command will help troubleshoot which libraries are missing in one shot versus adding one library at a time and getting a different error. You can get the path to local-chromium by echoing the output of puppeteer.executablePath() - it should be very similar to the one below except for the linux version number:

ldd /workspace/source/node_modules/puppeteer/.local-chromium/linux-869685/chrome-linux/chrome | grep not

Command #2 (from troubleshooting puppeteer)

The second command that was helpful was to run yum whatprovides on each missing library from the Dockerfile building the custom image. The only caveat is that in a few cases what provides resolved to the i686 library and that did not provide the missing library instead I had to switch to the x86_64 library version.

yum what provides libnss3.so

Magic Dockerfile

The hardest part of this task was to arrive to the correct Dockerfile. After iterating through the commands above, we arrived to the right dockerfile what allowed Puppeteer running headless chromium successfully:

FROM registry.access.redhat.com/ubi8/nodejs-12:latest

USER root

RUN yum install -y alsa-lib.x86_64 atk.x86_64 cups-libs.x86_64 gtk3.x86_64 \

    libXcomposite.x86_64 libXcursor.x86_64 libXdamage.x86_64 \

    libXext.x86_64 libXi.x86_64 libXrandr.x86_64 libXScrnSaver.x86_64 \

    libXtst.x86_64 pango.x86_64 xorg-x11-fonts-Type1 libdrm-2.4.101-1.el8.x86_64 \

    mesa-libgbm-20.1.4-1.el8.x86_64 libxshmfence-1.3-2.el8.x86_64 nss.i686 \

    && yum update -y && yum clean all 

USER 1001

Hopefully this guides saves some time for others trying to get this to work on a ubi8 image!



Thursday, May 16, 2019

Spring Integration demo for exposing a SOAP Web Service which consumes another Web Service

Even in 2019, we often still need to develop SOAP based web services due to the need to integrate with legacy applications still using them. It was hard to find code samples that use Spring Integration with Java DSL using Spring Boot to expose/consume SOAP based web services. I put together an example which uses all of the above:
  1. initial branch shows an initial implementation of a web service exposed using spring-ws, Spring Boot and Java DSL. This was loosely based on the "Producing a SOAP web service" getting started guide.
  2. inboundgateway branch takes the initial branch and converts it to using Spring Integration classes (i.e. MarshallingWebServiceInboundGateway and MessageChannel). Credit to tomask79's spring boot web service integration repo which was one of few sample codes I found to expose a SOAP service using Spring Integration with Java DSL. In particular this commit will show you how easy it is to go from Spring-WS to a Spring Integration implementation.
  3. outboundgateway branch introduces a MarshallingWebServiceOutboundGateway to call into another SOAP web service to get order details. It also introduces the use of @EnableIntegrationGraphController annotation to build a visualization of the flow using spring-flow-si

The visualization tool output is fairly impressive:


However, I'm not convinced Spring Integration is the best fit for this particular use case, i.e. integration flows primarily between web services and some JMS services. Here are some of the things I did not like:
  1. Code flow becomes hard to follow. Flows are connected primarily via channels. I introduced a ChannelNames class with constants to make it easier to follow the flow, but even with this, I had to find usages of the constant and then navigate to those classes.
  2. The request/response web service methods had to be decoupled into a "request" (request channel) and "response" (reply channel) method. I posted this question in StackOverflow and even though this method could be combined using a MessageGateway, it is not the ideal way of implementing this with Spring Integration. Even with the diagram above, you cannot tell that orderInboundGateway is a 2-way flow, instead it appears a one-way flow.
Don't get me wrong, I liked several aspects of Spring Integration and I would find it useful for other scenarios, but it seems for this one, the complexity introduced outweighs some of the benefits. 

Please chime in the comments if you've successfully implemented similar scenarios with Spring Integration!