Sending Apache Kafka data on web page

Question

I am building a real time energy monitoring system, where the data are from sensors. There will be new data every second. The data used will be aggregated to be rendered as charts. I have looked into real time stream processing with big amounts of data, and it lead me to Apache Kafka.

Right now my web app is using Express js. I am using kafka-node library. Now currently, I manually insert new data through the command line as a producer. In my server code, I have set-up a consumer that listens to Topic1.

Server code:

var express = require('express');
var app = express();
var http = require('http').Server(app);
var bodyParser = require('body-parser');
var urlencodedParser = bodyParser.urlencoded({ extended: false });

var server = app.listen(3001, ()=>{
  console.log("app started on port 3001");
});

var io = require('socket.io').listen(server);


var kafka = require('kafka-node');

let Consumer = kafka.Consumer,
    client = new kafka.Client(),
    consumer = new Consumer(client,
      [
        {topic: 'Topic1', partition: 0}
      ],
      {
        autoCommit: false
      }
  );

app.use(express.static('public'));

consumer.on('message', (message) => {
  console.log(message.value);
  callSockets(io, message.value);
});

function callSockets(io, message){
  io.sockets.emit('update', message);
}

Client code:

<script type="text/javascript">
  var socket = io('http://localhost:3001');

  socket.on('connect', ()=>{
    console.log("connected");
  })

  socket.on('update', (data) =>{
    console.log(data);
  })
</script>

I am using socket.io to emit the message consumed in Kafka. Is there any other way to send Kafka data to client side? It just seems to me using socket.io is not too elegant here. Have I approached it the right way? Any recommendations welcome!

Thank you.

Answer 1

在不具体讨论您的解决方案的情况下，我认为您正在做正确的事情，为客户端创建一个专用的 API 来读取数据。客户端需要能够从某个地方接收更新。唯一“更快”的方法是允许客户端直接从 Kafka 中提取数据，这会大大增加风险，因为 Kafka 并非设计为可公开访问的。

因此，与在 C# 或 Java 中执行相同的操作（需要更多代码）相比，使用 Node.js 的解决方案实际上相当优雅。