Trophy iconTrophy iconClose icon
Tutorials

Twitch Streaming Graph Analysis - Part 2

by
Katarina Supe
Blog hero image

Introduction

This blog is divided into three parts, depending on the part of the application we are building:

  • Part 1: data source and backend implementation
  • Part 2: frontend implementation
  • Part 3: streaming data from Kafka cluster

If you still haven’t, you can read the previously published Part 1 and then continue reading this post. Otherwise, you can use the already implemented backend. In this part, we are going to create React application and visualize general statistics and some interesting insights from the Twitch dataset. All implementation that will be mentioned in this part of the blog you can find in the frontend folder of the project.

Create a React App

Let’s figure out how we’re going to visualize all the data we have managed to collect. First, we have to create React app which, will work with our Flask application. We will install Node.js which, will provide us the npx command for creating a React app. Place yourself in the project root folder and run:

npm install -g [email protected]
npm init react-app frontend --use-npm
cd frontend
npm start

Now at http://localhost:3000 you can see a simple React app. We will configure our React app to work well with the Flask server. In package.json from the frontend folder, add a line at the end of the file: "proxy": "http://localhost:5000" This will tell React app to redirect any requests it receives on its port 3000 to port 5000, where our backend is implemented. We will use Semantic UI to build our webpage, so we have to do a few more things before dockerizing our React app. Run the following commands:

npm uninstall semantic-ui semantic-ui-css
npm install @craco/craco @semantic-ui-react/craco-less semantic-ui-less --save-dev

After that, update your package.json with:

{
  "scripts": {
    "start": "craco start",
    "build": "craco build",
    "test": "craco test",
    "eject": "craco eject"
  }
}

Create craco.config.js in the frontend folder and paste the following content in it:

module.exports = {
  plugins: [{ plugin: require('@semantic-ui-react/craco-less') }],
}

Create a semantic-ui/site folder in the src folder and then copy the entire node_modules/semantic-ui-less/_site folder content to src/semantic-ui/site. Also, create a theme.config file in the src/semantic-ui/ folder and then copy the file node_modules/semantic-ui-less/theme.config.example to src/semantic-ui/theme.config.

Update the theme.config file to:

/*******************************
            Folders
*******************************/

@themesFolder : 'themes';
@siteFolder  : '../../src/semantic-ui/site';

@import (multiple) "~semantic-ui-less/theme.less";
@fontPath : '../../../themes/@{theme}/assets/fonts';

If it’s not already installed, run: npm install semantic-ui-react --save-dev

Now you can use Semantic UI components, such as Button (don’t forget to paste all the imports):

import logo from "./logo.svg";
import "./App.css";
import "semantic-ui-less/semantic.less";
import { Button } from "semantic-ui-react";

function App() {
  return (
    <div className="App">
      <header className="App-header">
        <img src={logo} className="App-logo" alt="logo" />
        <p>
          Edit <code>src/App.js</code> and save to reload.
        </p>
        <Button>Click me</Button>
      </header>
    </div>
  );
}

export default App;

How to Dockerize a React App?

We will be adding a few new lines in our previously created project’s docker-compose.yml file. At the end of the file, add:

react-app:
    build: ./frontend
    volumes:
      - ./frontend:/app
      - /app/node_modules
    ports:
      - "3000:3000"
    depends_on:
      - twitch-app
    networks:
      - app-tier

You should also create a Dockerfile in the frontend folder like this:

# pull official base image
FROM node:14.17.5-alpine

# set working directory
WORKDIR /app

# add `/app/node_modules/.bin` to $PATH
ENV PATH /app/node_modules/.bin:$PATH

# install app dependencies
COPY package.json ./
COPY package-lock.json ./
RUN npm install --silent
RUN npm install [email protected] -g --silent

# add app
COPY . ./

# start app
CMD ["npm", "start"]

Here you can see we are copying package.json into the container. The best way to have all dependencies installed is to copy package.json file from here. All node modules will then be correctly installed in the container and npm start will run your React app. Node modules volume was added so that all packages don’t have to be installed each time you build your project.

All that is left to do is to create a .dockerignore file in the frontend directory:

node_modules
build
.dockerignore
Dockerfile

Also, make a little change in package.json for the proxy settings:

"proxy": "http://twitch-app:5000"

This change is made because we are running the backend as a Docker service on port 5000 called twitch-app (this is defined in the docker-compose.yml file). Our project structure now looks like this:

|   docker-compose.yml
|
+---backend
|       app.py
|       Dockerfile
|       requirements.txt
|
+---frontend
|   |   .dockerignore
|   |   craco.config.js
|   |   Dockerfile
|   |   package.json
|   |   package-lock.json
|   +---node_modules
|   +---public
|   +---src
|
+---memgraph
|   |
|   +---import-data
|   |       chatters.csv
|   |       moderators.csv
|   |       streamers.csv
|   |       teams.csv
|   |       vips.csv
|   +---mg_log
|   +---mg_lib
|   +---mg_etc

Now you can hit docker-compose build from your root project folder and docker-compose up after that. First memgraph-mage will run and then twitch-app. After that react-app will be run. Now you can make requests from your frontend.

Frontend Implementation with React and D3.js

Create folder components in your src file. Here you will make your components which you’ll use as puzzles for your web application. Let’s make a little part of the puzzle, where we’ll show the fetching of data from Memgraph. We will add node and edge counters to our webpage by making fetch requests in Counter.js. In the code below you can see that we are making a request depending on the props forwarded from the parent component.

  fetch() {
    fetch("/" + this.props.count)
      .then((res) => res.json())
      .then(
        (result) => {
          this.setState({
            isLoaded: true,
            counter: result[this.props.count],
          });
        },
        (error) => {
          this.setState({
            isLoaded: true,
            error,
          });
        }
      );
  }

On the left side, you can see the number of nodes, and on the right the number of edges in your database.

memgraph-tutorial-twitch-header

For game statistics, we are fetching top games from the backend server:

  fetchData(number) {
    fetch("/top-games/" + number)
      .then((res) => res.json())
      .then(
        (result) => {
          this.setState({
            isLoaded: true,
            games: result.games,
            players: result.players,
          });
        },
        (error) => {
          this.setState({
            isLoaded: true,
            error,
          });
        }
      );
    this.setState({
      numOfGames: number,
      header: "Top " + number + " games",
    });
  }

On the right side, you can see the table showing you the names of the games and the number of players that are playing that game (in our dataset).

memgraph-tutorial-twitch-stats

In a similar way, we are fetching top teams, VIPs, and moderators. For streamers, we are ranking them by the number of followers or number of views. Because of that, you have a dropdown menu from where you can choose the way of ranking.

memgraph-tutorial-twitch-streamers

Let’s talk about graph visualization a bit more. Here, we’re going to use D3.js. This is a package which we still cannot use. We need to set everything up so that we can draw our graphs using D3.js. Create a folder hooks in src folder and create useD3.js file (hooks are usually named with prefix “use”).

import React from "react";

import * as d3 from "d3";

export const useD3 = (renderGraph) => {
  const ref = React.useRef();

  React.useEffect(() => {
    renderGraph(d3.select(ref.current));

    return () => {};
  });

  return ref;
};

This will be your custom hook to allow D3.js to interact directly with the DOM. You can take advantage of the useRef and useEffect hook to link D3.js with the svg element that has been created, and specifies when your D3.js function should be executed. Don’t forget to import d3. Now we can render our graph using a custom hook useD3.js. Check the Graph.js component to see how the graph can be drawn. Using that component we can get information about your favorite streamer - its teams, games, and languages, like in the image below.

memgraph-tutorial-twitch-favstreamer

It is also possible to search all streamers who are playing some game in a certain language. Using the same Graph.js component, you get:

memgraph-tutorial-twitch-game-lang

Feel free to play with nodes and their forces by dragging them around. In the end, we will use powerful MAGE query modules - PageRank and Betweenness Centrality, to showcase how you can visualize your data in a pretty cool way with D3.js. For calculating PageRank, we have an API GET request in the backend server and in the frontend, we’re fetching that data from the PageRank.js component.

  fetchData(){
    fetch("/page-rank")
      .then((res) => res.json())
      .then(
        (result) => {
          this.setState({
            isLoaded: true,
            nodes: result.page_rank,
          });
        },
        (error) => {
          this.setState({
            isLoaded: true,
            error,
          });
        }
      );
  }

While PageRank results are being calculated, you will see your results loading. After the results are loaded, they are drawn with the graph component GraphPR.js, which uses D3.js. Next to the graph, you can see the results table with the names of the streamers and their calculated rank.

memgraph-tutorial-twitch-page-rank

You can see the refresh button above the visualized graph. It will be used later on in Part 3 of the tutorial, when we’ll show how you can stream your data using Kafka. We will stream new chatters of user BadBoyHalo and see how his rank improves with a larger number of chatters in his network. Similarly, we calculated the Betweenness Centrality and showed the results below.

memgraph-tutorial-twitch-bc

Conclusion

And that’s it for now! I hope you got everything right, but if you have any questions or want to give some feedback, feel free to join our Discord Community Server. Make sure to follow up on the last part of this blog, where you can learn how to tackle your streaming data with Kafka and analyze it with Memgraph in real-time.

Table of Contents
Sign up for our Newsletter

Continue Reading