Visualizing Your Cloud Landscape with meshStack’s Cloud Universe

From a technical point of view, the Cloud Universe is a conglomerate of different techniques to provide a solution with the best possible scalability. It is therefore important to take a look at the different topics before we can consider the Cloud Universe as a whole.

SVG in templates

Scalable Vector Graphics (SVG) is a vector image format for visualizing web-friendly icons or simple images. The general structure of an SVG is as follows:

<svg xmlns="http://www.w3.org/2000/svg" viewBox="[...]">
    <g>
        ...
    </g>
</svg>

It is also possible to embed a SVG into another SVG:

<svg xmlns="http://www.w3.org/2000/svg" viewBox="[...]">
    <g>
        <svg xmlns="http://www.w3.org/2000/svg" viewBox="[...]">
            ...
        </svg>
    </g>
</svg>

Angular supports SVG as templates. This means that you can reference instead of a HTML template url a SVG component template url. For example:

@Component({
  selector: 'app-svg',
  templateUrl: './svg.component.svg',
  styleUrls: ['./svg.component.css']
})
export class SvgComponent {
  ...
}

For more information please take a look into SVG as templates .

Sigma JS

Sigma JS is a modern JavaScript library for rendering large network graphs. It is based on Graphology .

• Graphology handles the graph data model and algorithms
• Sigma JS handles the rendering and interactions

It is a developer friendly library and supports a lot of functionality. For example, node and relation reducers. Reducer is a Sigma JS specific concept to manipulate the nodes asynchronously.

For more information please have a look at Sigma JS .

JS lib lazy loading

In general, if you want to use JS libraries within an Angular application, they will be added to main.js during the build process. This means that the size of the main bundle will increase significantly. That’s not what you want. The main bundle should only contain the dependencies that are necessary to get the application up and running. All other dependencies should not be included. It is really important to keep the main bundle size as small as possible to enable a fast initial load time of the application. Such a performance issue is really important to ensure a fast and friendly user experience.

A good example are chart libraries. Chart libraries are not really lightweight. They can easily exceed the size of ~500Kb. Therefore, it is important to exclude such data from the main.js.

That’s why we implemented lazy loading of JS libs. This means that we only fetch the necessary min.js from cdn when needed.

Web Worker

The general use case of Web Workers is to exclude CPU intensive computations in a background thread. For example, if there are heavy computations required that could block or slow down the main thread.

For more information please have a look at Web Worker .

HATEOAS

Hypermedia as the Engine of Application State (HATEOAS). Generally speaking, that the API should guide the client through the application by returning relevant information about the next potential steps, along with each response. It is a constraint of REST application architecture.

The Cloud Universe

The Cloud Universe describes a concept to visualize as user-friendly as possible available entities and their relationships. Sure it could be adapted to every use case but we’ll look at the meshcloud environment within this BlogPost. This means our entities are meshUser, meshGroup, meshCustomer, meshProject and meshTenant.

For more background information, please take a look into the meshcloud docs.

But why is it called Cloud Universe?

A universe consists of multiple different constellations. Each constellation is a collection of stars. If we now apply this definition to our use case then the entity types are our constellations and the stars are our entities. Alright we call it not stars. In our case we’ve chosen subjects. If we now relate this even further to meshcloud then it is important to build a transparent overview about existing entities and their relationships in the cloud landscape. The Cloud Universe!

A technical deep dive

During the implementation we encountered several technical challenges that had a big impact. Our main goal was to deliver a solution with low latency, high scalability and with a really good UI/UX, because there is a fine line between very usable and unusable.

Let’s take a closer look at the technical implementation.

What are the relevant domain models?

As with any HATEOS-driven application, the domain models play an important role. It is therefore useful to take a brief look at them, as they play a central role in the overall Cloud Universe integration.

Cloud Universe Subject Constellation

The Cloud Universe Subject Constellation represents a domain model that provides all necessary information about a constellation. We did a lot of work in this area and finally decided on a structure inspired by HATEOAS.

We provide for each constellation an endpoint to request the specific constellation. In our example we take a look at the meshUser entity. The constellation response looks like the following:

{
  "ids": [
    1,
    4,
        ...
  ],
  "_links": {
    "self": {
      "href": "http://localhost:8080/cloudUniverse/users"
    },
    ...
  }
}

The ids array includes all available entity ids. Mmmh… a plain ids array. Does this really follow HATEOAS? This is a valid objection. That is why we spoke of ‚inspired by‘ in the previous section. In our case we cannot fully realize HATEOAS. Let me give you some reasons.

In general self links are pointers to a specific resource and are in the _links section. If we now apply this to our case, it could look like this:

{
  "_links": {
    "self": {
      "href": "http://localhost:8080/cloudUniverse/users"
    },
    "...": {
      "href": "http://localhost:8080/cloudUniverse/users/1"
    },
    "...": {
      "href": "http://localhost:8080/cloudUniverse/users/4"
    },
    ...
  }
}

We see that we have a lot of duplicates in the response, which inflates the response data. This leads to very poor scalability and won’t work with > 10000 entities without leaving a mark on the user. We want to support the best possible scalability, but this is not possible with so many duplicates, which we can and should avoid. So we only include the entity ids. Based on the ids and the self-link of the response, we can build the specific self-link for each entity. This is an important fact that plays a big role in the rest of the blog post.

Besides that it is important to cache the constellation data within the frontend to prevent unnecessary API calls, which are very costly.

Now we know all about the data structure of the Cloud Universe Subject Constellation and why it looks the way it does.

Cloud Universe Subject

The Cloud Universe Subject represents a domain model that provides all the necessary information about a subject. It has the HAL format, which includes the displayName and all relationships to subjects from other constellations. In the context of meshcloud this would mean for a meshUser that we’ll include all assigned meshCustomers, meshProjects, meshTenants and meshGroups.

For example:

{
  "displayName": "...",
  "subjectRefs": [
    "http://localhost:8080/cloudUniverse/customers/3",
    "http://localhost:8080/cloudUniverse/projects/2",
    "http://localhost:8080/cloudUniverse/groups/94",
    "http://localhost:8080/cloudUniverse/tenants/97",
    "http://localhost:8080/cloudUniverse/tenants/103",
    ...
  ],
  "_links": {
    "self": {
      "href": "http://localhost:8080/cloudUniverse/users/4"
    }
  }
}

Cloud Universe SVG as view model

The Cloud Universe cannot be realized with the domain models alone. We need some handling around the representation and user interaction. It is a normal procedure in Angular to transform domain models into view models. The view models are a kind of structure to store additional information besides the domain model data. For example, to define a specific appearance or to store previously determined property values to improve the performance of the Angular lifecycle. It is always a best practice to provide all information directly, without additional mappings within functions. Function calls within the HTML template can be a major performance bottleneck. It is therefore important to define a proper data transformation chain.

We must first look at the individual necessary components to build the Cloud Universe user view.

• Constellations
• Constellation Relations
• Static Decoration

Each component needs their own representation information and logic to handle further actions. Besides that we need to build a connection between each component to handle emitted events for example. This means we need a central structure which orchestrates all behavior within the Cloud Universe. To bring all this together, we’ve defined the Cloud Universe SVG as our central view model. It consists of multiple CloudUniverseSubjectConstellationSVGs, CloudUniverseConstellationRelationsSVG and CloudUniverseStaticDecorationSVG.

CloudUniverseSubjectConstellationSVG

As you might have guessed, the Cloud Universe Subject Constellation domain model plays an important role here. Based on the cached information, we create the Subject Constellation SVG according to a fixed configuration. This means that each constellation has a specific view box, title, icon, color, backgroundColor, circle, animator, and two observables to get the total value and the subjects data source.

• The view box describes the dimension of the constellation SVG.
• The title, icon, color and backgroundColor are necessary for the look and feel.
• The circle information describes the position and the size within the defined view box.
• To provide a user-friendly experience, we’ve added an SVG animator to provide a smooth transition between the total value view and the detailed subject view.
• The total value observable will be determined depending of the ids total count from the cached Cloud Universe Subject Constellation domain model.
• The subjects data source contains mainly the self link of each subject and a cold observable to lazy load the Cloud Universe Subject ****domain model.

CloudUniverseConstellationRelationsSVG

One main goal of the Cloud Universe concept is to provide a good transparency about relationships. This means we need a view model which determines all available relationships and is able to highlight specific relationships.

CloudUniverseStaticDecorationSVG

The representation of the individual constellations is important. However, we need to add a few static representations that clarify the general understanding of the relationships and classification of the individual constellations.

Cloud Universe SVG in action

Creating the Cloud Universe SVG view model is one thing, but how are we going to use the view model inside the HTML template? This is another challenge to provide an embedded SVG view depending on such a complex structure. But let me elaborate a bit.

Angular supports out of the box the integration of SVG in templates. This means we can easily create a component to visualize a SVG representation. Now you could assume that based on the existing SVG view model information we can define separate components to display the Constellations, Relations and Static decorations. For single views this works, of course. But how do you want to realize a hierarchical SVG with multiple layers? A naive approach would be to pack all SVG components into a top-level SVG.

<svg xmlns="http://www.w3.org/2000/svg" ...>
  <g>
    ...
    <mst-cloud-universe-subject-constellation></mst-cloud-universe-subject-constellation>
    ...
  </g>
</svg>

This approach does not work because SVG can’t handle Angular related components. Now you might think that you can reach your goal with directives. Unfortunately, this is not a good approach either, because a lot of logic has to be handled. Therefore we’ve implemented a template based solution to provide a modularized way to easy add/remove parts from the Cloud Universe.

NgTemplateOutlet based SVG creation

The solution is to use NgTemplateOutlet combined with ng-container to build the Cloud Universe representation with all different parts.

Why ng-container? The reason the same argument like the approach with separate components. We don’t want to add SVG unknown DOM elements. Therefore we handle each template within a ng-container.

Additionally we need context information for each Cloud Universe SVG template. The information is easily accessible within the central Cloud Universe SVG view model. This would look like for the meshUser constellation like the following:

<svg xmlns="http://www.w3.org/2000/svg" ...>
  <g>
    ...
    <ng-container *ngTemplateOutlet="cloudUniverseConstellation; context: { 
      constellation: cloudUniverseSVG.userConstellation
     }">
    </ng-container>
    ...
  </g>
</svg>

<ng-template #cloudUniverseConstellation let-constellation="constellation">
  <svg xmlns="http://www.w3.org/2000/svg" ...>
    <g>
      ...
    </g>
  </svg>
</ng-template>

If we now combine the NgTemplateOutlet approach with our Cloud Universe SVG view model, the SVG representation looks like the following:

"Detailed subject view? Every constellation is empty." You are right, but you may have noticed while reading that the Cloud Universe Subjects have been omitted from the SVG model. The reason is that we need a different approach to handle the flood of data. In general, SVGs are used more for representing simple things. The advantage of SVGs is that you can easily handle user interactions because you can subscribe to user events just like normal HTML elements. SVGs are not a good option for displaying a lot of data because the whole SVG is attached to the DOM. This means that we would create a DOM element for each subject, and you can imagine that > 10000 subjects for each constellation would cause a very long loading time, which is a no-go from the user’s point of view.

Cloud Universe Subject details

At this point the question has to be asked: What is the alternative to SVG for displaying more than a few bubbles? We have to consider that there can be a lot of meshUsers, meshProjects, …. Therefore, we need a solution that works even with > 10000 meshUsers without the user noticing any difference in the behavior of the application during navigation. We did some research to get an overview about the options how to visualize big data sets in an attractive and performant way. A common way for visualizing different relationships are network graph. We stumbled upon Sigma JS.

Why Sigma JS?

The basic idea of Sigma JS is to handle large amounts of data. It is easy for the library to display many nodes and relations. I.e. it has been thought about data processing and visualization. There is an option to use WebGL. Compared to Canvas 2D, WebGL is much faster and can handle comparatively larger amounts of data without sacrificing UX.

Necessary Sigma JS tradeoff

So far we have only talked about the positive aspects. Basically, they outweigh. It seems to be the right library for our use case. However, there is one hurdle to using the library within an Angular application: It is a JS library. If we add them directly to the package.json as a dependency, they’ll be bundled into the main bundle. Do we really need Sigma JS during the application start? No. We only need the JS library within the Cloud Universe screen. Therefore it is important to think about a lazy load strategy for Sigma JS.

How do you connect Sigma JS with the CloudUniverseSVG?

Now we have reached the point of connecting the CloudUniverseSVG with Sigma JS. Like we previously mentioned the CloudUniverseSVG contains all necessary information about the available subjects within a specific constellation. This means we need some additional data processing to provide the necessary data source structure for each Sigma JS instance. You have understood correctly. We do not manage one graph for all constellations, we provide each constellation with its own Sigma JS instance. The benefit is that we’ll get a clear separation between each constellation and a better developer experience because we can focus within one Sigma JS only on one specific constellation. Maintainability is always an important topic.

Data provision is one thing. But how are you going to integrate the Sigma JS library into the SVG?

First let us take a look into the necessary information. We need the following information for each subject constellation to build the Sigma JS data source:

• Viewport size
• Node size
• Node keys
• Node color

Node size, node keys and node color sounds like static data sets. Based on the constellation view box and circle we determine the view port size. It is important to expand the view exactly over the subjects constellation SVG. The default of Sigma JS is to create a view port which fits exactly to the nodes. In our case we want to provide the exact size like the underlying subjects constellation SVG. We basically overlay the Sigma JS graph on top of the constellation SVG.

What happens then with the information? How do we get the connection from CloudUniverseSVG to Sigma JS? Isn’t there still a lot of processing to be done?

Yes. Indeed. Therefore we need to talk about another important piece of our whole Cloud Universe.

Web Worker to provide Sigma JS data source

In order to relieve the main thread and to prevent a laggy UX we handle the Sigma JS data source creation within a Web Worker based on view port size, node size, node keys and node color.

The Web Worker handles the following steps:

1. Node frame creation which describes the dimensions of the Sigma JS graph.
2. Every node needs a uniquely identifiable value as node key. Sadly Sigma JS does not assign automatically unique node keys and also not across multiple Sigma JS instances. But how can we achieve this without some random string generation? First we want to build the data source depending on the cached subjects constellation data. If we take a look into the cached subject constellation data then we recognize that we’ve everything in place to provide unique node keys. If we combine HATEOAS with Sigma JS then we see that the unique keys are the self links. A self link is in general a pointer to one specific resource. This means the self link is our uniquely identifiable value. Another advantage is that we need the self link anyway to request the additional node information.
3. Random node position generation depending on the view port size.

Finally, everything is broken down to the Sigma JS data source and returned.

Cloud Universe Subjects Details

If we now pass the Sigma JS data source to Sigma JS then this would look like this for the meshUser:

We would see for each existing meshUser a dot but nothing more. This makes the graph not really meaningful. Now we come to include the Cloud Universe Subject.

We introduced the Cloud Universe Subject domain model at the beginning. So far it has not played a role. Only the Cloud Universe Subject knows the displayName and all relations. This information is missing within the cached Cloud Universe Subject Constellation domain model and therefore we can’t show some valuable labels. It would be nice to fetch the Cloud Universe Subject information for each node during hovering. But how do you implement it?

First we need to look at what is available and what is missing. Available is the self link as node key for each subject. Missing is the displayName and relationship information for each subject. It is not the right way to request all available subjects in advance. This would produce a lot of unnecessary requests and this approach would be really bad in case of > 10000 subjects. Please always remember 🙂 We want to focus on requesting only the subjects that are relevant. The solution is to implement a lazy load strategy to request and cache the Cloud Universe Subject domain model. Therefore we’ve attached to every node a hover event handler. The up coming events will be handled within Sigma JS node reducers. If a node was hovered the first time then we show ‘Loading …’ and we’ll request in the background the information from the backend. This information will be cached. So if you hover again over the same node then we’ll show the cached information.

Cloud Universe Subject Relations

It’s a nice gimmick to hover over all the nodes. But it doesn’t deliver the value we want to create with the Cloud Universe. We want to show the relationships of a meshUser to other subjects in a transparent and easy way. That’s why we introduced another user interaction option besides hovering. Clickable nodes!

The fact is that the hover event always occurs before the click event. This means that we have all the subject information available at the click to visualize the relationships based on it. We’ve decided not to show a separate line for each relationship, because this would overload the view if you have > 10000 relationships. A better approach is to highlight the connection between each relevant constellation and hide all irrelevant subjects. This makes the view clean and very manageable.

At this point, only the meshUser subject information is loaded, all other selected nodes know only their node key. This is also an important path we took to avoid creating unnecessary requests.

Whats next?

The idea of the Cloud Universe is to provide the possibility to add more and more features. For example, we’ve mentioned hover and click handling. But you can imagine that there are other use cases that are important to add value within the manageable range. As a quick sneak peek, we’ve implemented additional interaction options such as zoom in/out, drag and drop, reset and search.

Conclusion

We have been able to collect a lot of positive feedback so far. Especially when it comes to presenting certain relationships transparently. It is sometimes really hard to visualize specific relationships across different domains depending on a bunch of data. The Cloud Universe concept is a good approach to solve the relationship topic in a performant way and it could also fit to other use cases.

Feedback is greatly appreciated. To submit feedback please use meshcloud.canny.io or if you don’t want to use an external tool please reach us at feedback@meshcloud.io .