Idriss Neumann

On l'a fait ! Tunis 🇹🇳 a enfin eu sa journée de conférence orientée pour les développeurs à la cité de la culture le 5 avril.

Comme annoncé dans un précédent blogpost, nous avions monté un très beau stand dans le but de challenger les conférenciers avec un concours IA, serverless et IoT et on a eu beaucoup de participant(e)s.

Félicitons encore nos gagnant(e)s: Zayneb, Ala Eddine et Yassmine¹!

Le code source de la démo est disponible sur github et si vous voulez plus d'explications, vous pouvez visionner cette courte vidéo :

J'ai également eu la chance d'avoir la scène pour parler de Quickwit, Grafana et OpenTelemetry avec une autre démo. Il était prévu de le faire en anglais mais finalement le public a préféré la langue de molière. Je m'excuse pour les personnes qui auraient souhaité le voir en anglais, il y aura d'autres occasions 😅.

Il y aura un replay, les slides et supports sont disponibles sur github également et si vous souhaitez en apprendre davantage, vous pouvez également lire ce blogpost.

J'ai également pu assister à la keynote très inspirante "how do you learn" d'Olivier et Sonyth Huber et vous recommande de visionner le replay lorsqu'il sera publié.

Et pour finir, j'ai également pu faire visiter Sidi Bou Saïd à mon ami speaker Yacine, la plus belle place de la région de Tunis. Yacine qui a également donné un super talk sur comment il a réussi à porter Doom sur navigateur en utilisant WASM, une merveilleuse technologie.

Si vous souhaitez garder le contact, en particulier si vous avez apprécié les démo et le challenge de CWCloud, nous avons un serveur discord communautaire que vous pouvez rejoindre.

Les prochaines conférences auxquelles j'assisterai seront DevoxxFR comme visiteur, SunnyTech et RivieraDev en tant que speaker. J'espère vous y voir nombreux(se)s comme d'habitude 🤩.

Comme vous le savez peut-être déjà avec nos récentes communications, un évenement Fork It aura lieu à la cité de la culture à Tunis 🇹🇳 le 5 avril 2025.

CWCloud aura un stand avec un concours IoT, IA et serverless qui consistera à lire un capteur de température et humidité DHT22 à l'aide d'un Raspberry Pi et de les envoyer à une fonction serverless et lowcode de CWCloud afin qu'elle fasse réagir des LLM avec des emojis pour indiquer s'il fait chaud ou froid. Vous aurez plus d'informations avec cette vidéo :

Il y aura des livres d'Aurélie Vache à gagner :

• Understanding Kubernetes in a visual way
• Understanding Docker in a visual way

Je présenterai aussi un talk à 16h55: Découvrons ensemble la relève de l'observabilité avec les logs et traces : Quickwit (le talk sera en anglais mais vous avez une version disponible en Français à BDX/IO).

Il est important de vous inscrire et de récupérer votre ticket ici. C'est vraiment peu cher pour un évènement technique de cette qualité et nous avons également un code promo qui permet de le faire descendre encore de 20% : COMWORK20.

Afin de vous enregistrer, vous devez cliquer sur "Get Tickets" :

Ensuite vous avez le choix pour payer en ligne soit en TND soit en Euros avec une carte de crédit :

Si vous utilisez tunis.events afin de payer en TND, voici comment ajouter le code promo en cliquant sur "code secret" :

Et si vous utilisez lu.ma afin de régler en Euros, pour utiliser le code promo vous devez cliquer sur "add a coupon" :

On espère vous voir très nombreux à l'évènement !

Vous l'aurez peut être constaté, nous avons changé d'identité visuelle et commencé à séparer les activités. CWCloud deviens un produit à part entière avec ses propres structures juridiques en cours de création (tant que c'est en cours, le produit reste sous la tutelle de la société comwork).

A cette occasion, CWCloud se munit de sa propre landing page et le blog lui a été transféré ici : cwcloud.tech.

Comwork va continuer à exister en tant que boite de service avec son propre site web qui pour rappel est le suivant : comwork.io.

Beaucoup de choses vont changer notamment vous pourrez le constater l'apparition de deux versions : community edition (opensource en licence MIT) et enterprise (propriétaire) avec des fonctionnalités en plus adaptés aux grands groupes. Les versions SaaS quant à elle pour les marchés européens/internationaux et tunisiens vont directement pointer sur des version enterprise.

Nous vous informons également que nous sommes en train de postuler chez YCombinator afin de mieux faire évoluer le produit. Nous vous tiendrons informer de l'évolution.

Bonne année 2025 à toutes et à tous 🎉. Commençons cette nouvelle année avec une rétrospective sur le mouvement DevOps.

Il existe déjà de nombreux articles et billets de blog¹ qui expliquent en détail ce qu’est ce mouvement mais je vais quand même passer rapidement dessus afin d'être sûr nous soyons sur la même longueur d'onde pour le reste de l'article.

Pour faire simple, DevOps est une sorte d'alignement stratégique entre les parties prenantes qui développent un produit et ses fonctionnalités (le build) et celles qui maintiennent la production (le run). On est censé mesurer la bonne application du DevOps par le fait de réussir à briser les frontières (ou silos) qu'il peux exister entre le build et le run dans une entreprise ou organisation.

Depuis un certain temps, le mot DevOps est dévoyé de son sens d’origine, notamment par les recruteurs, afin de désigner directement un ensemble de compétences techniques² parfois utilisées dans sa mise en œuvre. C’est pourquoi on peut lire beaucoup d’évangélistes DevOps qui martèlent que "DevOps n’est pas un rôle, c’est un ensemble de bonnes pratiques pour briser les silos", et ils ont raison d'une certaine manière.

Cependant, en tant que responsable technique souhaitant fournir des outils et des compétences précises, je pense que nous n'avons pas d'autres choix qu'accepter et s’adapter à l'usage du terme d'aujourd’hui. C’est pourquoi je n’ai aucun problème à ajouter le mot DevOps sur des CVs ou des offres d’emploi quand il s’agit de sélectionner des profils dont le rôle correspond davantage à des SRE³ ou des Platform Engineers. C’est pareil pour les outils que nous développons comme CWCloud. Ce qui compte le plus, c’est de répondre aux besoins des clients et utilisateurs et non pas pinailler sur l'origine d'un éthymologique d'un mot qui vient d'un mouvement qui n'adresse plus réellement les problèmes d'échelles rencontrés par les entreprises. Donc, si les clients et recruteurs pensent que DevOps est un ensemble de compétences et pratiques techniques, ce n’est pas un problème fondamentalement grave. Commençons par les approcher parce que nous sommes pertinents pour les aider, plutôt que de les corriger de manière dogmatique et irrévérencieuse.

Pour illustrer davantage le fait qu'il ne sert à rien de lutter contre le sens du courant, voyons par exemple comment GitLab se présente :

GitLab: The most-comprehensive AI-powered DevSecOps platform

Ce qui peux se traduire comme ceci :

GitLab : la plateforme DevSecOps alimentée par l’IA la plus complète

Avant le battage médiatique autour de l’IA, GitLab se définissait pendant des années comme la chaîne d’outils DevOps complète, malgré le fait que ses fonctionnalités (dépôts git, pipelines CI/CD et les fonctionnalités GitOps) n'impliquent pas nécéssairement que l'organisation soit DevOps. Beaucoup d’entreprises qui utilisent GitLab ne suivent pas du tout les principes DevOps. Personnellement, je pense qu’il en va de même pour les personnes capables d’automatiser des déploiements avec des compétences techniques comme ansible, terraform, helm, etc.

Cela étant, revenons au sujet principal de cet article : je pense personnellement que le mouvement DevOps en lui-même est mort et que nous revenons aux silos. C'est un phénomène récurrent qui se produit chaque décennie dans toutes les industries en croissance, et dans le cas de l'IT, ce dernier retour aux silos est la conséquence directe du passage au cloud moderne.

Définissons d’abord ce qu’est le cloud moderne : c’est essentiellement une couche d’abstraction de la complexité des infrastructures via des API ou interfaces simples à consommer pouvant être directement par des product owners, des développeurs, des data scientists... bref, des parties prenantes qui ne sont pas expertes en hébergement d'infrastructure et gestion d'applications en production. Et ces API, avec différents niveaux d’abstraction, sont fournies As a Service⁴.

Le cloud moderne peut être délégué à des hébergeurs ou hyperscalers et c'est ce qu'on appelera le cloud public (fournisseurs comme AWS, GCP, Azure, Scaleway...) ou mis en place dans des infrastructures privées (on parlera donc de cloud privé) via des outils d'IaaS comme OpenStack, OpenShift, Kubernetes, des plateformes FaaS... bref, tout ce qui permet de donner de l’autonomie aux équipes de développement pour le déploiement de leur code.

Et c’est pour cela que nous assistons à un retour des silos :

• des équipes de Platform Engineers qui fournissent les outils pour aider les développeurs à déployer leur code (registries d’images, CI/CD, moteurs serverless, observabilité...)
• des équipes de SRE⁵ qui sont souvent d’anciens développeurs gérant les incidents en production et apportant des solutions à court et long terme, parfois en corrigeant directement le code
• des équipes consommatrices (développeurs, product owners, data scientists...) de la plateforme⁶
• des équipes OPS qui s’occupent de l’infrastructure physique : matériel, réseau, administration système de bas niveau

La seule différence entre le cloud public et le cloud privé est que certains des intervenants de ces silos travaillent directement comme employés de l'hébergeur. Il s’agit d’une mutualisation des ressources humaines dans de grandes organisations qui n’ont jamais réellement adopté le mouvement DevOps d'ailleurs.

Mais du coup, cela ne ressemble t-il pas à ce que nous avions avant l'ère DevOps ? Quelle est la différence ?

La principale différence réside dans le fait que les SLA⁷ et le time to market étaient très mauvais pour plusieurs raisons :

• manque d’agilité dans la planification entre les équipes non-alignées en terme d'objectifs
• certaines personnes étaient des goulets d’étranglement par manque d’automatisation et d’abstraction de leurs interventions
• d’anciens cadres méthodologiques comme ITIL ou CMMI qui géraient tout via l’ITSM⁸

Comme pour les méthodologies agiles avant lui, DevOps était trop axé sur la suppression des silos, ce qui est impossible dans les grandes organisations. Et puisque le but de toute entreprise est de croître, ce n’était pas une solution durable. Une méthodologie non scalable n’est pas durable à long terme.

Alors est-ce vraiment un problème si nous revenons aux anciens silos ? Je ne pense pas. Comme pour Agile (et même ITIL, CMMI, COBIT, DDD, TDD, etc.), nous progressons en piochant les principes qui nous intéressent au moment opportun. Bien sûr, nous continuerons à améliorer l’automatisation, la CI/CD, l’observabilité, nos SLA dans la résolution d’incidents et notre time to market pour les évolutions via l’ingénierie pragmatique, pas en suivant religieusement une méthodologie. Le dogmatisme et le pragmatisme sont souvent opposés, et en tant qu’ingénieurs, nous devrions rester pragmatiques et chercher la meilleure solution avec le meilleur ROI⁹.

Donc encore une fois, bonne année, et espérons que 2025 soit une nouvelle ère d’amélioration de nos pratiques et produits de gestion des déploiements et infrastructures. Nous avons plein de surprises qui arrivent en matière d’observabilité et d’automatisation (peut-être avec de l’IA 😱).

Hi and Merry Christmas 🎄 (again yes, I didn't thought that I was going to publish another blogpost so soon 😄).

In this blogpost we'll see how to use CWCloud and Quickwit to setup beautiful dashboards like this in replacement of Google Analytics:

Before going in detail, let's start to give you a bit of context of what brought us to do this transition.

First, Google Analytics ain't comply with the GDPR¹. So basically it was becoming illegal to continue to use it despite it was an amazing tool to analyze our websites and application usages.

With the last case law, we started to use Matomo as a replacement and we're still providing Matomo as a Service in our CWCloud SaaS. And it worked pretty well (even if I find the UI a bit old-fashion)...

However I didn't like to maintain multiple stacks which, from my perspective, are serving the same purpose: observability. And yes web analytics should be part of it from my perspective.

I already explained why we choosed Quickwit as our observability core stack in previous blogposts:

• Quickwit, the next generation of modern observability
• Quickwit for prometheus metrics

So the idea was to use the same observability stack to track visitors data and index and display those on Grafana. And to be able to achieve this, we needed something very easy to add in our various frontend like a one-pixel image:

<img src="https://api.cwcloud.tech/v1/tracker/img/{mywebsite}" style="display: none;"></img>

As you can see, we provided it as an endpoint in CWCloud to complete the observability features and it's documented here.

This endpoint is writing a log which looks like this:

INFO:root:{"status": "ok", "type": "tracker", "time": "2024-12-20T13:46:23.358233", "host": "82.65.240.115", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 18_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1.1 Mobile/15E148 Safari/604.1", "referrer": "https://www.cwcloud.tech/", "website": "www.cwcloud.tech", "device": "mobile", "browser": "safari", "os": "ios", "details": {"brand": "apple", "type": "iphone"}, "infos": {"status": "ok", "status_code": 200, "city": "Saint-Quentin", "region": "Hauts-de-France", "country": "France", "region_code": "HDF", "country_iso": "FR", "lookup": "FRA", "timezone": "Europe/Paris", "utc_offset": "FR", "currency": "EUR", "asn": "AS12322", "org": "Free SAS", "ip": "xx.xx.xx.xx", "network": "xx.xx.xx.0/24", "version": "IPv4", "hostname": "xx-xx-xx-xx.subs.proxad.net", "loc": "48.8534,2.3488"}, "level": "INFO", "cid": "742b7629-7a26-4bc6-bd2a-3e41bee32517"}

So at the end, it contain a JSON payload we can extract and index:

{
  "status": "ok",
  "type": "tracker",
  "time": "2024-12-20T13:46:23.358233",
  "host": "82.65.240.115",
  "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 18_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/18.1.1 Mobile/15E148 Safari/604.1",
  "referrer": "https://www.cwcloud.tech/",
  "website": "www.cwcloud.tech",
  "device": "mobile",
  "browser": "safari",
  "os": "ios",
  "details": {
    "brand": "apple",
    "type": "iphone"
  },
  "infos": {
    "status": "ok",
    "status_code": 200,
    "city": "Saint-Quentin",
    "region": "Hauts-de-France",
    "country": "France",
    "region_code": "HDF",
    "country_iso": "FR",
    "lookup": "FRA",
    "timezone": "Europe/Paris",
    "utc_offset": "FR",
    "currency": "EUR",
    "asn": "AS12322",
    "org": "Free SAS",
    "ip": "xx.xx.xx.xx",
    "network": "xx.xx.xx.0/24",
    "version": "IPv4",
    "hostname": "xx-xx-xx-xx.subs.proxad.net",
    "loc": "48.8534,2.3488"
  },
  "level": "INFO",
  "cid": "742b7629-7a26-4bc6-bd2a-3e41bee32517"
}

So let's start by creating the Quickwit mapping:

{
  "doc_mapping": {
    "mode": "lenient",
    "field_mappings": [
      {
        "name": "time",
        "type": "datetime",
        "fast": true,
        "fast_precision": "seconds",
        "indexed": true,
        "input_formats": [
          "rfc3339",
          "unix_timestamp"
        ],
        "output_format": "unix_timestamp_nanos",
        "stored": true
      },
      {
        "indexed": true,
        "fast": true,
        "name": "cid",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "indexed": true,
        "fast": true,
        "name": "website",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "indexed": true,
        "fast": true,
        "name": "device",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "indexed": true,
        "fast": true,
        "name": "os",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "indexed": true,
        "fast": true,
        "name": "browser",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "indexed": true,
        "fast": true,
        "name": "host",
        "type": "ip"
      },
      {
        "indexed": true,
        "fast": true,
        "name": "hostname",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "indexed": true,
        "fast": true,
        "name": "user_agent",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "indexed": true,
        "fast": true,
        "name": "referrer",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "indexed": true,
        "fast": true,
        "name": "lookup",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "name": "details",
        "type": "object",
        "field_mappings": [
          {
            "indexed": true,
            "fast": true,
            "name": "brand",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "type",
            "type": "text",
            "tokenizer": "raw"
          }
        ]
      },
      {
        "name": "infos",
        "type": "object",
        "field_mappings": [
          {
            "indexed": true,
            "fast": true,
            "name": "status",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "name": "status_code",
            "fast": true,
            "indexed": true,
            "type": "u64"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "city",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "region",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "country",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "region_code",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "country_iso",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "timezone",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "utc_offset",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "currency",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "asn",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "network",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "ip",
            "type": "ip"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "org",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "version",
            "type": "text",
            "tokenizer": "raw"
          },
          {
            "indexed": true,
            "fast": true,
            "name": "loc",
            "type": "text",
            "tokenizer": "raw"
          }
        ]
      }
    ],
    "timestamp_field": "time",
    "max_num_partitions": 200,
    "index_field_presence": true,
    "store_source": false,
    "tokenizers": []
  },
  "index_id": "analytics-v0.4",
  "search_settings": {
    "default_search_fields": [
      "website",
      "cid",
      "host",
      "referrer",
      "infos.ip",
      "infos.country",
      "infos.country_iso",
      "infos.city",
      "infos.region_code",
      "infos.timezone",
      "infos.currency",
      "infos.version"
    ]
  },
  "version": "0.8"
}

Note: as you can see, we moved the lookup field to the root document in order to be able to use the Geomap plugin of Grafana.

Once it's done, we can use Vector, as usual, to parse this log line with the following remap function:

remap_analytics:
    inputs:
      - "kubernetes_logs"
    type: "remap"
    source: |
      .time, _ = to_unix_timestamp(.timestamp, unit: "nanoseconds")

      .message = string!(.message)
      .message = replace(.message, r'^[^:]*:[^:]*:', "")

      .body, err = parse_json(.message)
      if err != null || is_null(.body) || is_null(.body.cid) || is_null(.body.type) || .body.type != "tracker" {
        abort
      }

      .cid = .body.cid
      .website = .body.website
      .browser = .body.browser
      .device = .body.device
      .os = .body.os
      .host = .body.host
      .referrer = .body.referrer
      .user_agent = .body.user_agent
      .infos = .body.infos
      .details = .body.details

      if is_string(.infos.lookup) {
        .lookup = del(.infos.lookup)
      }

      del(.timestamp)
      del(.body)
      del(.message)
      del(.source_type)

And then the sink²:

sinks:
  analytics:
    type: "http"
    method: "post"
    inputs: ["remap_analytics"]
    encoding:
      codec: "json"
    framing:
      method: "newline_delimited"
    uri: "https://xxxx:yyyyy@quickwit.yourinstance.com:443/api/v1/analytics-v0.4/ingest"

Once it's done you'll be able to do some visualization in Grafana using the Geomap plugin:

Very nice, isn't it?

Have a nice end of year and Merry Christmas 🎄 again!

Hi and Merry Christmas 🎄.

With all the demos we've done lately, some people asks us a way to install CWCloud easily on localhost to give it a try, especially for the serverless part.

Let's start with a quick reminder on what is CWCloud: it's an agnostic deployment accelerator platform which provides the following features:

• DaaS or Deployment as a Service: you can checkout this tutorial to understand how DaaS is working with cwcloud and what's the difference between IaaS, PaaS and DaaS.
• FaaS or Function as a Service: you can checkout this blogpost to understand what is the purpose of this feature
• Observability and monitoring: you can checkout this tutorial

At the time of writing, here's the different component used by CWCloud to run:

• A RESTful API
• A Web GUI¹
• Some asynchronous workers to schedule run the serverless function
• ObjectStorage
• PostgreSQL as relational and JSON database
• Redis for the cache and message queuing
• Flyway DB SQL migrations

It can be seen as a bit heavy but believe me it's not, it can run on a single Raspberry PI!

In order to self-host CWCloud, we provide three ways (the three are relying on docker images):

• Docker compose
• Helmchart for Kubernetes
• Ansible role

But this is not enough to bootstap it in seconds. In this blogpost we will show you how to run CWCloud with our CLI cwc using kind² in order to use some feature which doesn't not depends on the external services like the FaaS or the monitor features.

Just a bit of reminder, here's how to install kind, kubect and helm with brew:

brew install kubectl
brew install helm
brew install kind

Then you can also install our cwc cli using brew³:

brew tap cwc/cwc https://gitlab.comwork.io/oss/cwc/homebrew-cwc.git 
brew install cwc

Once it's done, you can create your cluster with kind:

kind create cluster

And then, simply run the following command:

cwc bootstrap

Then, wait until the pods are Running:

kubectl -n cwcloud get pods

Then you can open port-forward to the API and GUI in order to be able to open the GUI in a web browser:

cwc bootstrap pfw

You'll be able to access the GUI through this URL: localhost:3000

The default user and password are the following:

• Username: sre-devops@comwork.io
• Password: cloud456

Of course if you need to override some helm configurations, you can with this command:

cwc bootstrap --values my-values.yaml

It's might be necessary if you want to configure the DaaS feature which is in a "no operation" mode by default. In order to fully use it, you'll have to follow all those configurations tutorials depending on the cloud provider you want to enable.

And finally if you want to uninstall, here's the command:

cwc bootstrap uninstall

Now I'll let you with this five minutes video tutorial on how to use the FaaS, you can fully reproduce on your local environment:

Enjoy!

In a previous blogpost we explained how we reduced our observability bill using Quickwit thanks to its ability to store the logs and traces using object storage:

We also said that we were using VictoriaMetrics in order to store our metrics but weren't satisfied by it lacks of object storage support.

We always wanted to store all our telemetry, including the metrics, on object storage but weren't convinced by Thanos or Mimir which still rely on Prometheus to work making them very slow.

The thing is for all of cwcloud's metrics, we're using the OpenMetrics format with a /v1/metrics endpoint like most of the modern observable applications following the state of art of observability.

Moreover, all of our relevant metrics are gauges and counter and our need is to set Grafana dashboards and alerts which looks like this:

In fact, we discovered that it's perfectly perfectly feasible to setup the different threshold and do some Grafana visualizations based on simple aggregations (average, sum, min/max, percentiles) using the Quickwit's datasource:

However, if you're used to also search and filter metrics using PromQL in the metrics explorer, you'll have to adapt your habits to use lucene query instead:

As you can see, it's not a big deal ;-p

That been said, in order to scrap and ingest the prometheus/openmetrics http endpoints, we choosed to use vector¹ with this configuration:

sources:
  prom_app_1:
    type: "prometheus_scrape"
    endpoints:
      - "https://api.cwcloud.tech/v1/metrics"

transforms:
  remap_prom_app_1:
    inputs: ["prom_app_1"]
    type: "remap"
    source: |
      if is_null(.tags) {
        .tags = {}
      }

      .tags.source = "prom_app_1"

sinks:
  quickwit_app_1:
    type: "http"
    method: "post"
    inputs: ["remap_prom_app_1"]
    encoding:
      codec: "json"
    framing:
      method: "newline_delimited"
    uri: "http://quickwit-searcher.your_ns.svc.cluster.local:7280/api/v1/prom-metrics-v0.1/ingest"

Note: you cannot transform the payload structure the way you want unlike other sources like kubernetes-logs or docker_logs sources but you can add some tags to add a bit of context. That's what we did in this example adding a source field inside the tags object.

And this is the JSON mapping to be able to match with the vector output sent to the sinks and that will make you able to make aggregations on the numeric values:

{
  "doc_mapping": {
    "mode": "dynamic",
    "field_mappings": [
      {
        "name": "timestamp",
        "type": "datetime",
        "fast": true,
        "fast_precision": "seconds",
        "indexed": true,
        "input_formats": [
          "rfc3339",
          "unix_timestamp"
        ],
        "output_format": "unix_timestamp_nanos",
        "stored": true
      },
      {
        "indexed": true,
        "fast": true,
        "name": "name",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "indexed": true,
        "fast": true,
        "name": "kind",
        "type": "text",
        "tokenizer": "raw"
      },
      {
        "name": "tags",
        "type": "json",
        "fast": true,
        "indexed": true,
        "record": "basic",
        "stored": true,
        "tokenizer": "default"
      },
      {
        "name": "gauge",
        "type": "object",
        "field_mappings": [
          {
            "name": "value",
            "fast": true,
            "indexed": true,
            "type": "f64"
          }
        ]
      },
      {
        "name": "counter",
        "type": "object",
        "field_mappings": [
          {
            "name": "value",
            "fast": true,
            "indexed": true,
            "type": "f64"
          }
        ]
      },
      {
        "name": "aggregated_summary",
        "type": "object",
        "field_mappings": [
          {
            "name": "sum",
            "fast": true,
            "indexed": true,
            "type": "f64"
          },
          {
            "name": "count",
            "fast": true,
            "indexed": true,
            "type": "u64"
          }
        ]
      },
      {
        "name": "aggregated_histogram",
        "type": "object",
        "field_mappings": [
          {
            "name": "sum",
            "fast": true,
            "indexed": true,
            "type": "f64"
          },
          {
            "name": "count",
            "fast": true,
            "indexed": true,
            "type": "u64"
          }
        ]
      }
    ],
    "timestamp_field": "timestamp",
    "max_num_partitions": 200,
    "index_field_presence": true,
    "store_source": false,
    "tokenizers": []
  },
  "index_id": "prom-metrics-v0.1",
  "search_settings": {
    "default_search_fields": [
      "name",
      "kind"
    ]
  },
  "version": "0.8"
}

To conclude, despite the fact that Quickwit isn't a real TSDB² (time-series database), we found it pretty easy with vector to still use it as a metrics backend with vector. And this way we still can say to our developer to rely on the OpenMetrics/Prometheus SDK to expose their metrics routes to scrap. However we're still encouraging some of our customer to use VictoriaMetrics because it's still experimental and some of them need more sophisticated computation capabilities³.

One of the improvements that we immediatly think about, would be to also implement the OpenTelemetry compatibility in order to be able to push metrics through OTLP/grpc protocol. We opened an issue to the quickwit's team to submit this idea but we think that it can be also done using vector as well.

During the last decade, you should have heard about serverless architecture or Function as a Service (or FaaS) many times. But sometimes you might have heard the word "serverless" also for other cloud services such as Database as a Service (or DBaaS) or Container as a Service (or CaaS).

What does those things have in common to get called "serverless"? At the beginning this word implied two conditions that I'll remind in this blogpost to start. Then I'll focus on the FaaS and explain my mind on why I think it has evolved last couple of years.

The first condition is you ain't supposed to know about the infrastructure that hosts the service you're using.

• For a DBaaS, you just get an endpoint to connect your apps with and don't have to worry about the cluster sizing, scaling, hardware capabilities...
• For a CaaS, you just have to tell to a simple API which container image and tag to deploy and don't have to worry about the clustering of your containers orchestrators. The CaaS might be built on top of Kubernetes (or K8S) with knative and the K8S API with the knative's CRD (Custom Resource Definition) can be considered as some sort of serverless API if you don't have to worry about the K8S cluster running behind
• For a FaaS, you just have to implement a function in a supported programing language and don't have to worry about how this function will be built as a microservice¹, exposed as a webservice and trigger with multiple events²

The second condition is the "pay as you go" kind of billing on public cloud: you ain't supposed to pay for dedicated clusters but only for the network, compute³ and storage used during the runtime of your code or transactions.

For example with a serverless database, you should get billed only for the data you'll ingest or fetch and the queries you'll run and not for an entire running cluster. Same with a CaaS or FaaS you should only get billed for the runtime of your containers or the necessary compute and network used during a function's call.

We can give more well known example of serverless offers you might have heard about on big cloud players:

• AWS Lambda the very well known FaaS engine of amazon that has kind of set the developer experience of the FaaS in my opinion
• GCP Cloudrun which is a CaaS built on top of K8S and knative
• GCP Cloud functions the FaaS engine of GCP built on top of Cloudrun⁴
• Azure function the FaaS engine of Microsoft Azure

Moreover, the GCP approach of building everything on top of K8S with knative leads the way for other cloud providers to provide similar experiences. It's the case for Scaleway which is also providing a CaaS and a FaaS built on top of knative.

That been said, I think the key feature of serverless and especially the Function as a Service isn't the "pay as you go" but it's more about adding an abstraction layer with the infrastructure allowing the developers to ship their code more quickly and get focus only on the business logic. That's why there's also FaaS engine you can install on premises such as OpenFaaS or our own cwcloud FaaS engine.

That's also something the industry is looking for decades with tons of tools you might have encounter:

• BPM (Business Process Management)
• ETL (Extract Transform Load)
• CI/CD (Continuous Integration / Continuous Deployment) pipelines orchestrators
• Workflow engine such as Airflow, Temporal, Cadence, Apache Nifi...
• API backend frameworks: Spring, Laravel, FastAPI... to lower the complexity of exposing your code as an API or microservices
• Nocode / Low code
• etc

Those tools are different, meets different needs for different populations of IT workers, for example:

• developers who want to focus only on the business logic and not how to expose this business logic as a service
• data scientists who needs ETL or data pipelines
• electronics engineers and IoT makers who needs to push notifications from their sensor and trigger some treatments on their devices and enjoy to do it with a lowcode editor⁵
• product owners technical enough to use BPM, nocode or lowcode to translate their needs
• system administrators who needs to collect and transform some logs for observability purposes or schedule some tasks
• SRE (System Reliability Engineers) who needs to setup CI/CD pipelines

However they do have something in common: all those tools will generate functions (which are sometimes called "workflow" or "job" or "pipeline" or whatever) that will require some compute capabilities and an orchestrator to trigger and launch it. Moreover, those tools are designed to get rid of the maximum of technical aspect and make the IT workers focus only on the business aspects. Sounds like the promise of the serverless, doesn't it?

Because nowadays most of those tools are still bringing their own compute orchestrator, it might be very expensive for the maintainance. Lots of companies which are recruiting multiple kind of IT workers for their different needs find themselve installing all those solutions in their infrastuctures which requires dozens of SRE to handle this heavy maintainance. I used to work with scale-up asking to install all the tools I mentioned in this blogpost in K8S. It means installing dozens of jobs orchestrator on a job orchestrator (because K8S is also a job and pipeline orchestrator). This is ironic, isn't it?

There's modern tools, mainly in the CI/CD area, which are designed to work on top of K8S in a gitops and serverless way. By that I mean re-using the K8S capabilities to orchestrate ephemeral tasks or even applications. It's the case of knative of course but also Tekton or ArgoWorkflow which are pretty similar tools allowing us to define serverless pipelines or workflows without having to install runners or particular runtime unlike most of the other CI/CD tools.

However, most of the other kind of tools I mentioned earlier will require to install their own orchestrator engine and reserve lot of resources in advance in order to be able to trigger their tasks, and that ain't serverless friendly. It's the case for Talend, Airflow, Cadence, gitlab or github runners, etc... We still have to work with those tools because they've not been completely replaced by FaaS engine even if we can notice that some cloud provider are trying to provide multiple services built on top of it⁶.

That's why, we decided with CWCloud to implement a single FaaS engine which aims to bring several "dev XP (developer experiences) for those different populations of IT workers and which is agnostic from the infrastructure running it⁷.

It's only the beginning but we already provide:

• A code editor supporting the following programing languages: Python, Go, Javascript and even Bash
• A lowcode editor supporting Blockly which is suitable for IoT makers, lowcode developers and product owners

• An API and CLI to be able to templatize the function's creation

Therefore, the created functions can be exposed as:

• HTTPs endpoints like a RESTful API
• Async workers which can be triggered with different kind of event: scheduler, cron expressions, etc...

Finally, you can choose to invoke the function and wait for the result in the http response in a blocking way (we discouraged it but sometimes you ain't got no choice), or set async callbacks. We're supporting the following callbacks:

• HTTP webhook
• MQTT or WSS (websockets) queues which are very suitable for IoT makers as well

This video tutorial might give you an ideo on the current dev XP:

To conclude, I believe that all those tools are the very definition of the "framework" concept for all these IT worker populations, in the sense that it allow them to focus on their business logic. The framework used to allow companies to produce more and faster, involving more people and reusing more resources, which also had the effect of increasing the quality of IT systems. That's why I strongly believe that FaaS is the new generation of modern frameworks.

In this blogpost, I ain't describe in detail what Pulumi is doing. I already talked about many times in previous blogpost but also in IT conference such as DevoxxFR:

Yeah I know it's in French, I'm sorry for non-French speaker¹. Let me give you a bit of context: in this conference, we presented how Pulumi is working to allow people to us to use your favorite programming language to do some IaC and also how we can use it to transform this IaC as a real product with its own API and CLI. We called it Deployment as a Service or DaaS.

And that's why we're using it in our driver system² for cwcloud:

If you want to learn more about it, we also detailed the DaaS concept in this tutorial.

So now we've said that this tool is more suitable for people who want to deliver their IaC as a Service, I'll also try to explain my point of vue of why this tool is better for almost everyone including people who enjoy using Terraform or IaC with declarative languages such as HCL (Hashicorp Configuration Language).

First of all, I think a declarative language such as HCL is kind of a bad compromise for people who ain't working the same way:

• classic system administrators who wants to only configure and avoid implementing any kind of logic
• SRE (System Reliability Engineer) or Platform Engineer who wants to use a turing complete programing language and be able to implement business logic in their IaC

The usual way to solve this is to use a configuration language such as YAML which is easy to read but also templatisable using an engine such as jinja2 (used by Ansible), or go template (used by helm).

However Hashicorp tried to reunite both needs with a single language for all its products including Terraform and it leads to something nobody likes very much, neither the developers nor the system administrators if you want my honest opinion³.

Here's an example of service we'd like to enable or disable with a enable_my_service flag and also manage high availability⁴ with another high_availability flag:

resource "aws_instance" "my_service" {
  count = (var.enable_my_service == true ? (var.high_availability == true ? 3 : 1) : 0)
  ami                         = data.aws_ami.ubuntu.id
  instance_type               = "t2.micro"
  subnet_id                   = aws_subnet.subnet_public.id
  tags                        = merge(local.common_tags)
}

You find this ugly? Wait for this: sometimes when the terraform provider isn't supporting an endpoint from the IaaS API, you have to use an "external" datasource.

Few years ago, that's something I had to do in order to get a PCS (Private Service Connect)⁵ id from GCP (Google Cloud Platform) and inject-it in the elastic provider to make the connection:

data "external" "get_psc_id" {
  program = ["bash", "${path.module}/get_psc_id.sh", var.region, var.gcp_host_project, google_compute_forwarding_rule.psc_consumer[0].name]
  count   = (var.enable_psc == true ? 1 : 0)
}

resource "ec_deployment_traffic_filter" "traffic_filter" {
  name   = "${var.stage}-${var.project}-${var.region}-filter"
  region = "${var.region_prefix}-${var.region}"
  type   = "gcp_private_service_connect_endpoint"
  count  = (var.enable_psc == true ? 1 : 0)

  rule {
    source = data.external.get_psc_id[0].result.pscConnectionId
  }
}

And the program invoked by the external datasource has to be written in another programing language, such as bash in this example:

#!/usr/env/bin bash

set -eu

region="$1"
projet="$2"
name="$3"

jsonOutput="$(gcloud --project="$projet" compute forwarding-rules describe "$name" --format json --region "$region")"
pscId="$(echo "$jsonOutput"|jq -r .pscConnectionId)"
pscIp="$(echo "$jsonOutput"|jq -r .IPAddress)"

echo "{\"pscConnectionId\": \"${pscId}\", \"IPAddress\": \"${pscIp}\"}"

Using Pulumi, I'd be able to parse the output of the gcloud directly in my Python or Go code.

Having a tool reuniting multiple needs might be a good thing but in this case, I'd prefer when this tool is providing several languages. And that's exactly what Pulumi offers.

Indeed you can use Pulumi with your favorite programming language but there's also a simple declarative YAML interface available. Here's what it looks like (example from the pulumi blogpost):

name: yamldemo
runtime: yaml
resources:
  bucket:
    type: aws:s3:Bucket
    properties:
      website:
        indexDocument: index.html
  index.html:
    type: aws:s3:BucketObject
    properties:
      bucket: ${bucket.id}
      content: <h1>Hello, world!</h1>
      contentType: text/html
      acl: public-read
outputs:
  url: http://${bucket.websiteEndpoint}

In my opinion it's a better approach to continue to answer everyone's needs: choose the language you like to work with, including the most known declarative language used for configuration which is YAML nowadays. And even if you don't like YAML, it's pretty easy to produce a YAML file from another format (way more easy than producing code from templates) ;p

That been said, it's hard to catch up after years or decades of cloud players or vendors interfacing their IaaS (Infrastructure as a Service) or SaaS (Software as a Service) with public Terraform providers available on public registries. However at some point it happen: we can remember the hegemony of Puppet before Ansible during years.

In the case of Pulumi, despite the fact they've already done the job for most of the big cloud players, they also made very smart moves like:

• providing tools to convert a terraform provider into pulumi SDK in every supported languages
• you can also include terraform as dependancy directly in your Pulumi code
• you can convert the state of resources earlier created with terraform into pulumi states

This makes it so easy to mitigate until all the providers understand that Pulumi is the way!

Pulumi isn't only an alternative for Terraform and IaC world with classical IaaS resources but it's shaking up the Kubernetes (K8S) world beeing the first solid alternative to helm.

Despite I like it very much and despite the fact that the go template doesn't bothers me at all, I must admin that it's hate by a lot of people who prefer to use kustomize despite the duplication it generate or even prefer to handle K8S manifests themselve with some piece of code...

That's why Pulumi is providing a Kubernetes package and I think it could be the right call for those people because they'll be able to implement deployment logic using the programing language they like instead of a templating engine they despise.

And what is beautiful is that you can also include and re-use public helmcharts coming from public regisries exactly the same way you can re-use Terraform provider inside your Pulumi code.

Here's an example of invoking with values the nginx-ingress helmchart inside a Pulumi Python's code (example from the official documentation):

from pulumi_kubernetes.helm.v3 import Chart, ChartOpts, FetchOpts

nginx_ingress = Chart(
    "nginx-ingress",
    ChartOpts(
        chart="nginx-ingress",
        version="1.24.4",
        fetch_opts=FetchOpts(
            repo="https://charts.helm.sh/stable",
        ),
        values={
            "controller": {
                "metrics": {
                    "enabled": True,
                },
            },
        },
    ),
)

Amazing isn't it?

To conclude, we can see how Pulumi smartly meets a large number of needs in the IaC world: people who prefer to configure, those who prefer to develop, people working with K8S or classical IaaS resources...

Like I said multiple times in my previous blogposts for PaaS (Platform as a Service) or frameworks and it includes IaC tools as well: interoperability, agnosticity and polyglotism are keys to success.

In this blog post, I'll try to explain why we moved from ElasticStack to Quickwit and Grafana and why we choosed it over other solutions.

First, we've been in the observability world for quite some time and have been using ElasticStack for years. I personally used Elasticsearch for more than 10 years and Apache SolR before for logging and observability usecases even before Elasticsearch's birth!

We also succeed to use ElasticStack for IoT (Internet of Things) projects and rebuilt our own images of Kibana and Elasticsearch for ARM32 and ARM64 before Elastic (the company) starts to release official images. We had a lot of fun with it.

However everyone who works with it on premises know that Elastic is a big distributed system which brings everyone lot of struggles such as:

• The log retentions because it's on filesystem and storage on disk is expensive¹
• Like most of highly distributed databases developed in Java, it has a very high footprint, consumes a lot of RAM...
• You have also some issue such as "split brains" when you're dealing with HA (High Availability)

On the other hand, there's SaaS (Software as a Service) observability solutions such as Datadog or Elastic cloud which are saving you the trouble of managing clusters but which are very expansive. And even putting the price aside, most of our customers are required to keep all the data on an infrastructure they own.

That been said, Grafana proposed an alternative which is called Grafana Loki which is storing the data on object storage. The idea of using object storage is great because it's often implementing HA by design on most of the big cloud players and it lower the price a lot. Moreover, even when you're on premises, you often want to only ensure the HA of fewer components, the object storage amongs them.

However we weren't convinced because Loki ain't implemented a real search engine such as Apache Lucene used by both Elasticsearch and SolR. It also appears to be very slow as well with bad feedbacks from the community such as this one.

So we were looking for a solution who combines the advantages of both worlds: an efficient search engine which compensates the slowness brought by the use of the object storage's API.

And yet we discovered Quickwit \o/.

Quickwit is built on top of Tantivy which is similar to Lucene but written in Rust², and also store the indexed data on object storage. That's the main reason making Quickwit better than Loki³ and Elasticsearch in my opinion.

Quickwit is also bringing lot's of integration with the CNCF ecosystem⁴:

• A datasource for Grafana
• OpenTelemetry interoperability for traces and logs ingestion
• Jaeger's GRPC API interoperability which allows us to use Quickwit as a storage backend for traces and keep the Jaeger UI or Jaeger datasource on grafana. This is the only known solution to store Jaeger traces on object storage
• Elasticsearch or Opensearch⁵'s API interoperability
• Falcosidekick which can use Quickwit as an output
• Glasskube which makes easier the Quickwit's installation on Kubernetes⁶

That's why we decided to propose Quickwit as our main observability solution in cwcloud DaaS (Deployment as a Service) platform. You can checkout this tutorial to get more informations.

Moreover, we also started to migrate most of our customers infrastructures to Quickwit instances and recommand to design their new applications with the OpenTelemetry's SDK available in their stack when it's possible or use Vector from datadog which is bringing lot of advantages as well:

• It's very fast and has a very low footprint comparing to some other well-known solutions such as Fluentbit, Logstash and even Filebeat from ElasticStack (probably because it's written in Rust :p ).
• It provides a very powerful VRL (Vector Remap Language) language in order to remap your logs and make-it compliants with some already existing indexes mapping⁷.
• It's working with Kubernetes but also with docker and even logs written on filesystem by legacy applications. And this is very convenient for us because as explained in my previous blog post Docker in production, is it really bad?, we have lot of customer who are using docker in production (through cwcloud's DaaS) instead of Kubernetes.

For most of them as for our own internal use, we have divided the compute consumption at least by 3 while increasing the retention. Larger companies successfuly created astronomical logging service with Quickwit such as Binance with 100PB of stored data.

So now Quickwit is covering our observability needs in terms of logs and traces but we still miss the metrics. For the metrics usecase we're using VictoriaMetrics which is working pretty well but lacks the support of object storage. We know that Quickwit plans to handle this usecase one day with a real TSDB (Time Series Database) which sounds really promising. I'm quite convinced that separating the compute from the storage and propose object storage is now a success key factor for building modern observability solutions.

To conclude, I still think ElasticStack is a great product with a bigger company behind which is providing more advanced features including AI (Artificial Intelligence) capabilities. I might still offer it to some customers who might be interested by some of those features or even using Elasticsearch as a full-text search engine as a dependancy of some applications or microservices (Quickwit isn't the best choice in this case, it's more suitable for observability usecases only).