2 posts tagged with "elasticsearch"

Using quickwit as a web search engine in your website

January 15, 2026 · 11 min read

founder cwcloud.tech

Happy new year 2026 🎉 . Let's hope this year will be full of professional and personal success for everyone.

Twelve years ago, a long time before the creation of CWCloud, we launched uprodit.com, a social network dedicated to Tunisian freelancers.

The platform was developed in Java, using the Spring framework, Tomcat, and Vert.x later. One of our main ambitions was to build an intelligent search engine capable of leveraging the information users freely entered in their profiles. At the time, achieving this level of flexibility and relevance was extremely challenging with traditional relational databases RDBMS¹.

That's why it was my second experience with Elasticsearch in production (before the product focused on observability area) after few years with Apache SolR and it worked like a charm.

After the launch of CWCloud, uprodit migrated from physical servers to Scaleway instances using CWCloud. At the same time, we started modernizing the backend by adopting container-based architectures. However, the cost of maintaining an Elasticsearch cluster gradually became too expensive for the organization. Here was the technical architecture of the project at that time:

uprodit-old-arch

One year ago, we started working with Quickwit² and replaced our entire observability and monitoring stack with it. This included logs, traces, Prometheus metrics, and even web analytics.

Although Quickwit wasn't originally designed to serve as a web search engine, our experience with metrics and analytics showed that it could effectively meet our needs while significantly reducing infrastructure costs (because of the less amount of RAM consumption and the fact that the indexed data are stored on object storage).

As you may know, since we mentioned it in a previous blog post, Quickwit provides an _elastic endpoint that is interoperable with the Elasticsearch and OpenSearch APIs. The idea, therefore, was to replace Elasticsearch while keeping the existing Vert.x Java code with minimal changes, like this:

uprodit-new-arch

In other words, we wanted to keep the Elasticsearch client library to build the queries dynamically with the Elasticsearch DSL³. In this blog post, we'll detail the few key point to succeed the migration.

Keeping the Elasticsearch client library

As we mentioned before we wanted to have the less code rewriting possible, so we choose to keep the Elasticsearch client library to benefit from the DSL builder. Here's the maven dependancy (in the pom.xml file):

<dependency>
    <groupId>co.elastic.clients</groupId>
    <artifactId>elasticsearch-java</artifactId>
    <version>9.2.4</version>
</dependency>

Creating the Quickwit client

We created a IQuickwitService interface because we may have multiple implementations using other http client later:

package tn.prodit.network.se.utils.singleton.quickwit;

import io.vertx.core.json.JsonObject;

import java.io.Serializable;

public interface IQuickwitService {
  <T extends Serializable> void index(String index, String id, T document);

  <T extends Serializable> void delete(String index, String id);

  String search(String indeix, JsonObject query);
}

Then the default implementation using both Vert.x asynchronous http client for the write methods (index and delete) and default java http client for the search method.

import io.vertx.core.Vertx;
import io.vertx.core.http.HttpClient;
import io.vertx.core.http.HttpClientOptions;
import io.vertx.core.http.HttpClientResponse;
import io.vertx.core.http.HttpMethod;
import io.vertx.core.json.JsonObject;

import tn.prodit.network.se.utils.singleton.IPropertyReader;
import tn.prodit.network.se.utils.singleton.quickwit.IQuickwitService;
import tn.prodit.network.se.utils.singleton.quickwit.QuickwitTimeout;

import jakarta.inject.Inject;
import jakarta.inject.Singleton;

import java.net.URI;
import java.net.http.HttpRequest;
import java.net.http.HttpResponse;
import java.nio.charset.StandardCharsets;
import java.time.Duration;
import java.util.Arrays;
import java.util.Base64;

import static org.apache.commons.lang3.StringUtils.isBlank;

@Singleton
public class QuickwitService implements IQuickwitService {
  private static final Logger LOGGER = LogManager.getLogger(QuickwitService.class);

  @Inject
  private IPropertyReader reader;

  @Inject
  private Vertx vertx;

  private HttpClient client;

  private java.net.http.HttpClient syncClient;

  private String basicAuth;

  private QuickwitTimeout timeout;

  private String uri;

  private QuickwitTimeout getTimeout() {
    if (null == this.timeout) {
      this.timeout = QuickwitTimeout.load(reader);
    }

    return this.timeout;
  }

  public String getUri() {
    if (null == this.uri) {
      String host = reader.getQuietly(APP_CONFIG_FILE, QW_KEY_HOST);
      Integer port = reader.getIntQuietly(APP_CONFIG_FILE, QW_KEY_PORT);
      String scheme = reader.getBoolQuietly(APP_CONFIG_FILE, QW_KEY_TLS) ? "https" : "http";
      this.uri = String.format("%s://%s:%s", scheme, host, port);
    }
    return this.uri;
  }

  private java.net.http.HttpClient getSyncClient() {
    if (null == this.syncClient) {
      QuickwitTimeout timeout = getTimeout();
      this.syncClient = java.net.http.HttpClient.newBuilder()
          .connectTimeout(Duration.ofMillis(timeout.getConnectTimeout()))
          .followRedirects(java.net.http.HttpClient.Redirect.NORMAL).build();
    }
    return this.syncClient;
  }

  private HttpClient getClient() {
    if (null == this.client) {
      String host = reader.getQuietly(APP_CONFIG_FILE, QW_KEY_HOST);
      Integer port = reader.getIntQuietly(APP_CONFIG_FILE, QW_KEY_PORT);
      Boolean tls = reader.getBoolQuietly(APP_CONFIG_FILE, QW_KEY_TLS);
      QuickwitTimeout timeout = getTimeout();

      LOGGER.info(String.format(
          "[quickwit][getClient] Creating http client to quickwit: host=%s, port=%s, tls=%s",
          host,
          port,
          tls));
      this.client = vertx.createHttpClient(new HttpClientOptions().setDefaultHost(host)
          .setDefaultPort(port).setSsl(tls).setConnectTimeout(timeout.getConnectTimeout())
          .setReadIdleTimeout(timeout.getReadTimeoutInSec()));
    }
    return this.client;
  }

  private String getBasicAuth() {
    if (isBlank(this.basicAuth)) {
      String user = reader.getQuietly(APP_CONFIG_FILE, QW_KEY_USER);
      String password = reader.getQuietly(APP_CONFIG_FILE, QW_KEY_PASSWORD);
      LOGGER.debug(
          String.format("[quickwit][getBasicAuth] Creating basic auth header for user=%s", user));
      this.basicAuth = String.format("Basic %s",
          Base64.getEncoder().encodeToString(
              String.format("%s:%s", user, password).getBytes(StandardCharsets.UTF_8)));
    }

    return this.basicAuth;
  }

  @Override
  public <T extends Serializable> void index(String index, String id, T document) {
    String url = String.format("/api/v1/%s/ingest", index);
    getClient().request(HttpMethod.POST, url, ar -> {
      if (ar.failed()) {
        LOGGER.error("[quickwit][index] Request creation failure: " + ar.cause().getMessage(),
            ar.cause());
        return;
      }

      JsonObject payload = JsonObject.mapFrom(document);
      ar.result().putHeader("Content-Type", "application/json")
          .putHeader("Accept", "application/json").putHeader("Authorization", getBasicAuth())
          .send(payload.toBuffer(), resp -> {
            if (resp.failed()) {
              LOGGER.error("[quickwit][index] Request sending failure: " + ar.cause().getMessage(),
                  ar.cause());
              return;
            }

            HttpClientResponse response = resp.result();
            response.bodyHandler(body -> {
              String logMessage = String.format(
                  "[quickwit][index] response url = %s, code = %s, body = %s, payload = %s",
                  url,
                  response.statusCode(),
                  body.toString(),
                  payload);
              if (response.statusCode() < 200 || response.statusCode() >= 400) {
                LOGGER.error(logMessage);
              } else {
                LOGGER.debug(logMessage);
              }
            });
          });
    });
  }

  @Override
  public <T extends Serializable> void delete(String index, String id) {
    String url = String.format("/api/v1/%s/delete-tasks", index);
    getClient().request(HttpMethod.POST, url, ar -> {
      if (ar.failed()) {
        LOGGER.error("[quickwit][delete] Request creation failure: " + ar.cause().getMessage(),
            ar.cause());
        return;
      }

      JsonObject payload = new JsonObject();
      payload.put("query", "id:" + id);
      payload.put("search_fields", Arrays.asList("id"));
      ar.result().putHeader("Content-Type", "application/json")
          .putHeader("Accept", "application/json").putHeader("Authorization", getBasicAuth())
          .send(payload.toBuffer(), resp -> {
            if (resp.failed()) {
              LOGGER.error("[quickwit][delete] Request sending failure: " + ar.cause().getMessage(),
                  ar.cause());
              return;
            }

            HttpClientResponse response = resp.result();
            response.bodyHandler(body -> {
              String logMessage = String.format(
                  "[quickwit][delete] response url = %s, code = %s, body = %s, payload = %s",
                  url,
                  response.statusCode(),
                  body.toString(),
                  payload);

              if (response.statusCode() < 200 || response.statusCode() >= 400) {
                LOGGER.error(logMessage);
              } else {
                LOGGER.debug(logMessage);
              }
            });
          });
    });
  }

  @Override
  public String search(String index, JsonObject query) {
    String path = String.format("/api/v1/_elastic/%s/_search", index);
    HttpRequest request =
        HttpRequest.newBuilder().timeout(Duration.ofMillis(getTimeout().getReadTimeout()))
            .uri(URI.create(String.format("%s%s", getUri(), path)))
            .header("Content-Type", "application/json").header("Accept", "application/json")
            .header("Authorization", getBasicAuth())
            .POST(HttpRequest.BodyPublishers.ofString(query.toString())).build();
    try {
      HttpResponse<String> response =
          getSyncClient().send(request, HttpResponse.BodyHandlers.ofString());
      return response.body();
    } catch (IOException | InterruptedException e) {
      LOGGER.error(String.format("[quickwit][search] Unexpected exception e.type = %s, e.msg = %s",
          e.getClass().getSimpleName(),
          e.getMessage()));
      return null;
    }
  }
}

Adding an abstraction layer

To make a smooth transition, we created two interfaces with an Elasticsearch implementation then replacing by a Quickwit implementation using dependancy injection⁴.

Indexing interface

For the the writing part, here's the interface had two implementations (an Elasticsearch's one and a Quickwit's one).

import tn.prodit.network.se.indexation.adapter.IndexedDocAdapater;

import java.io.IOException;
import java.io.Serializable;

public interface IIndexer {
  <T extends Serializable> void index(IndexedDocAdapater<T> adapter, String id, T document)
      throws IOException;

  <T extends Serializable> void delete(IndexedDocAdapater<T> adapter, String id) throws IOException;
}

And here's the Quickwit implementation:

package tn.prodit.network.se.utils.singleton.indexer.impl;

import tn.prodit.network.se.indexation.adapter.IndexedDocAdapater;
import tn.prodit.network.se.pojo.IQuickwitSerializable;
import tn.prodit.network.se.utils.singleton.indexer.IIndexer;
import tn.prodit.network.se.utils.singleton.quickwit.IQuickwitService;

import jakarta.inject.Inject;
import jakarta.inject.Named;
import jakarta.inject.Singleton;

@Singleton
@Named("quickwitIndexer")
public class QuickwitIndexer implements IIndexer {
  @Inject
  private IQuickwitService quickwit;

  @Override
  public <T extends Serializable> void index(IndexedDocAdapater<T> adapter, String id, T document)
      throws IOException {
    quickwit.index(adapter.getIndexFullName(),
        id,
        document instanceof IQuickwitSerializable ? ((IQuickwitSerializable<?>) document).to()
            : document);
  }

  @Override
  public <T extends Serializable> void delete(IndexedDocAdapater<T> adapter, String id)
      throws IOException {
    quickwit.delete(adapter.getIndexFullName(), id);
  }
}

And the implementation using the quickwit client:

import tn.prodit.network.se.indexation.adapter.IndexedDocAdapater;
import tn.prodit.network.se.pojo.IQuickwitSerializable;
import tn.prodit.network.se.utils.singleton.indexer.IIndexer;
import tn.prodit.network.se.utils.singleton.quickwit.IQuickwitService;

import jakarta.inject.Inject;
import jakarta.inject.Named;
import jakarta.inject.Singleton;

@Singleton
@Named("quickwitIndexer")
public class QuickwitIndexer implements IIndexer {
  @Inject
  private IQuickwitService quickwit;

  @Override
  public <T extends Serializable> void index(IndexedDocAdapater<T> adapter, String id, T document)
      throws IOException {
    quickwit.index(adapter.getIndexFullName(),
        id,
        document instanceof IQuickwitSerializable ? ((IQuickwitSerializable<?>) document).to()
            : document);
  }

  @Override
  public <T extends Serializable> void delete(IndexedDocAdapater<T> adapter, String id)
      throws IOException {
    quickwit.delete(adapter.getIndexFullName(), id);
  }
}

You can observe that we introduced the IQuickwitSerializable interface for value objects (VO)⁵ because some types are not fully supported by Vert.x and therefore cannot be serialized directly. For example, lists of nested objects are not properly handled.

Let’s imagine we have a List<SkillVO> skills, which is a nested list of objects. To address this limitation, we had to duplicate the data in the index mapping using two array<text> fields: one containing only the searchable subfields (such as name), and another preserving the full object structure so it can be unmarshalled correctly and remain interoperable.

{
    "indexed": true,
    "fast": true,
    "name": "searchableSkills",
    "type": "array<text>",
    "tokenizer": "raw"
},
{
    "indexed": false,
    "name": "skills",
    "type": "array<text>"
}

Then, on the original VO class, we had to implement the to() method of the IQuickwitSerializable interface like this:

class PersonalProfileVO implements IQuickwitSerializable<QuickwitPersonalProfile> {
    private List<SkillVO> skills;

    public List<SkillVO> getSkills() {
        return this.skills;
    }

    public void setSkills(List<SkillVO> skills) {
        this.skills = skills;
    }

    @Override
    public QuickwitPersonalProfile to() {
      QuickwitPersonalProfile dest = new QuickwitPersonalProfile();

      if (null != this.getSkills()) {
        dest.setSkills(this.getSkills().stream()
          .map(s -> JSONUtils.objectTojsonQuietly(s, SkillVO.class)).collect(Collectors.toList()));
        
        // we will perform the search query only on the name
        dest.setSearchableSkills(
          this.getSkills().stream().map(s -> s.getName()).collect(Collectors.toList()));
    }

    return dest;
  }
}

Then duplicate the VO class and implements the to() the other way arround:

class QuickwitPersonalProfile implements IQuickwitSerializable<PersonalProfileVO> {
    private List<String> skills;

    private List<String> searchableSkills;

    public List<String> getSkills() {
        return this.skills;
    }

    public void setSkills(List<String> skills) {
        this.skills = skills;
    }

    public List<String> getSearchableSkills() {
        return this.searchableSkills;
    }

    public void setSearchableSkills(List<String> searchableSkills) {
        this.searchableSkills = searchableSkills;
    }

    @Override
    public PersonalProfileVO to() {
      PersonalProfileVO dest = new PersonalProfileVO();

      if (null != this.getSkills()) {
        dest.setSkills(this.getSkills().stream()
          .map(s -> JSONUtils.json2objectQuietly(s, SkillVO.class)).collect(Collectors.toList()));
    }

    return dest;
  }
}

Reading interface

For the reading part, here's the interface which also had two implementations (an Elasticsearch's one and a Quickwit's one).

import co.elastic.clients.elasticsearch._types.SortOrder;
import co.elastic.clients.elasticsearch._types.query_dsl.BoolQuery;
import co.elastic.clients.elasticsearch.core.SearchRequest;
import co.elastic.clients.json.jackson.JacksonJsonpGenerator;
import co.elastic.clients.json.jackson.JacksonJsonpMapper;
import com.fasterxml.jackson.core.JsonFactory;
import jakarta.json.stream.JsonGenerator;
import tn.prodit.network.se.pojo.SearchResult;
import tn.prodit.network.se.pojo.fields.IFields;

public interface ISearcher {
  JacksonJsonpMapper mapper = new JacksonJsonpMapper();

  <T extends Serializable> SearchResult<T> search(String index,
                                                  BoolQuery query,
                                                  Optional<Sort> sort,
                                                  Integer startIndex,
                                                  Integer maxResults,
                                                  Class<T> clazz);

  <T extends IFields> SearchResult<String> search(String index,
                                                  List<String> fields,
                                                  List<String> criteria,
                                                  Integer startIndex,
                                                  Integer maxResults,
                                                  Class<T> clazz);

  static String toJson(SearchRequest request, JacksonJsonpMapper mapper) {
    JsonFactory factory = new JsonFactory();
    StringWriter jsonObjectWriter = new StringWriter();
    try {
      JsonGenerator generator =
          new JacksonJsonpGenerator(factory.createGenerator(jsonObjectWriter));
      request.serialize(generator, mapper);
      generator.close();
      return jsonObjectWriter.toString();
    } catch (IOException e) {
      return null;
    }
  }

  default JacksonJsonpMapper getMapper() {
    return mapper;
  }

  default String toJson(BoolQuery query) {
    return toJson(query, getMapper());
  }
}

Notes:

you can override the default method getMapper() with an already instanciated mapper
the toJson() method will allow the Quickwit implementation to transform the SearchRequest object into a String containing json that will be sent to the _elastic endpoint

Then, the implementation for Quickwit were pretty straight forward:

import co.elastic.clients.elasticsearch._types.query_dsl.BoolQuery;
import co.elastic.clients.elasticsearch.core.SearchRequest;
import io.vertx.core.json.JsonArray;
import io.vertx.core.json.JsonObject;

import tn.prodit.network.se.pojo.IQuickwitSerializable;
import tn.prodit.network.se.pojo.SearchResult;
import tn.prodit.network.se.pojo.fields.IFields;
import tn.prodit.network.se.utils.singleton.quickwit.IQuickwitService;
import tn.prodit.network.se.utils.singleton.searcher.ISearcher;
import tn.prodit.network.se.utils.singleton.searcher.Sort;

import jakarta.inject.Inject;
import jakarta.inject.Named;
import jakarta.inject.Singleton;

import static org.apache.commons.lang3.StringUtils.isBlank;
import static tn.prodit.network.se.utils.CommonUtils.initSearchResult;
import static tn.prodit.network.se.utils.JSONUtils.json2objectQuietly;

@Singleton
@Named("quickwitSearcher")
public class QuickwitSearcher implements ISearcher {
  private static final Logger LOGGER = LogManager.getLogger(QuickwitSearcher.class);

  @Inject
  private IQuickwitService quickwit;

  private JsonObject performSearch(String index,
                                   BoolQuery query,
                                   Optional<Sort> sort,
                                   Integer startIndex,
                                   Integer maxResults) {
    SearchRequest r = getSearchRequest(index, sort, startIndex, maxResults, query);
    LOGGER.info(String.format("[searcher][quickwit][performSearch] index = %s, request = %s",
        index,
        toJson(r)));
    JsonObject jsonQuery = new JsonObject(toJson(r));
    String quickwitResponse = quickwit.search(index, jsonQuery);
    LOGGER.debug(
        String.format("[searcher][quickwit][performSearch] response body: %s", quickwitResponse));
    if (isBlank(quickwitResponse)) {
      LOGGER.warn("[quickwit][performSearch] blank response...");
      quickwitResponse = "{\"hits\": {\"hits\": []}}";
    }

    return new JsonObject(quickwitResponse);
  }

  @Override
  public <T extends Serializable> SearchResult<T> search(String index,
                                                         BoolQuery query,
                                                         Optional<Sort> sort,
                                                         Integer startIndex,
                                                         Integer maxResults,
                                                         Class<T> clazz) {
    SearchResult<T> result = initSearchResult(Long.valueOf(startIndex), Long.valueOf(maxResults));
    List<T> elems = new ArrayList<>();
    JsonObject quickwitResponse = performSearch(index, query, sort, startIndex, maxResults);
    JsonObject hits = quickwitResponse.getJsonObject("hits");
    if (null == hits) {
      LOGGER.warn(
          String.format("[quickwit][search][obj] hits are empty: response = %s", quickwitResponse));
      return result;
    }

    JsonArray hits2 = hits.getJsonArray("hits");
    for (int i = 0; i < hits2.size(); i++) {
      if (IQuickwitSerializable.class.isAssignableFrom(clazz)) {
        JsonObject hit = hits2.getJsonObject(i);
        if (null == hit) {
          LOGGER.warn(String.format("[quickwit][search][obj] hit is empty: hits2 = %s", hits2));
          continue;
        }

        JsonObject source = hit.getJsonObject("_source");
        if (null == source || isBlank(source.toString())) {
          LOGGER.warn(String.format("[quickwit][search][obj] source is empty: hit = %s", hits));
          continue;
        }

        IQuickwitSerializable<?> obj =
            (IQuickwitSerializable<?>) json2objectQuietly(source.toString(),
                IQuickwitSerializable.getSerializationClazz(clazz));
        elems.add((T) obj.to());
      }
    }

    result.setResults(elems);
    result.setTotalResults(Long.valueOf(elems.size()));
    return result;
  }
}

Few difficulties to address

Here are some of the challenges we encountered:

Quickwit ain't replace documents by ID in case of updates, so queries must explicitly filter and select the most recent document version.
Document deletion is not as immediate as in Elasticsearch, meaning deleted profiles may still appear in search results for a short period of time.
Quickwit ain't support regex queries, which required us to rewrite those queries using term or match queries instead.

Conclusion

Given the difficulties previously mentioned, it's understandable that many applications may argue that Quickwit ain't the best solution for this use case. However, in the context of uprodit, search result relevance ain't need to be perfect considering the significant infrastructure cost savings we achieved with this migration.

Relational Database Management System ↩
Before it was acquired by datadog ↩
Domain Specific Language ↩
The prodit-se module was already written in Vert.x using Google Juice and JSR330 (javax.inject package) ↩
Value Object ↩

Quickwit, the next generation of modern observability

September 4, 2024 · 6 min read

Idriss Neumann

founder cwcloud.tech

In this blog post, I'll try to explain why we moved from ElasticStack to Quickwit and Grafana and why we choosed it over other solutions.

First, we've been in the observability world for quite some time and have been using ElasticStack for years. I personally used Elasticsearch for more than 10 years and Apache SolR before for logging and observability usecases even before Elasticsearch's birth!

We also succeed to use ElasticStack for IoT (Internet of Things) projects and rebuilt our own images of Kibana and Elasticsearch for ARM32 and ARM64 before Elastic (the company) starts to release official images. We had a lot of fun with it.

rpi-elastic

However everyone who works with it on premises know that Elastic is a big distributed system which brings everyone lot of struggles such as:

The log retentions because it's on filesystem and storage on disk is expensive¹
Like most of highly distributed databases developed in Java, it has a very high footprint, consumes a lot of RAM...
You have also some issue such as "split brains" when you're dealing with HA (High Availability)

On the other hand, there's SaaS (Software as a Service) observability solutions such as Datadog or Elastic cloud which are saving you the trouble of managing clusters but which are very expansive. And even putting the price aside, most of our customers are required to keep all the data on an infrastructure they own.

That been said, Grafana proposed an alternative which is called Grafana Loki which is storing the data on object storage. The idea of using object storage is great because it's often implementing HA by design on most of the big cloud players and it lower the price a lot. Moreover, even when you're on premises, you often want to only ensure the HA of fewer components, the object storage amongs them.

However we weren't convinced because Loki ain't implemented a real search engine such as Apache Lucene used by both Elasticsearch and SolR. It also appears to be very slow as well with bad feedbacks from the community such as this one.

So we were looking for a solution who combines the advantages of both worlds: an efficient search engine which compensates the slowness brought by the use of the object storage's API.

And yet we discovered Quickwit \o/.

quickwit-gui

Quickwit is built on top of Tantivy which is similar to Lucene but written in Rust², and also store the indexed data on object storage. That's the main reason making Quickwit better than Loki³ and Elasticsearch in my opinion.

Quickwit is also bringing lot's of integration with the CNCF ecosystem⁴:

A datasource for Grafana
OpenTelemetry interoperability for traces and logs ingestion
Jaeger's GRPC API interoperability which allows us to use Quickwit as a storage backend for traces and keep the Jaeger UI or Jaeger datasource on grafana. This is the only known solution to store Jaeger traces on object storage
Elasticsearch or Opensearch⁵'s API interoperability
Falcosidekick which can use Quickwit as an output
Glasskube which makes easier the Quickwit's installation on Kubernetes⁶

quickwit-gui

That's why we decided to propose Quickwit as our main observability solution in cwcloud DaaS (Deployment as a Service) platform. You can checkout this tutorial to get more informations.

quickwit-cwcloud

Moreover, we also started to migrate most of our customers infrastructures to Quickwit instances and recommand to design their new applications with the OpenTelemetry's SDK available in their stack when it's possible or use Vector from datadog which is bringing lot of advantages as well:

It's very fast and has a very low footprint comparing to some other well-known solutions such as Fluentbit, Logstash and even Filebeat from ElasticStack (probably because it's written in Rust :p ).
It provides a very powerful VRL (Vector Remap Language) language in order to remap your logs and make-it compliants with some already existing indexes mapping⁷.
It's working with Kubernetes but also with docker and even logs written on filesystem by legacy applications. And this is very convenient for us because as explained in my previous blog post Docker in production, is it really bad?, we have lot of customer who are using docker in production (through cwcloud's DaaS) instead of Kubernetes.

For most of them as for our own internal use, we have divided the compute consumption at least by 3 while increasing the retention. Larger companies successfuly created astronomical logging service with Quickwit such as Binance with 100PB of stored data.

So now Quickwit is covering our observability needs in terms of logs and traces but we still miss the metrics. For the metrics usecase we're using VictoriaMetrics which is working pretty well but lacks the support of object storage. We know that Quickwit plans to handle this usecase one day with a real TSDB (Time Series Database) which sounds really promising. I'm quite convinced that separating the compute from the storage and propose object storage is now a success key factor for building modern observability solutions.

To conclude, I still think ElasticStack is a great product with a bigger company behind which is providing more advanced features including AI (Artificial Intelligence) capabilities. I might still offer it to some customers who might be interested by some of those features or even using Elasticsearch as a full-text search engine as a dependancy of some applications or microservices (Quickwit isn't the best choice in this case, it's more suitable for observability usecases only).

We know that Elasticsearch is providing object storage compatibility with the searchable snapshot feature but it's not available in the opensource version on one hand, and only recommanded on cold data which are not supposed to be fetch too much on the other hand. ↩
Tantivy is 2x faster than Lucene according to this benchmark, this compensate the slowness brought by the use of the object storage. ↩
Quickwit also provides this benchmark with Loki, trying to make a fair comparison. ↩
I'm involved myself to contribute to lot of them, missioned by Quickwit Inc. (the company). ↩
OpenSearch is a fork of ElasticStack initiated by Amazon AWS. ↩
I wrote a blog post directly on the Quickwit's blog if you want to get more informations. ↩
You see an example of remap function in order to make the docker logs compliant with the default otel-logs-v0_7 index in this tutorial. ↩

Keeping the Elasticsearch client library​

Creating the Quickwit client​

Adding an abstraction layer​

Indexing interface​

Reading interface​

Few difficulties to address​

Conclusion​

Footnotes​