Skip to content

CVE fetchers should work using a bulk saving context or perform bulk_create #16

Open

Description

@RaitoBezarius

Currently, ingesting of the initial 230K of CVEs takes around 25 minutes on a very fast CPU (130-200 CVE/s).

In practice, SQLite can do much more than that (96K inserts/s are easy.).

Because we have a lot of M2M all arounds, it's not trivial by using the "nice" ORM API.

Two solutions:

Using Django vanilla API

The classical way to solve this is to use Model.through which is an automatically generated table composed of id, $containing_id, $contained_id fields which can be used to bulk create the M2M rows.

Therefore, all the fetchers code should be reworked to take lists all the time (single item is just [x]) and return a list of models (not yet saved!) and all of them are bulk created in the call-site.

Topological sort has to be done manually, usually, we do:

  • create isolated elements with empty M2M
  • create the M2M relations

Using a bulk saver context

We can also remove all reference to save and use a custom API à la https://gist.github.com/crucialfelix/7fa53265ed11e6761531f1b2e0d1f36a to coalesce any operation we need to.

It's unclear if it would make performance faster as-is.

Activity

  1. added
    datasomething about quality or quantity of ingested data
    and removed on Sep 26, 2024
  2. DarshanCode2005 commented on Mar 31, 2026

    @DarshanCode2005
    Collaborator

    Hi @fricklerhandwerk, I’m currently looking into this issue and would like some time to investigate it more deeply, especially to evaluate whether the proposed implementation improves throughput. Would you mind assigning it to me? This will help avoid duplicate efforts and ensure no one else unintentionally starts working on the same issue. Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Labels

datasomething about quality or quantity of ingested datagood first issueGood for newcomershelp wantedExtra attention is needednice to haveNice to have, but not necessary for the time beingperformance

Projects

No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions