You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
CVE fetchers should work using a bulk saving context or perform bulk_create #16
Currently, ingesting of the initial 230K of CVEs takes around 25 minutes on a very fast CPU (130-200 CVE/s).
In practice, SQLite can do much more than that (96K inserts/s are easy.).
Because we have a lot of M2M all arounds, it's not trivial by using the "nice" ORM API.
Two solutions:
Using Django vanilla API
The classical way to solve this is to use Model.through which is an automatically generated table composed of id, $containing_id, $contained_id fields which can be used to bulk create the M2M rows.
Therefore, all the fetchers code should be reworked to take lists all the time (single item is just [x]) and return a list of models (not yet saved!) and all of them are bulk created in the call-site.
Topological sort has to be done manually, usually, we do:
Hi @fricklerhandwerk, I’m currently looking into this issue and would like some time to investigate it more deeply, especially to evaluate whether the proposed implementation improves throughput. Would you mind assigning it to me? This will help avoid duplicate efforts and ensure no one else unintentionally starts working on the same issue. Thanks!
Activity
DarshanCode2005 commentedon Mar 31, 2026
Hi @fricklerhandwerk, I’m currently looking into this issue and would like some time to investigate it more deeply, especially to evaluate whether the proposed implementation improves throughput. Would you mind assigning it to me? This will help avoid duplicate efforts and ensure no one else unintentionally starts working on the same issue. Thanks!