Repository navigation
Add kernel and memory-transfer breakdowns to Nsight trace analysis #2942
Description
Search before asking
- I have searched the Inference issues and found no similar feature requests.
Description
Hi @dkosowski87, following the merge of #2878, I’d like to discuss a follow-up to the trace analyzer.
We now have per-iteration host and GPU-projected timings, scoped NVTX ranges, and GPU operation counts. I’m interested in extending that analysis to show which kernels ran and what memory transfers occurred during the capture.
For a first version, I’m thinking of adding these measurements to analysis.json:
- Kernel names, execution counts, and durations, using the
cuda_gpu_kern_sumreport. - Memory-transfer direction, bytes, and duration, using the
cuda_gpu_tracereport, which includes operation duration, byte count, and source and destination memory types.
I’d keep these separate from the existing range timings. NVIDIA defines a GPU-projected range as spanning the first enclosed GPU operation’s start through the last operation’s end. Summed kernel durations measure something different, particularly when operations overlap.
Use case
When a preprocessing stage looks expensive, the range summary gives us somewhere to investigate, but we still need to inspect the trace or export more reports to understand the work involved. Having kernel and transfer summaries alongside it could help narrow that investigation to computation or data movement.
One detail I’d want to verify is how reliably we can associate those operations with the full NVTX scope. NVIDIA’s nvtx-name option prefixes kernel names with the innermost enclosing range name. That alone wouldn’t preserve our distinction between preprocessing.resize and postprocessing.resize. If reliable correlation needs more work, I’d start with summaries for the capture as a whole.
Additional
I saw your implementation-selection metadata work in #2903, including the reference to separate inference-profiler work. Does this extension fit the direction you have in mind, or is that work already covering it?
For validation, I’d use small CUDA targets with known kernel launches and host/device transfers, plus a CPU-only capture. I’d verify the report formats against Nsight 2025.1.3 and include captured CSV fixtures so the parser tests can run without a GPU.
Are you willing to submit a PR?
- Yes, I’d like to help by submitting a PR!
Activity
PushpakAg commentedon Sep 16, 2026
It sounds like you're looking to add more detailed kernel and memory-transfer information to the analysis. How do you plan to handle potential overlaps in kernel execution durations?
Silas-Asamoah commentedon Sep 18, 2026
Good question. The merged analyzer stores Nsight’s projected range duration as
iterations[].gpu_projection.projected_nsin development/profiling/analysis.py. Its trace analysis guidance also says not to treat that value as a sum of kernel durations.For this proposal, I’d label summed kernel durations as cumulative kernel time, separate from the projected interval. If two 10 ms kernels run concurrently, their cumulative time is 20 ms, while the projected interval is about 10 ms. I’d include an overlapping-launch test to make that distinction clear in the output.
dkosowski87 commentedon Sep 25, 2026
Hi @Silas-Asamoah sorry it took so long to answer.
That's exactly the issue here. we have a separate tool for profiling and optimisation. Those two reports (any many more) are included there. That tool is mostly developed so we have a standardised approach to improving our performance on different inference paths on different devices. For now we are not open sourcing that one, it might be the case in the future.
The profiling skill in this repo was added earlier so people doing small performance improvements (which was most often the case) would provide us with a report to validate their approach.
Could you describe your use case? I wouldn't like to discourage you from contributing so perhaps there is path here that we can move forward with.
Silas-Asamoah commentedon Sep 26, 2026
Thanks for explaining, @dkosowski87. My motivation was to build on #2878 and improve the public inference profiling tools incrementally.
The use case I see is someone running Roboflow Inference locally or in a self-hosted NVIDIA GPU deployment who wants to understand performance in their own workload, whether or not they’re contributing code. They could reproduce the relevant work using the existing profiling target workflow, then inspect kernel activity and memory transfers alongside the timings.
Looking through existing contributions, I found a couple of examples:
.cpu()calls by batching transfers per chunk. Reporting both copy count and bytes would help assess that tradeoff: fewer transfers can involve a different amount of data moved.The dispatch investigation on #2650 also found that many small operations spent far more time in dispatch than in GPU computation. Kernel execution counts and cumulative durations, read alongside host timings, could help flag that pattern for further investigation.
This also seems useful for people using coding agents to investigate performance. The repository’s profiling skills provide a repeatable workflow, and the structured results give both the agent and the user measurements to inspect when considering a change.
I thought collecting this evidence in
analysis.json, with the existing workload and run provenance, would make investigations easier to repeat and share in a performance issue or PR. I started with summaries for the whole capture to keep the first extension small; more precise attribution could follow.I understand your internal profiler already covers these reports. The value I see here is making this evidence easier to collect through the public workflow for developers investigating their own Inference workloads, as well as contributors validating changes. Would that fit the role you see for the public profiling skill?
dkosowski87 commentedon Oct 2, 2026
@Silas-Asamoah let me discuss internally whether the
inference-profilercould be subject to open-source. I need to confirm whether we can make that concrete before asking you to redirect any work. I wouldn't like to maintain overlapping profiling implementations.Silas-Asamoah commentedon Oct 2, 2026
That’s understood @dkosowski87. Thanks for the feedback!