> ## Documentation Index
> Fetch the complete documentation index at: https://docs.specterops.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Troubleshooting

Nemesis has a number of services to help with assist with troubleshooting.

**Note** that Grafana + Jaeger tracing are only available if you use the `--monitoring` flag when launching Nemesis!

## Grafana

Navigating to the **Help** menu reachable in the bottom left of the Nemesis interface and clicking the `/grafana/` route link will take you to the Grafana interface. Clicking the **Metrics** Grafana link on the Help page will take you to the general metrics visualization:

<img src="https://mintcdn.com/specteropsdocs/p-HS7mR7kxSK3hXw/nemesis-docs/docs/images/grafana-metrics.png?fit=max&auto=format&n=p-HS7mR7kxSK3hXw&q=85&s=e51bd221a3015ed6d3975fbb65ef2180" alt="Grafana Metrics" width="3024" height="1716" data-path="nemesis-docs/docs/images/grafana-metrics.png" />

Clicking the **Logs** Grafana link on the Help page will take you to the general logs Loki idexing in Grafana:

<img src="https://mintcdn.com/specteropsdocs/p-HS7mR7kxSK3hXw/nemesis-docs/docs/images/grafana-logging.png?fit=max&auto=format&n=p-HS7mR7kxSK3hXw&q=85&s=79d9b0404e6e0843d9f3a5bde0cc380c" alt="Grafana Logging" width="3024" height="1715" data-path="nemesis-docs/docs/images/grafana-logging.png" />

Filtering by a specific service name will allow you to drill down into the logging for that service, which can easily be searched, for example with the `nemesis-file-enrichment` service:

<img src="https://mintcdn.com/specteropsdocs/p-HS7mR7kxSK3hXw/nemesis-docs/docs/images/grafana-logging-details.png?fit=max&auto=format&n=p-HS7mR7kxSK3hXw&q=85&s=a9e1c94cd2472808511701db04e305e7" alt="Grafana Logging" width="3024" height="1712" data-path="nemesis-docs/docs/images/grafana-logging-details.png" />

Clicking the dashboards link in Grafana will bring you to a few preconfigured dashboards as well:

<img src="https://mintcdn.com/specteropsdocs/p-HS7mR7kxSK3hXw/nemesis-docs/docs/images/grafana-dashboards.png?fit=max&auto=format&n=p-HS7mR7kxSK3hXw&q=85&s=5f081c52515c35dd945a73237f6bce7e" alt="Grafana Dashboard" width="3024" height="1711" data-path="nemesis-docs/docs/images/grafana-dashboards.png" />

## RabbitMQ Dashboard

While the queueing system for Nemesis is swappable with Dapr, Nemesis currently uses RabbitMQ. Navigating to the **Help** menu reachable in the bottom left of the Nemesis interface and clicking the `/rabbitmq/` route link will take you to the RabbitMQ interface. This interface can be used to track message delivery rates/etc.

<img src="https://mintcdn.com/specteropsdocs/OgofQVXE8JwYYgvt/nemesis-docs/docs/images/rabbitmq.png?fit=max&auto=format&n=OgofQVXE8JwYYgvt&q=85&s=cf4e20d9f91cd54637843bf506048127" alt="RabbitMQ Dashboard" width="3024" height="1715" data-path="nemesis-docs/docs/images/rabbitmq.png" />

## Jaeger Tracing

Navigating to the **Help** menu reachable in the bottom left of the Nemesis interface and clicking the `/jaeger/` route link will take you to the Jaeger tracing interface. Reaching Jaeger via this link will filter for the `file-enrichment: /TaskHubSidecarService/StartInstance` trace type by default (the Dapr file\_enrichment workflow trace):

<img src="https://mintcdn.com/specteropsdocs/p-HS7mR7kxSK3hXw/nemesis-docs/docs/images/jaeger-traces.png?fit=max&auto=format&n=p-HS7mR7kxSK3hXw&q=85&s=7af704c34e33e14451d2bb27a33c541b" alt="Jaeger traces" width="3024" height="1715" data-path="nemesis-docs/docs/images/jaeger-traces.png" />

Clicking a trace will give you more information on the trace:

<img src="https://mintcdn.com/specteropsdocs/p-HS7mR7kxSK3hXw/nemesis-docs/docs/images/jaeger-trace-details.png?fit=max&auto=format&n=p-HS7mR7kxSK3hXw&q=85&s=beb58aa64c7f24a9af810a30138b371b" alt="Jaeger trace details" width="3024" height="1714" data-path="nemesis-docs/docs/images/jaeger-trace-details.png" />

This can help track down locations for slowdown or other failures- for example by filtering across services for `error=true`:

<img src="https://mintcdn.com/specteropsdocs/p-HS7mR7kxSK3hXw/nemesis-docs/docs/images/jaeger-trace-error.png?fit=max&auto=format&n=p-HS7mR7kxSK3hXw&q=85&s=2f6df7202a2edb4ca256618b4313e14e" alt="Jaeger trace error" width="3024" height="1299" data-path="nemesis-docs/docs/images/jaeger-trace-error.png" />

## Lazydocker

While [Lazydocker](https://github.com/jesseduffield/lazydocker) is not a Nemesis specific project, we highly recommend it for general troubleshooting when using Docker containers:

<img src="https://mintcdn.com/specteropsdocs/p-HS7mR7kxSK3hXw/nemesis-docs/docs/images/lazydocker.png?fit=max&auto=format&n=p-HS7mR7kxSK3hXw&q=85&s=0a373b699a7988fdc4d26a21d218ce31" alt="Lazydocker" width="3016" height="1745" data-path="nemesis-docs/docs/images/lazydocker.png" />