Troubleshooting

Common issues and their fixes, organized by symptom.

Docker daemon not running

Symptom: Cannot connect to the Docker daemon or docker: command not found

Fix:

macOS/Windows: Start Docker Desktop from the Applications menu. Wait for the whale icon in the system tray to stop animating.
Linux: Start the Docker service:
```
sudo systemctl start docker
```
Codespaces: Docker is pre-installed. If you see this error, the devcontainer may not have finished initializing. Wait 1-2 minutes and try again.

Port already in use

Symptom: Bind for 0.0.0.0:8080 failed: port is already allocated

Fix: Check for existing containers or processes using the port:

# Check for existing workshop containers
docker compose ps

# If containers are running from a previous session, stop them
docker compose down

# If a non-Docker process is using the port
lsof -i :8080
# Kill the process, or change the port mapping in compose.yaml

Services will not start in Codespaces

Symptom: Services fail to start or are not reachable after creating a Codespace.

Fix: The devcontainer’s post-create script takes 2-3 minutes to complete. Check its status:

Open the VS Code terminal
Look for the “Setting up dev container…” notification
Wait for it to finish before running docker compose up

If services are running but not reachable from your browser, check that port forwarding is configured:

Open the Ports tab in VS Code (next to Terminal)
Verify that ports 3000 (Grafana), 8080-8084 (services) are listed
If a port is missing, click Add Port and enter the port number
Set visibility to Public if you need to share links

Forwarded port shows 404 or a blank page in your browser (Codespaces)

Symptom: You open a forwarded port – most often Grafana on 3000 – and the browser shows a 404 Not Found page or a blank page, even though the Ports tab lists the port as forwarded and the container is healthy.

This is almost never the Codespace. The service is running; the problem is how your browser reaches the Codespaces port-forwarding domain. Work through these in order.

Confirm the service itself is up from the Codespaces terminal (this bypasses the browser entirely):
```
curl -sI http://localhost:3000/    # Grafana; expect HTTP/1.1 200 OK
```
If you get 200, the service is fine and the issue is browser/tunnel-side – continue below.
Try a different browser (Chrome or Edge). If the URL loads there but not in your original browser, the original browser is blocking the forwarding tunnel. The tunnel authenticates with a SameSite=None; Partitioned cookie, which Firefox’s Enhanced/Total Cookie Protection (and some privacy extensions) will block, producing a 404. In Firefox, click the shield icon in the address bar and turn Enhanced Tracking Protection off for *.app.github.dev, allow its cookies, then reload.
Check for a corporate proxy. Corporate networks often allow the editor domain (github.dev) but block the separate forwarded-ports domain (app.github.dev), or route only the browser through a proxy that curl does not use. Check your browser’s own proxy/PAC settings. A Codespace restart rotates the tunnel, so a port that worked earlier can start failing after a restart even though nothing on the server changed.
Most reliable workaround – forward the port to your local machine with the GitHub CLI and open it over localhost (exempt from corporate proxies and third-party-cookie rules). Run this on your local machine, not in the Codespace terminal:
```
# list your codespaces to get the name
gh codespace list
# forward Grafana (repeat/add more for 9090 Prometheus, 3200 Tempo, etc.)
gh codespace ports forward 3000:3000 -c <codespace-name>
```
Then open http://localhost:3000 in any browser.
Do not use the VS Code “Simple Browser” for Grafana. Grafana sends X-Frame-Options: deny, so it will not render inside the Simple Browser’s embedded frame (you get a blank page). Use a real browser tab or the gh codespace ports forward route above.

Grafana shows no data

Symptom: Dashboards are empty, Explore shows no traces or logs.

Fix: This is almost always a timing issue. The observability stack needs 30-60 seconds to initialize after containers start.

Wait 60 seconds after docker compose up completes
Check that the OTel Collector is running and healthy:
```
docker compose logs otel-collector | tail -20
```

Check that Tempo, Prometheus, and Loki are running:

docker compose ps | grep -E "tempo|prometheus|loki"

Generate some traffic so there is data to display:

newman run ../../tests/collections/01-checkout-happy-path.json \
  -e ../../tests/environments/local.json

In Grafana, verify datasources are configured: Settings (gear icon) then Data Sources. You should see Tempo, Prometheus, and Loki.

Code change does not show up after a rebuild

Symptom: You edited a source file, ran docker compose up --build -d <service>, and the new span attribute, log line, or metric still does not appear.

Fix: Work through these in order.

Are you looking at a new trace? Spans already in Tempo are immutable – your change only affects traces created after the rebuild. Run a fresh checkout and open that trace, not the one you were already looking at.
Did the build actually recompile? Docker’s layer cache can decide nothing changed and reuse the previous compiled output – most commonly with the Quarkus/Maven build layer. The container gets recreated, so it looks like the rebuild worked, but it is running the old bytecode. Force a clean build:
```
docker compose build --no-cache <service>
docker compose up -d <service>
```
Note these are two separate commands. --no-cache is a docker compose build flag; passing it to docker compose up fails with unknown flag: --no-cache.
Did you edit the file for your language? Each language track has its own tree (exercises/quarkus/, exercises/python/, exercises/dotnet/). Editing the Python file while running the Quarkus stack changes nothing.
Give metrics time. New metrics (unlike spans) need an export cycle plus a scrape before they appear – up to ~60 seconds on the Quarkus and .NET tracks. Spans show up within seconds.

Newman tests fail

Symptom: connect ECONNREFUSED 127.0.0.1:8080 or test assertions fail.

Fix:

Verify all services are running and healthy:

docker compose ps
# All services should show "Up" and "(healthy)"

If services are starting but not yet healthy, wait 30 seconds. The health checks take time on first startup.

Check service logs for errors:

docker compose logs order-service | tail -20

If using Codespaces, make sure you are using the correct environment file and that port forwarding is active. The local.json environment file assumes localhost – in Codespaces, you may need to use the forwarded URL.

Kafka consumer not receiving events

Symptom: The Notification service is running but no notifications appear in logs or metrics.

Fix:

Check that Kafka is running and the topic exists:

docker compose exec kafka kafka-topics --bootstrap-server localhost:9092 --list
# Should include "order-events"

Check the consumer group status:

docker compose exec kafka kafka-consumer-groups \
  --bootstrap-server localhost:9092 \
  --group notification-service-v2 \
  --describe

Check the Notification service logs:

docker compose logs notification-service | tail -30

If the consumer group shows no members, restart the Notification service:
```
docker compose restart notification-service
```

Traces do not appear in Tempo

Symptom: Grafana Explore shows no traces, even after generating traffic.

Fix:

Wait 30 seconds. The OTel Collector batches spans before exporting (default batch timeout is 1 second, but Tempo needs time to index).

Check the Collector is receiving and exporting spans:

docker compose logs otel-collector | grep -i "span"

Check the Collector’s metrics for dropped spans:

curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted_spans

Verify the Tempo datasource in Grafana points to http://tempo:3200.

Structured logs missing domain fields

Symptom: Log lines in Loki do not contain order.id, customer.id, or customer.tier.

Fix: The domain fields only appear when the code uses DomainContext (or equivalent) to scope them. Check that:

The use case wraps its logic in a DomainContext block:
- Quarkus: try (var ctx = DomainContext.open(...)) in the saga or use case
- Python: with DomainContext(...) in the saga or use case
- C#: using var domainContext = new DomainContext(...) in the saga or use case
The logging framework is configured for structured JSON output. Check the service’s configuration:
- Quarkus: quarkus.log.console.json=true in application.properties
- Python: structlog is configured with JSONRenderer in otel_setup.py
- C#: builder.Logging.AddOpenTelemetry() in Program.cs

Metrics show NaN or no data

Symptom: Dashboard panels show “No data” or NaN values.

Fix: Metrics need traffic to produce data points. A single checkout is often not enough for rate-based queries.

Generate sustained traffic:

# Run the test collection multiple times
for i in {1..5}; do
  newman run ../../tests/collections/01-checkout-happy-path.json \
    -e ../../tests/environments/local.json
done

Wait 30-60 seconds for Prometheus to scrape the new data points.

Check that Prometheus is scraping the Collector:

curl -s http://localhost:9090/api/v1/targets | jq '.data.activeTargets[].health'

If using rate() or histogram_quantile() in a query, you need at least two data points – which means two scrape intervals (default 15 seconds each).

Container runs out of memory

Symptom: A container exits with code 137 (OOM killed).

Fix: The workshop stack requires approximately 8 GB of RAM. If you are running on a constrained machine:

Increase Docker’s memory allocation:
- Docker Desktop: Settings then Resources then Memory, set to at least 8 GB
Stop other Docker containers that are not part of the workshop:
```
docker ps  # check for unrelated containers
```
If still constrained, reduce the memory limits in compose.yaml. The infrastructure containers (Tempo, Loki) are the largest consumers.