Raspberry Pi Network Monitor: Free Dashboard for Home Internet Traffic

Raspberry Pi Network Monitor: Free Dashboard for Home Internet Traffic

traffic monitoring software with prometheus and grafana

When trying to figure out why the internet is slow, it can be hard to learn exactly which device on the network is eating up all the bandwidth. Many solutions to this problem require software to be installed on every device to be monitored. Instead, I tried to build a custom Raspberry Pi network monitor.

This post will show you how to monitor all internet traffic for every device on your network, without buying any specialty hardware.

This open-source solution has been used by readers of this site for monitoring family internet usage, LAN parties, and more.

This project came about when retrofitting our cabin in the woods to a smart-home. There internet providers out here advertise about 30 Mbps (down) and 2 Mbps (up). This isn’t much to work with. Plus, we have many home-made IOT devices scattered around the house. So… who knows where the traffic is going?

My goal was to not require any special software, yet monitor the internet traffic for every network device.

Ideally, I also wanted to create a beautiful Grafana dashboard. This would let me see what sites the devices were contacting with the Raspberry Pi home network monitor.

It seemed like a couple others on Reddit were also interested in a solution to this problem, so I decided to give it a try. The result was an open-source Python script / Docker container, meant to be run on a Raspberry Pi, that exports data to Prometheus. While I used a Raspberry Pi, the code should run on any Linux distribution. There are Docker images for both arm and amd.

Skip to the end of this post for source code & installation.

But first, let’s consider the different ways to monitor traffic on a home network…

Internet Traffic Monitor: Approaches

Based upon experience and some research, these are the possibilities I came up with:

  1. Pi as a router 
    The obvious way to monitor network traffic. The Raspberry Pi sits between the devices to be tracked and the internet (e.g., acting as a router or access point). Unfortunately, this can slow down the network, which causes many to avoid the approach (see the next section).
  2. Router reporting 
    Some modern routers provide features along these lines. But generally custom firmware is required.
  3. Device reporting 
    The standard protocol for this is SNMP, which will rely upon device side installations to self-report. It integrates well with Prometheus/Grafana though.
  4. Packet sniffing
    You could theoretically monitor the wireless traffic (if all you care about is WiFi). This is the same concept that allows attackers to sniff traffic on a WiFi network.

Each of these has its drawbacks. I did not want to buy a new router, so router reporting was not an option. I could not install the necessary software on all the IOT devices, which prevents device reporting. And packet sniffing is an interesting idea, but I wanted to be able to handle wired as well as wireless traffic.

This left only one approach: a Raspberry Pi network monitor.

Raspberry Pi Home Network Monitor

If all internet traffic is going to pass through a device, it is good to use caution.

The first concern is that of security. I won’t say too much about that here, except to mention that a firewall of some kind is a good idea. I went with Uncomplicated Fire Wall (ufw) because it is, well, uncomplicated.

A less obvious concern is that of speed. When traffic passes through a router/switch, the primary bottleneck is the ethernet hardware. In other words, the CPU and RAM are not as important as in other cases. This was something of a problem with the Raspberry Pi 3B (and lower). However, the Raspberry Pi model 4 has an upgraded on-board 1000 Mbps eth0 port.

Make sure that the ethernet hardware meets the needs.

Failure to do so could slow down the entire network!

With that in mind, here is the exact list of parts I used.

If you're new to Raspberry Pi, the popular CanaKits are a great place to start. I prefer to buy the Raspberry Pi 4, power adapter, micro SD cards, and heatsinks separately. Not only is this cheaper, but it reduces e-waste.

The following are affiliate links to other parts I used in this project. I never link to a product that I have not personally used.

With the parts in-hand, I drew up this Raspberry Pi 4 router design:

raspberry pi network monitor traffic diagram
This router to switch connection diagram shows how we monitor network traffic.

If you’re using the above parts list…

Some important points:

  • The WiFi router is in Bridge mode. This means that eth0 must act as the DHCP server (assign IP addresses to the network).
  • Traffic between devices on the network will not flow through the Raspberry Pi. See the Performance Tests, below.
  • This means that the Pi is only a bottleneck for internet traffic. With 1000 Mbps hardware and an ISP that only provides 30 Mbps, we won’t be hitting this limit any time soon.

There are many ways to set up the eth1 <> eth0 connection. You could configure this using internet bonding software. This would let you add another internet connection (eth2) to make the internet connection even faster. For a more complete DIY Raspbery Pi router solution:

Alternate Design

If you’re not using the Raspberry Pi as a router, this section is for you.

The traffic must flow through the device.

You must have two network interfaces over which the traffic you wish to capture passes.

You could flip the WiFi router and Rasbperry Pi from the above diagram. This approach can work better if you prefer to not use the Raspberry Pi as a router:

  • The WiFi router connects to the modem/internet (not in bridge mode).
  • The Raspberry Pi connects to the internet through the WiFi router.
  • The Raspberry Pi should have a static IP assigned by your WiFi router (see its documentation).

However, it does have one major disadvantage: the WiFi traffic (going to the router) will not be monitored. But the major advantage is: if you ever want to remove the Raspberry Pi network monitor, just plug the WiFi router directly in to the switch.

You could also run a separate DHCP server on the WAN side of the Raspberry Pi. In this case, again, the Pi is not the router. However, if the two network interfaces are bridges, then the traffic is flowing through the Pi.

No matter the design, the device acting as the router connects to the internet, and the device connected to the switch is in bridge mode. In other words, you must manually bridge the two interfaces on the Raspberry Pi. Therefore, the Pi’s eth0 is able to see traffic passing in and out of the LAN.

Performance Tests

After implementing the Pi as a router, I saw no decrease in speed for intra-network traffic. This was tested with iperf3. It showed:

  • ~910 Mbits/sec for two computers connected via a physical switch.
  • ~180 Mbits/sec when separated by a long WiFi hop.
Our ISP only advertises 30 Mbps! We never saw speeds above that before implementing the Pi as a router.

Shockingly, I saw improved external (internet) speeds with the Raspberry Pi network monitor. I already had Node RED running a speedtest every 5 minutes and recording the data to Home Assistant + Prometheus. When using the CenturyLink provided DSL router, I rarely saw speeds above 25 Mbps (down). Now we consistently seeing speeds in the ~33 Mbps range. This is likely because the Raspberry Pi is using pppoeconf to establish the DSL connection directly, and it does a better job managing this connection than the modem provided by the ISP.

Accuracy Tests

Now, to test the accuracy of the Raspberry Pi network monitor.

Using Prometheus for throughput/bandwidth will not be perfectly accurate on a short time scale. This is due to the way a rate is averaged over an interval. However, by downloading a large file, I was able to compare the reported download speed from Chrome with that of the traffic graph:

raspberry pi network monitor dashboard with Grafana
The ~2.4 MB/s reported by Chrome matches the rate reported by Grafana, albeit with some reporting-time lag.

In addition, the total download size matched that reported by Chrome:

Raspberry Pi network monitor dashboard downloads by server
Note: this graph is computed via an increase on a counter. If the monitoring software gets reset, the values will not be accurate. If anybody knows a better way to do this with PromQL, please do let me know.

Screenshots, Installation, & Source Code

This project is open-source. It is available as a Python script or Docker image.

Find the code & detailed documentation for the Raspberry Pi Network Monitor in the network-traffic-metrics Github repository.

The most important part of the configuration is setting up the tcpdump filters. For example, the following will restrict the captured traffic to that which flows in or out of the 192.168.0.0/24 subnet:

(src net 192.168.0.0/24 and not dst net 192.168.0.0/24) or (dst net 192.168.0.0/24 and not src net 192.168.0.0/24)

For more help setting up the filters, check out this blog post by Daniel Miessler on isolating traffic with tcpdump. If you’re having trouble with anything else, please comment below or contact me directly.

Step-by-Step + Configuration File

I’ll try to give copy and paste instructions below. However, as I mentioned in the “Using the Pi as a Router” section, some of these steps may be highly individual. The bridge steps, in particular, can depend on the exact Linux version you have installed. I’ll assume you…

  • Have eth0 (LAN) and eth1 (WAN) on your device.
  • Pi will reside at 192.168.0.1, either as the router or as a pass-through (alternate design).
  • IP addresses will be handed out on the 192.168.0.0/24 subnet.
  • You want to use Google’s DNS servers (8.8.8.8 and 8.8.4.4)
  • Running Raspbian Buster.

1. Build or Configure the Router

If you wist to use the Raspberry Pi as the router (first option), please see this article. If you wish for the WiFi router to connect to the internet (alternate design), follows its instruction manual to assign the Pi a static IP address (192.168.0.1 in this example).

2. Run the Raspberry Pi network monitor script

If you’re comfortable with it, a Docker/Kubernetes install may be easier than these manual steps. Otherwise…

sudo apt-get install git python3-pip tcpdump
sudo pip3 install argparse prometheus_client
git clone https://github.com/zaneclaes/network-traffic-metrics.git
cd ./network-traffic-metrics
sudo python3 ./network-traffic-metrics.py "(src net 192.168.0.0/24 and not dst net 192.168.0.0/24) or (dst net 192.168.0.0/24 and not src net 192.168.0.0/24)"

Open your web browser to http://192.168.0.1:8000/metrics to see the counters being exported for Prometheus. Verify that you’re seeing data that seems to match the traffic on your network. It may be hard to read, but upon refreshing the page you should see more lines added as people connect to different sites.

You can change several options by passing certain command-line flags to the script. For example, the script assumes that you want to listen to the eth0 interface. If you aliased this to the name lan, per the Raspberry Pi router guide, you could add the --interface lan flag. Or, the --port 80 flag would change from listening on port 8000 to port 80. Note that all the configuration variables may also be set via environment variables, like NTM_INTERFACE.

To make the script start on reboot, type sudo crontab -e and add:

@reboot python3 /home/pi/network-traffic-metrics/network-traffic-metrics.py "(src net 192.168.0.0/24 and not dst net 192.168.0.0/24) or (dst net 192.168.0.0/24 and not src net 192.168.0.0/24)" &

Alternatively, you could create a systemd service (generally preferred).

3. Install Prometheus

Prometheus and Grafana can be run anywhere on the same network. I’d recommend not running them on the same device doing the metric exporting. This prevents slowing down the machine. The following instructions are copied more or less exactly from the official Prometheus docs:

Download the latest release of Prometheus for your platform, then extract and run it:

tar xvfz prometheus-*.tar.gz
cd prometheus-*

Create prometheus.yaml (see comments):

global:
  scrape_interval:     15s # How frequently to report
  external_labels:
    monitor: 'network-traffic-metrics'

scrape_configs:
  - job_name: 'network-traffic-metrics'
    static_configs:
      - targets: ['192.168.0.1:8000'] # The Network Traffic Metrics IP/port

Run Prometheus: ./prometheus --config.file=prometheus.yml

Check that you can access Prometheus: localhost:9090/metrics (or wherever it is located).

4. Install Grafana

Again, the official docs are a good place to start (these are copied fairly directly):

sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
wget -q -O - https://packages.grafana.com/gpg.key | sudo apt-key add -
sudo apt-get update
sudo apt-get install grafana-enterprise

Open up http://localhost:3000/ to find Grafana. Add Prometheus as a data source:

  1. Open the side menu by clicking the Grafana icon in the top header.
  2. In the side menu under the Dashboards link you should find a link named Data Sources.
  3. Click the + Add data source button in the top header.
  4. Select Prometheus from the Type dropdown.
Internet traffic monitor data in Prometheus and Grafana
Verifying that Grafana can see the Prometheus data.

The default options should match your installation from above. If you’ve used containers or otherwise installed Prometheus differently, you will need to use the appropriate URL for the Prometheus server. For example, I used the URL http://prometheus-server with a Kubernetes helm deployment. If your data source is configured correctly, you should now be able to use the Explore section to see the data in Prometheus.

You could fiddle with this yourself and create a dashboard to your liking. Or, you could use my Raspberry Pi network monitor dashboard. Grafana has instructions for importing dashboards. In short, you use the + button on the left to Import dashboard 12619 (or the JSON file).

Import the internet traffic monitor dashboard.
Importing the dashboard from GrafanaLabs.

You should now have a working Raspberry Pi network monitor that can be accessed from Grafana. The only thing left to do is make sure you have the filters set correctly. When you open the dashboard, at the top of the screen, are the settings which configure what data are shown:

  • LocalIPs: the IP addresses on your LAN to show.
  • Services: e.g., http, https
  • Protos: e.g., tcp, udp
  • ExcludedServers: regex for servers (outside your network) to hide.

You should verify that your local IPs show up in the dropdown, and that you have not excluded any data you might want to be visualizing. For example, I intentionally filter speedtests and similar such traffic, as they add noise to the data:

Filtering traffic in the grafana dashboard

If your IP addresses are not showing up there, it is likely the case that the IP addresses on your local network do not conform to local subnet ranges. To fix this, open up the dashboard settings and look at the LocalIP variable’s regex. This regex filters all possible values, showing you the values which meet the regex at the bottom of the screen. You’ll need to modify the regex so that the IP addresses on your network show up in the “Preview of Values:”

Editing grafana dashboard network traffic filters.
The regex determines which values are considered “local” IP addresses.

Build Guides

Looking for even more detail?

Drop your email in the form below and you'll receive links to the individual build-guides and projects on this site, as well as updates with the newest projects.

... but this site has no paywalls. If you do choose to sign up for this mailing list I promise I'll keep the content worth your time.

Written by
(zane) / Technically Wizardry
Join the discussion

50 comments
  • Thank you for the tutorial!

    Can you please let me know what is wrong?
    However, I encounter following errors while running this step

    2.Run the Raspberry Pi network monitor script.
    pi@raspberrypi:~/Downloads/network-traffic-metrics $ sudo python ./network-traffic-metrics.py (src net 192.168.0.0/24 and not dst net 192.168.0.0/24)
    bash: syntax error near unexpected token `(‘

    My python is version 3.7
    pi@raspberrypi:~/Downloads/network-traffic-metrics $ python –version
    Python 3.7.0

    I have installed both argparse and prometheus_client

    pi@raspberrypi:~/Downloads/network-traffic-metrics $ pip3 install prometheus_client
    Looking in indexes: [link to pypi.org], [link to www.piwheels.org]
    Requirement already satisfied: prometheus_client in /home/pi/.local/lib/python3.7/site-packages (0.8.0)

    pi@raspberrypi:~/Downloads/network-traffic-metrics $ pip3 install argparse
    Looking in indexes: [link to pypi.org], [link to www.piwheels.org]
    Requirement already satisfied: argparse in /home/pi/.local/lib/python3.7/site-packages (1.4.0)

    • Hey, glad you enjoyed it! And sorry, this was my mistake — looks like the example got garbled when I formatted the code. You need quotation marks around the filters argument, so that the command is: sudo python ./network-traffic-metrics.py "(src net 192.168.0.0/24 and not dst net 192.168.0.0/24)". The post should be updated with the fix now, as well.

      In case you’re curious, it’s because the entire filter clause is a single argument into the python script. Without the quotes, bash is trying to parse the arguments itself before passing them to the script, which it does not know how to do.

      Hope that helps!

      • Thanks for the quick respnse! I understand what could have been the issue. However, I am hitting another error. Sorry. I’m really bad at regex, so no idea what’s wrong here.

        pi@raspberrypi:~/Downloads/network-traffic-metrics $ sudo python ./network-traffic-metrics.py “(src net 192.168.0.0/24 and not dst net 192.168.0.0/24)”
        File “./network-traffic-metrics.py”, line 29
        return f'(?P{pattern})’
        ^
        SyntaxError: invalid syntax

        • Ah. I suspect that you have both python2 and python3 installed. To check, try python --version. Whatever it says is the default Python version on your machine.

          If it is <=3, as I suspect, first make such that which python3 works. You should discover the location to the actual python3 executable. If that exists, the easiest “fix” is to replace sudo python ... with sudo python3 .... Another approach would be to make an alias from /usr/bin/python to python3 so that v3 becomes your default python environment. Or you could use a python version management tool. Which approach is right for you depends on your circumstance, but the first is probably the least hassle.

          • Thanks!
            Yes! Indeed. My python installations seems to be a mess. I will fix that first. Thanks again.

      • Thanks for the quick reply. However, I am getting another error now. Sorry.

        pi@raspberrypi:~/Downloads/network-traffic-metrics $ sudo python ./network-traffic-metrics.py “(src net 192.168.0.0/24 and not dst net 192.168.0.0/24)”
        File “./network-traffic-metrics.py”, line 29
        return f'(?P{pattern})’
        ^
        SyntaxError: invalid syntax

  • Nice guide. I’m trying a slightly different approach – using a docker for graphana and another for Prometheus.
    With that in mind, when creating the dashboard in the last step, the JSON file has the “DS_PROMETHEUS” that wont work. Is there any way to make this work with this docker setup? What do I have to change?

    Thanks

    • Thanks for saying so! FWIW, I do have one container for Grafana and one for Prometheus, just like you. The difference may be that I use Kubernetes to deploy the containers, not the Docker agent. The problem you’re experiencing suggests that you do not have Prometheus configured as a “Data Source” in Grafana, or that somehow the name of that datasource does not match the convention (DS_PROMETHEUS).

      Others have had small hiccups importing the dashboard as well. You prompted me to do a little research and add the dashboard to the GrafanaLabs shared dashboards website. You can follow the import instructions with the GUID 12619. Hopefully that works better! First time I’ve shared a Grafana dashboard publicly 🙂

  • Hi, this is probably perfect for what I’m looking for. The instructions seem good too. One question, I would probably do this for the non-wifi devices so keep my router as the internet connection. That has a 192.168.1.1 ip address and i’d like to keep the other devices on the same subnet. Does the pi HAVE to be 192.168.0.1 or could it be 192.168.1.2 meaning all other devices which have static IP addresses would not have to change?

    • Glad to hear it 🙂 In that case, what you describe is actually preferable. The subnet I used was for example purpose. If your Pi is behind the router, it’s easiest to keep it on the same subnet. To do so, you just want to bridge the traffic between the two interfaces without adding a DHCP server or anything of that nature. This links might help:
      Bridging eth0 and eth1. If that doesn’t work or you need more help lmk.

  • Hello and thank you so much for the guide! I’m running into a problem running the monitor script, and I’d appreciate your help.

    I’ve confirmed that pip3 installed prometheus_client, but when I run the script straight out of the box, I get [code]ModuleNotFoundError: No module named ‘prometheus_client'[/code] I tried to fix this by adding sys.path.append() on line 2 to the path pip3 gave for prometheus_client. That cleared up that error, but gave me this one:

    File “/usr/lib/python3.7/subprocess.py”, line 1522, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: ‘tcpdump’: ‘tcpdump’

    • Hey Mike,

      It sounds like you’re having Python environment problems. You may have multiple versions of Python3 installed and/or some strange symlinks, given that python3 couldn’t find the package installed by pip3. Specifically, it sounds like Python3 is not referencing the same import paths as pip3 is installing to. If forcing an absolute path worked on the import, great, though I do tend to worry this may be causing other problems down the line.

      The second problem suggests that you don’t have tcpdump installed on the machine, or that Python3 cannot find it. The latter would match with your prior problem. Namely, that the shell environment from which Python3 is being run is not resolving your user paths. For example, on my RPi, which tcpdump gives /usr/sbin/tcpdump. If that command also works for you, it suggests that you’re invoking Python3 in such a way that this requirement is not resolved in the same way it is from your shell. You could just edit line 71 of the script to invoke the absolute path to tcpdump, just like you did with the prior problem, but again I worry that your shell environment may continue to cause problems.

      Cheers! Hope that helps a little…
      – Z

      • Thanks for your help!

        For the Python issue, I tried defining PYTHONPATH in my .bashrc, but that didn’t help. So, for now I stuck with the edit on your script.

        The tcpdump issue was simply that it wasn’t installed. Easy fix.

        After I got those resolved, I started the script and received an error that eth0 couldn’t be found. Following the router guide, I had given eth0 the alias ‘lan’ and eth1 ‘wan’. So I updated the one instance of eth0 and now it seems to be running happily.

        “(src net 192.168.1.0/24 and not dst net 192.168.1.0/24) or (dst net 192.168.1.0/24 and not src net 192.168.1.0/24)”
        tcpdump: listening on lan, link-type EN10MB (Ethernet), capture size 262144 bytes
        [SKIP] 06:33:43.603149 IP

  • I love this post – this is EXACTLY what I have been looking for, for like six months.
    I’ve even converted one of my old machines to pfsense just to do this, but had to scrap it because it was a big waste, even as VM.

    Question: Would this conflict with PiHole? It is my DHCP provider at the moment.

    Also a problem, I am running this:
    sudo python3 ./network-traffic-metrics.py “(src net 192.168.1.0/24 and not dst net 192.168.1.0/24) or (dst net 192.168.1.0/24 and not src net 192.168.1.0/24)”
    and receiving this error “ModuleNotFoundError: No module named ‘prometheus_client'”
    I’ve tried installing pyEnv to fix this but no dice.

    Do you happen to have a docker container or yml with everything running? That would be much easier for people.

    • Hey! Thanks for saying so.

      I don’t see any reason this would conflict with PiHole. I run AdGuard Home myself these days, but I used to run PiHole. You should be able to substitute the DHCP server for PiHole’s.

      Your problem with prometheus_client being missing is likely because you need to use sudo pip3 install. If you install without the sudo, but then try to run sudo python3, you’ll be using a different environment.

      Please refer to the section of the README devoted to deploying via Docker. There’s also a section devoted to Kubernetes based deployment, which is what I use personally.

        • This is working now – The graphs are empty though in graphana.
          The question I have is why is it not doing a passthrough from eth0 (WAN) to eth1 (LAN)?
          I don’t think this was part of this guide. Do I need to manually bridge the interfaces?

          • You have already mentioned that you are not using the “Pi as a Router” guide (no DHCP server). Therefore, technically you fall under the “Alternate Design” header in the tutorial (which indicates you must bridge the two interfaces). This would also explain the lack of data in Grafana. FWIW, I’d recommend taking the time to stop at the Prometheus step and perform the recommended validations, and then go “Explore” the data in Grafana, so you understand what is actually going on.

            Note that in the Alt design section I mainly referred to a Wifi router. You need to reason about your network and ensure that the traffic you wish to capture is flowing through your Pi, one way or another. That’s the key. As long as that is true, then the script will capture the data across those two network interfaces. There are lots of ways to do this, and without analyzing a network diagram of your setup it is hard to be perfectly accurate in my descriptions =/

          • No worries you put me on the right track after all – I am still new to Prometheus and grafana so I’ll play around with them.
            Yes – regarding the network setup I’ve already done that and everything should be flowing from the PI once I am done with the bridging.
            VDSL Router (No Wifi) => PI => Switch => Rest of the network.
            So nothing will connect to the internet without hitting the PI first.

            Was unable to reply to your last comment for some reason so I put this here.

            Many thanks for your support so far.

  • HI,

    Could you please expand on how to set up the alternate design? It seems like wifi traffic cannot be captured in this design.

      • What kind of firewalls rules did you put in place at the pi?
        I’m concerned about the security using pi as the router.

        • In the “Pi as a Router” guide which this page links to, there is section dedicated to firewalls. tl;dr: I use firewalld with zone-based routing. Start with a “secure by default” mentality and only open the ports you need and you’ll be fine.

          If you really don’t want to go that route, there is one other option, though it requires more hardware. Use another physical router upstream of the Pi. But keep your WiFi router in bridge mode, downstream of the Pi.

          • Hi! This may sound very very amateur, but I am extremely new to coding and program. I see a lot of helpful information online and I see the examples in the gray boxes. My question is where do you put those codes? What program are you using and how do you make those codes work on your computer? I get the process but I don’t get those at first initial step I’m very new. I use macOS and have a raspberry pi 4. I’m stuck at this very initial step.

          • Hello, this tutorial requires at least a beginner understanding of the “shell” (or “terminal” or “command prompt”). Every computer has one. I would recommend you follow some tutorials from the official Raspberry Pi website first and make sure you understand concepts like “SSH” and “sudo.” If you’re not at least familiar with these ideas, there is a good chance you will end up breaking something trying to follow upper-beginner tutorials like these.

          • Hello! I know about shell, Sudo, Linux, and I know perfecting SSH. My specialty is the command line. I just don’t know the first basics about coding. I’m taking a few online courses and they have been helping. I will be there soon. I have my raspberry pi-4 and speedify, I want to be able to monitor everything on my network- (like Lil Snitch without the GUI drivel). My question is what program or text editor are you plugging those commands in on. The ones in the boxes. Is it something like sublime or atom atom? Or is it something via the command line and terminal. That is my basic question and I appreciate the follow through because you are the only one I found online who still gives a damn!

            everything I mean at work that’s why I chose is the tutorialis my thing command line is my thing I just don’t know what program those boxes are referring to is it a tech text editor like sublime or Adam’s at something else.

          • Hello? Thank you for the follow through I really appreciate it. I do know the command line, SSH, and also the command prompt. Coding is my next hurdle and I plan to conquer that by the end of summer. I’m already in rolled in online classes and they’re actually paying off. I’ve seen these boxes all over the Internet and they seem helpful. Yet, has a single one site has ever said where they paste those codes. Is it into a text editor like Atom or Sublime? Or directly into the command prompt? Please help me out I really appreciate it.

      • Hello? Thank you for the follow through I really appreciate it. I do know the command line, SSH, and also the command prompt. Coding is my next hurdle and I plan to conquer that by the end of summer. I’m already in rolled in online classes and they’re actually paying off. I’ve seen these boxes all over the Internet and they seem helpful. Yet, has a single one site has ever said where they paste those codes. Is it into a text editor like Atom or Sublime? Or directly into the command prompt? Please help me out I really appreciate it.

        • The gray boxes in the “Step-by-Step” section are shell commands. I’m assuming you can recognize common commands, like cd, tar, apt-get, etc. The last of these, for example, is how you install software on the Raspberry Pi. When it comes to editing files, it makes absolutely no difference which editor you use (which is why nobody talks about it). Thankfully, your shell will have a default editor built-in, like nano or vi, that lets you edit files from the command line (Google these commands for help).

          If you haven’t made it far enough to understand these concepts, with all due respect, I highly suggest you start with a book or more basic resource than this post. I’m writing for a relatively technical audience, and need to skip over things I think they understand. If you don’t know how to edit a file from the command line, try googling that specific problem first. Then work your way up.

          • Of course I recognize those commands those are classic Linux commands. With all due respect, it’s answers like those that keep people silent and unknowing for so long. No I’m not too familiar with editing a file at the command line. Instead of telling me of what you think I don’t know you could at least provide links or books are helpful to beginners. I went from hopeful to now feeling shamed. “A true wizard teaches their apprentice without forethought …” *said like Tyrion Lannister*

          • This was not my intent at all. I am here responding to you because I wish to help. At this point, I’m honestly confused myself, though. It’s sometimes very hard to tell what someone’s skill level is. If I under-estimated yours, I apologize for sounding condescending by explaining “down” to you. You’re right to say that I could/should have provided you with direct links to resources to learn things. I actually Googled for some for you while I was writing my last comment, but I didn’t have the time to sort through them and try to figure out what would be most useful to you. My second paragraph was not meant to be snarky (though I can see how it would come across that way). Being able to Google your way out of something you don’t understand is, tbh, the most important skill I think exists wrt programming. I probably search Google over 100x a day for answers on computer questions I’m working on. I was hoping I could show you how to break down your question/problems into chunks and research them yourself.

            Best,
            – Zane

  • Firstly, Zane, thank you very much for this write-up. I am very interested in getting this set up. Secondly, I am having a variety of issues starting with the correct setup of the RPi. I have tried to sign up for you mailing list multiple times and keep getting a system error from your end. I’m assuming this is what I need to get access to the build guide.

    I am not using RPi hardware any longer. Rather, I am using Proxmox VM’s but setting them up as RPi’s. This has worked well. I have added a second USB Ethernet adapter and the basic Debian install is working. Python 3.7.3 is installed. I have another VM already running with Grafana, Influxdb and also successfully installed Prometheus.

    I’m not sure where my problems are but believe once I can review the build instructions, I can troubleshoot it. Frankly, I am confused regarding the proper configuration of the ethernet. I am using a Cisco router with Ubiquiti network switches and AP. My home network is on 192.168.2.x.

    Any assistance and guidance is appreciated. Thanks in advance.

    • Hi Mike, thanks. Not sure what you mean about the system error; the mailing list is hosted by Mailchimp. You can drop me a line on the contact page, if it helps.

      I’ve never personally used Proxmox, though I use a LOT of Docker+Kubernetes. I can’t quite tell what your problem is based on the description, but I can say that VMs really mess with networking. In the case of Docker/K8s, everything happens on a virtual private bridge network… isolated from the main network. I’m sorry that I don’t know enough about Proxmox to comment if this is specifically related to your case or not. One way to test this would be to get into a command prompt inside the VM. If you can ping the Cisco router at 192.168.2.1 (or whatever), all is well.

      Next, I’m not sure exactly what you’re trying to accomplish with the VM per se. The crux of this post is that the RPi can “spy” on the traffic flowing through it. It requires two physical ports, bridging the traffic from the LAN to the WAN. The Raspberry Pi must act as a pass-through for the traffic, which can be done via the two methods described in this post. Once that is the case, the github repo / script will work to collect stats on that traffic which is already being passed through the Pi.

      • Thanks, Zane. I tried again to sign up but was again rejected due to a “system problem”. If you are using MailChimp, it may have something to to with running PiHole on my network.

        There is no magic to Proxmox. I have used Debian VM’s to replace all the utilities that I used to have running on RPi’s. I have no problem pinging my Cisco router. I have a second USB Ethernet adapter that is seen by the VM. I will try again. I also just learned that I can mirror a port on my Cisco router which may allow me to mirror the WAN port for monitoring. I’ll post an update in the coming days/week and let you know if I’ve gotten any further.

        • Got it. I guess I’m used to having to work around the K8s abstractions 😉 That’s interesting about port mirroring on the Cisco router. I had no idea such a feature existed (I’m just a hobbyist). But it seems like a great way to avoid the bandwidth bottlenecks you might otherwise impose with a Raspberry Pi pass-through (?).

          • After a few days of research and tweaks, I have this mostly working.

            I did not use RPi’s since I did not want to purchase more RPi’s. I have several older RPi 3’s but did not want to invest in RPi 4. I moved to Proxmox VM on an Intel NUC some time ago for my utility devices.

            I created a new VM using Debian 10 (Buster) and struggled a bit to add a second USB network adapter. But I finally succeeded in getting it working after tweaking /etc/network/interfaces. I now had two working NIC’s on the VM.

            After researching promiscuous mode some more, I was able to set one adapter into promiscuous mode but was only able to read broadcast packets. And this made sense since I still was not reading ALL network traffic.

            I have a Cisco SMB router and Ubiquiti switches and AP. I was able to mirror the outbound port of my Cisco SMB router (all network traffic) to another network port which I plugged into the second network adapter on the VM. After running a test with tcpdump, I was able to see all network traffic. I also tested it with the Python script and verify data was flowing to Prometheus. So far, so good.

            Next I added the Grafana dashboard and this is where I am currently stuck. I validated that I can read the ntm data elements in both Prometheus console and Grafana. However, the Grafana dashboard is not working. I suspect it has something to do with the queries and the exclusions. Any suggestions on troubleshooting this are appreciated. Many thanks in advance.

          • Progress! The issue was that the regex used in the Grafana local server dropdown was filtering to 192.168.0.x subnet. My home network is on 192.168.2.x. Once I updated the regex, the Grafana dashboard worked!

            I still need to do some performance analysis to see if this will work using a Proxmox VM long term. Thanks so much, Zane, for making this available. I learned a lot during the implementation on my home network.

  • Hello again! I’ve successfully gotten my Pi running as both a router and network monitor. As I’m error checking it before implementing it, I have a question about name resolution for local hosts.

    I put all of my local devices in /etc/hosts, and from looking at the NTM metrics, it appears they are resolving correctly:

    ntm_bytes_total{dst=”Zero”,proto=”tcp”,service=”https”,src=”1e100.net”} 25187.0

    However, in Grafana, all the instances of that host are showing up as the IP 192.168.0.10 in the By Host, Bytes Transferred, and individual detail dropbowns.

    Since it looks like the data Prometheus is getting has the alias, how can I get it to display it in the graphs?

    • That’s very strange. Based upon the Prometheus line you pasted, it’s not even recording the local IP address. Are you sure you’re not accidentally viewing old data? You may want to use the data explorer directly in Grafana, not the pre-built dashboard. Try filtering for the device on both the sending and receiving side.

  • BLUF: If your router resolves local devices you need to edit your REGEX field for the local network variable.

    Hey — excellent guide. Your ability to step through things is uncanny.

    Wanted to highlight an issue I had. I run opnsense (fork of pfsense) and a netgear switch that can port mirror. I basically mirror everything coming into the opnsense router from my wifi network (this represents 90% of traffic, only my main PC and pihole are on the wire). The mirrored packets go to a raspi that runs the monitor script — all good I see everything.

    Problem was the opnsense resolves all of my local devices. So the graphs showed nothing as variables that define the local network contains only the private address space. I replaced the regex with [A-Za-z0-9\.\-]{0,} and it starts showing stuff. I have a couple of things left to massage before its all right, but thought I’d put it out there that resolved devices on the local network will break the graphs.

    Thanks again for this write up!

    • Hey Jason, thanks for saying so!

      Good point about the resolved devices. Since that didn’t seem possible with my setup, I indeed coded it to look for IP addresses instead. But I would certainly prefer to have cleartext names instead of obscure IP addresses, myself. Maybe eventually I’ll figure out how to do this without opnsense. It seems like since I’m running the DHCP server on the Pi anyways, it should be able to resolve these names…

  • I’m able to explore the data in Grafana. But when importing the dashboard, there is no data to show. I have checked and metrics are being produced just not in the dashboard.

  • I have been looking around at the many network monitoring tools and the overall data capture and presentation of yours cannot be topped.

    I am also planning to install piHole on my system according to [link to www.smarthomebeginner.com]. I saw a comment earlier in this post but wanted to get a fresh one going. How would I add that server in your above diagram. I am also thinking that piHole should be on a separate RP unless you think differently. Any help with the settings would also be appreciated. I am a bit newer to the RPi and my Unix days are decades back.

    Any suggestions would be appreciated.

    • Thanks! Glad you found it useful.

      TBH, I would highly recommend AdGuard Home over PiHole. The Home Assistant folks switched over the entire community many months ago, and I agree with that decision. It’s much easier to use. I run it on my router, actually. I just installed it via the official instructions on the router. No other steps required, IIRC. But you could equally well run it on a different Pi.

  • Great write-up. I was able to follow everything with no errors. The piece I’m struggling with is: “Open your web browser to [link to 192.168.0.1] to see the counters being exported for Prometheus.” I get a timeout from the raspi.
    I currently have (WAN)->Raspi_WAN->Raspi_LAN->WirelessRouter_Bridged->Ethernet_Client. If I force stop the script on the Raspi, I get something like this:

    pi@raspberrypi:~/network-traffic-metrics $ sudo python3 ./network-traffic-metrics.py -i lan “src net 192.168.0.0/24 or dst net 192.168.0.0/24”
    tcpdump: listening on lan, link-type EN10MB (Ethernet), capture size 262144 bytes
    [SKIP] 04:09:52.472470 IP
    ^C914994 packets captured
    915130 packets received by filter

    • Thanks! First, just to address the obvious — have you checked that you’re trying to connect to the correct IP address? If the IP of your Pi is not `192.168.0.1`, you will need to change it. If that is correct, try SSHing into the Pi and doing a `curl localhost:8000/metrics` to see if you can access the Prometheus endpoint from the pi itself. If that works, then something about your network topology (firewall?) is preventing the other machine from accessing the Pi. If it doesn’t work, then for some reason the script is not listening or creating the webserver. Check out `journalctl -xe` to look for errors reported by the script, perhaps. You could also try changing the port it listens on on, e.g., `–port 8001`.