NetBehave.org

NetBehave - the story - architecture

2019-03-03T19:30:00+00:00

Summer 2018, in hiatus between projects I got working on taking these to the next level.

I leveraged the experience of the 2 past years to create a full fledge app.

Based on more than 20 years building software, and over 15 in information security, and over 2 years of thinking on and off about the Netflow analysis approach, I used the following principles to guide me in my design: Built on the Unix philosophy: make each program do one thing well; this would also allow me to create building blocks for different use cases; Minimize attack surface - what threat modeling calls trust boundaries; Build iteratively with continuous testing (individual/manual CI/CD): whether you call it agile, don’t wait til the end to make sure all is working well (and my initial not completely defined requirements called for an “agile” methodology anyways).

I had been wanting to learn Docker for some time, but either lacked time or a good project for that purpose. This seemed like the right time to get back to it. Docker containers are basically application level virtualisation. It allows for better process isolation than running regular processes, but is not considered full segmentation. This allows to create “blocks”, meating one of my first goals. Blocks were also provided by the Fluentd plugins. These “blocks” are the core plumbing that I can use to tailor to different business needs (aka make the system extensible). It can allow us to create various “solutions” from the existing building blocks, as well as by creating other custom blocks as needed.

As I was building simple building blocks, I went for docker-compose, the simple way to create a fixed set combination of docker containers.

Docker is often used with other technologies such as Docker Swarm and Kubernetes. Those 2 technologies are used for dynamically start containers based on rules.

Kubernetes, docker swarm and other technologies would be other options but were beyond the needs of this version (but could be required for larger organisations).

Collecting NetFlow/IPFix

NetFlow/IPFix runs on UDP, which means that packets can be lost, moreso over long distances and/or with traffic spikes (flows will also be delayed a few minutes, but are near real-time). So it made sense to split the collector of NetFlow/IPFix into it its own container. This would allow us to have multiple collectors in different locations. In previous versions, I had complex fluentd configuration including copies of messages due to different field in different NetFlow/IPFix versions and templates. So I decide to create another plugin to normalize the information into a “flow” object. As I was testing the system on my home network using flows from my trusty Ubiquiti Edge RouterX, I also found another issue. I saw a lot of flow coming from an external IP hitting my external interface (NATed traffic). That information was not very useful, so I create yet another plugin to DeNat by keeping a list of outgoing connections from my internal network. The last plugin used in the container is a standard Fluentd plugin that forward traffic. Note, this is one for internal (trusted) networks as no encryption is performed; a secure forward plugin exists.

Now I created what I called the core that received using the standard Fluentd forward input. Now I had a flow of normalized traffic, but ip addresses are not that useful by themselves. The beauty of tools such as Fluentd (which I describe as the Unix/Linux command line on steroids in JSON) is that it allows for easy modification of objects through the plugin system. So I decided that I wanted to add domain names from the DNS to the feed. Initial tests trying to do reverse lookups of the IP address had a huge performance impact. So I went about it in another way.

I decide that I would create my own internal DNS server (I was running it on my own network) to save the IP address/domain name pair that would be returned to the internal system. Dnsmasq worked well for this and provided great logs. It would obviously run in its own container (the DNS server). But it had one flaw, it uses unencrypted dns (udp and tcp) that can be eavesdropped, and even potentially modified on its way to other dns servers. So I looked at other options. I found that unbound works well for my needs, but the logs it provided were not detailed enough for me to parse. So I decide to combine both, and have dnsmasq forward requests to unbound (in its own container, DNS Server TLS) which would go outside gather the required information.

The dnsmasq logs were written to a shared folder of the app, and read by another Fluentd-based container: DNS Log Parser. This one consisted of a standard tail input plugin, a new filter plugin to parse the logs and transform that into messages the different log events (query, reply, cached). Then I created an output plugin to send that information into some form of a cache.

The DNS cache

While I had used MySql in the past, I wanted to keep the information in each container for this version. I had been using SQLite in other projects (such as the software I used to manage versions of my book) so I decided this would do the trick. But the DNS cache would need some form of security model and user roles. I opted for having no “authentication” per se, but using 3 different ports for different roles (read-only, read-write, admin), and restricting access to those ports instead (through exposing ports in docker). Since all fluentd plugins are written in Ruby, I opted to keep using that same programming language. I ended making that basic server with roles a class that I could reuse (which I did for the IPAM module). The cache uses a “least recently used” model that I capped at 1024 items, but that number is a parameter. Once it reaches the max size, it removes at least ¼ of all items. The DNS Log Parser connects to the read-write port, while a plugin in the core container (reverse DNS filter plugin) connects to the read-only port.

An alternative to a DNS cache? In some cases I was not getting DNS responses. And reverse DNS was not efficient. So I went back to one of my “blackbox” pentesting approach: using Arin (.net). American Registry for Internet Numbers (ARIN) is a registry of IP blocks to registered organisations of Canada, the United States, and many Caribbean and North Atlantic islands. It does not include everything (Latam, Asia/Pacific, Europe), but it covers the largest blocks of addresses. Since it has an API, you can retrieve through its whois method. That information gets added to the “arin” sub-object of each relevant src/dst field of the flow object. That information is also cached in another SQLite database.

IPAM

IPAM stands for IP Address Manager, in this case that is basically a database of IP addresses found (within a defined internal network). As with the DNS cache, we have 3 different ports for different roles (read-only, read-write, admin). This container needed a database, so I used another SQLite one (asset.db). When a new IP address is added, a line is added to a JSON log file called ipam.new. We could also easily add a log file for changes in MAC address if we had such a need. The ipam.new file is shared with yet another Fluentd container (Alerting) which reads the file and sends an email (another standard Fluentd plugin) to notify a new IP address was found. Other alerting methods are possible through existing or new plugins.

Back to the core

A IPAM output plugin connects to the IPAM read-write port to update when a new IP is found (a temporary cache is used to prevent too many calls for performance reasons). I also added a filter plugin to add service information (protocol/port, which you can generally find in /etc/services on Linux/Unix systems). In my first version, I had created an ACL matching plugin, so I decide to create another filter plugin to match an acl. I used a csv local csv file for simplicity (and time reason) but that could easily be modified. The rules (one per line) include a name (that will be added to the rule once a match if found) and a series of field name, operator (=,!=,<,>) and value (which can include wildcards at the beginning and/or end). The fields are must all match (AND operation between all field matches). The search stops once a match is found, thus rules should be in the order from most specific to generic. The easiest output was the using the standard Fluentd file output plugin. But I wanted to easily be able to query that data. So I created another output plugin using SQLite has the output format. But I wanted to ensure the file sizes did not get too out of hand, I so opted to make it a daily database file (thus was born the out dailydb plugin). But what if you wanted to search in more real-time? Well I guess you would need some sort of a search plugin… You could use the same ACL matching code from earlier, but how do you get that to a user? That’s where we’re heading nexT

The web interface

I wanted a simple interface to provide easier access to the system. I used the httpd server part of Alpine (for size and easiest config), but you could use Apache or nginx. I only used basic authentication and no encryption (I would recommend adding it for production). All web interfaces use static files (html, css, javascript) based on bootstrap, and Ruby-based CGIs. Yep CGIs. Using the read-only port of the IPAM, it was easy to build a simple list of IP assets. The daily DB files are listed in another page which allows to download the file (from previous days, as the file will be opened for writing during the day). This can allow to setup the system at the client and have the client download the file and upload it to us for offline analysis. Now we come to one of the most complex pieces of the system, the Flow Search portion. It is composed of a static page that connects to what I call the “middle” server. This Ruby-based server is a threaded application that opens a pair of connections each time the static page asks for a connection. For the client it creates a websocket connection (a technology similar to TCP sockets within web browsers) and a TCP connection for the core. A file of commands is used to tell the core what to do, from connecting to performing the searches. Once the browser connects a open command (with the port number) is written to the instruction file read by the core search output plugin. The browser can send request for an ACL match to the websockets, which gets written to the instruction file, and the search plugin returns the matching values through the TCP socket, which are funnelled to the browser via the websocket.

So now we have an initial set of 9 containers, 4 Fluentd-based, 2 Ruby-based TCP servers, 2 regular Linux applications, and a web server with simple Ruby CGIs and a intermediary server. The containers are bound together using docker-compose, with some shared file systems for logs and data. But how do we get logs properly managed, within this environment. The solution I found for this (and other things) is S6.

The container architecture

You may or may not have heard of Dan J. Bernstein (DJB), the famed Chicago professor. He proposed the famous elliptic curve called Curve25519. He his the author of qmail and tinydns. Qmail used a subsystem called daemon tools that allowed for process supervision. s6 is a process supervision suite much in that line that provides run script support (for starting processes - for example httpd and the ruby middle.rb server in my web container). It also provides support for changing permissions prior to starting, and to do logging and log rotation capturing the stdout standard output. I also found people who were using it within docker containers.

Since supply chain management (managing the risk of the subcomponents you do not control, and there will always be some) was a concern, I decided I would leverage docker creation examples found online, but would repackage them myself. When docker builds an image, it creates layers for most commands it runs in its build process. So I decided that I would try to limit the number of individual layers and have all my containers derive from as few as possible. Alpine Linux is the most commonly used Linux distro for docker. It allows for some of the smallest image. So my base image for all containers (called netbehave-docker-base) was Alpine with the s6. This image is never instantiated, but is referenced by other images. From this image, I derive all the non-Fluend containers. For Fluend containers, a contained another subimage (netbehave-docker-fluentd) that is similar to the base image, but had to be separate since integrating Alpine, s6 and Fluentd gave me headaches. Most custom-written plugins used in the subcontainers are copied in that base image. Each docker image I created copies the rootfs folder to /, simplifying my edits of files including configs, code, and scripts.

NetBehave - the story - mvp versions

2019-03-02T21:00:00+00:00

My presentation at the BSides Ottawa 2018 conference on November 8th, 2018 covered some of this, but I figured since the talk was not recorded that I’d present a more detailed recollection of how all of this project evolved. I thought this would have been a quick post, but I now realize that I want to put a bit more depth in this. This post will cover the initial versions of tool in 2016 and 2017. The next post will cover NetBehave.

Without further ado, let us begin.

v1.0 - The MVP

MVP, is a commonly used term in agile and startup worlds that stands for Minimal Viable Product. One common analogy, is looking at a car. The MVP for a car is not just the steering wheel, but just enough to get moving. as shown in the image below (taken from this post).

This project started almost 3 years ago (in spring 2016) when I got hired as a consultant to help an organization achieve PCI DSS compliance in a short amount of time. In order to have any inklings of a chance of remediating this quickly required massive scope reduction from an acquired organization’s flat network. In the end, we achieved a reduction to 20% of the original scope (CDE and connected, see my model for detail) which gave us the chance we needed to pull that off. There were more efforts than these tools (it was a team effort), but the scoping effort was instrumental (many remediations were also required, most of which I also led).

When doing my initial research, I was looking to get access to raw data in a format that I could use in whatever way made more sense. I also had little budget. I did look at many tools, but I finally had to choose between two similar systems: Fluentd and Logstash. Fluentd won the day by being lighter weight (half the memory according to comments at the time).

So, I built the tools iteratively. Initially, I fixed the Fluentd Netflow plugin (there were a few bugs) and added IPFix support (also referred to as Netflow v10, still missing to this day)… all without knowing Ruby (the language Fluentd is written in). I used other standard Fluentd plugins (filter_grep, out_file) and made sure the output was a CSV file. Then I just used standard command line tools (grep, cut) and Excel to do an initial analysis.

As we created new network zone behind a firewall, I had to find a way to validate the firewall rules I had defined, and thus started creating my first plugin to match traffic to predefined rules (again in a CSV format). Plugins can greatly simplify the work that can be done through existing Fluentd plugins and configuration. I later added MySql support (through existing plugins) so I could facilitate access, but I’ll get back to this later. We used the tools and reduced a flat network of about 800 (mostly virtualized server) to a segmented one and dropped the scope (CDE and connected systems for PCI DSS, see the aforementioned model for more information) to under 20% of this (~160 servers). We also used the tool to create and validate the required firewall rules, and removed the “ip any any” rule live during daily operations, with no notice from the business. I only missed one less regular application, a Veeam backup process that a sysadmin inquired about later that same week. This reduction in scope allowed the team to meet PCI DSS compliance in only 19 weeks. Note that this is compliance (focus on protecting payment card information only), but still an impressive feat (we also had a very talented and competent team).

v2.0 - A slightly more user friendly version

As we moved from PCI DSS compliance to addressing security of the combined organization holistically, we went at expanding the products we used focussing on the CIS top 20. This is what our initial list looked like late summer 2016.

CSC #	Control	Solution
1	Inventory of Authorized and Unauthorized Devices	TSC, Netflow
2	Inventory of Authorized and Unauthorized Software	TSC
3	Secure Configurations for Hardware and Software on Mobile Devices, Laptops, Workstations, and Servers	TSC
4	Continuous Vulnerability Assessment and Remediation	TSC
5	Controlled Use of Administrative Privileges	???
6	Maintenance, Monitoring, and Analysis of Audit Logs	TSC + FEK411
7	Email and Web Browser Protections	Proxy, MobileIron, Phishme
8	Malware Defenses	A/V
9	Limitation and Control of Network Ports, Protocols, and Services	Netflow->SharePoint
10	Data Recovery Capability	Processes
11	Secure Configurations for Network Devices such as Firewalls, Routers, and Switches	TSC
12	Boundary Defense	FireEye, Palo Alto
13	Data Protection	DLP, FireEye
14	Controlled Access Based on the Need to Know	HR+processes
15	Wireless Access Control	Airtight,TSC+Netflow
16	Account Monitoring and Control	TSC + FEK411 + PAM
17	Security Skills Assessment and Appropriate Training to Fill Gaps	LMS
18	Application Software Security	F5(ASM), Burp, SDLC
19	Incident Response and Management	Forensics with FireEye
20	Penetration Tests and Red Team Exercises	External Firm

As you can tell TSC (Tenable Security Center) was out main tool, with my Netflow tool (what became NetBehave) a close seconds. The FEK411 would be the evolution of another tool I built to meet PCI DSS compliance. It consisted of NXlog community edition agent on Windows devices to send windows security events to a rsyslog server (which also received regular syslogs), which was read by another Fluentd instance and parsed. It had a simple Bootstrap-based web interface to review logs (that code is similar to what I did for the web portion of NetBehave). It would have added Elastic Search and Kibana (of the standard Elk stack), and 411, a tool developed by Etsy which I saw at DefCon that summer. There were a few tools and firms involved, but I never got to see this through as I left for other opportunities later that year.

But the tool continued to improve nonetheless during that fall. As mentioned previously, I wanted something that did not require transferring files back to my PC. A great number of Fluentd plugins have been produced by the community that one could leverage. I opted for MySQL as a backend. This made it easier, and allowed me to make the required reporting.

A cleaned up version of those tools is basically what I presented at the BSides Calgary 2017 conference and you can find the slides and code on the blog of my book’s website.

In the next post, I’ll describe the design and implementation of NetBehave.

Introducing NetBehave!

2018-11-13T00:30:00+00:00

I’m happy to announce the official release of the NetBehave that I presented at the BSides Ottawa 2018 conference on November 8th, 2018 in, obviously, Ottawa, Canada.

The code for the system is now live. You can get a copy of the presentation here.

Getting started

Step 1. Get a copy of the code (download zip/unzip or git clone)

git clone https://github.com/netbehave/netbehave.git

Step 2. Edit docker-compose.yml file to edit the ENV variable values for netbehave-alerting and netbehave-core.

...
    netbehave-alerting:
        image: netbehave-alerting
        build: ./netbehave-alerting
        environment:
        - ENV SMTP_USER=account@example.com
        - ENV SMTP_PASSWORD=PASSWORD 
        - ENV SMTP_RECIPIENT=account@example.com
        - ENV SMTP_SERVER=smtp.example.com
        - ENV SMTP_PORT=2525 # Is 25 blocked by default?
        volumes:
        - logs:/var/log
        - data:/opt/netbehave

        
    netbehave-collector-nfipfix:
        image: netbehave-collector-nfipfix
        build: ./netbehave-collector-nfipfix
        expose:
        - "2055/udp"
        ports:
        - "2055:2055/udp"
        environment:
        - ENV LOCAL_NETWORKS="home=192.168.0.;docker=172.17.;NAT=1.1.1.1"
...

Step 3. Go to the folder and run: docker-compose build [sudo may be required]

docker-compose build

Step 4. Go to the folder and run: docker-compose up -d [sudo may be required]

docker-compose up -d