TECHPRO.MS

I’m excited to announce that the Network Monitor 3.2 Beta has released. There are some exiting new features listed below of which many I have already spoken about in this blog:

http://blogs.technet.com/netmon/archive/2007/12/28/santa-s-bag-was-full.aspx

One feature I didn’t mention is our new capture buffer to avoid dropping frames. After adding our drop frames counter, we found out under certain busy situations it would go above zero and we just couldn’t have that. We now buffer the frames before parsing and displaying them. While this does add some complexities to capturing, it insures that packets are more reliably captured which is obviously very important. I will blog about this feature specifically, but I wanted to call it out here. Where are the bits?

As usual we are hosting our beta at http://connect.microsoft.com under the Network Monitor 3 Project. If you are already signed up, you should have it listed on the main page. If not, look for the project and join up with us to help us find bugs. There is also a web page forum front end to our newsgroup if you need to get support or ask general questions. What’s New since Netmon 3.1

• Process Tracking: View all the processes on your machine generating network traffic (process name and PID).  Use the conversation tree to view frames associated with each process.

• Find conversations:  Quickly isolate frames in the same network conversation.  Isolate TCP streams, HTTP flows etc.

• PCAP capture file support
• Capture engine re-architecture to improve capture rate in high-speed networks.   Network Monitor 3.2 drops significantly fewer frames that NM3.1
• Extensive parser set:  Parsers for over 300 protocols!  Parsers for the protocols covered by the Windows Open Protocol Specifications (see http://msdn.microsoft.com/en-us/library/cc216517.aspx).
• NM API: Create your own applications that capture, parser and analyze network traffic!
• Better parser management:  By default only a subset of parsers are loaded.  You can load the full parser set by changing the parser search order in Tools>Options>Parser
• Support for frame truncation.  Go to Tools>Options and limit the number of bytes captured per frame to improve performance.
• More extensive documentation of the NPL which includes documentation on the new NMAPI.  Access the documentation from Help > NPL and API Documentation
• Enhanced filtering on items within NPL while loops or arrays.  You can specify an index into the array or while loop to filter on
• IA64 version now available.
• ContainsBin Plug-in:  Search frames for arbitrary byte sequences or strings. For example, ContainsBin(FrameData, ASCII, “msn”).
• More UI indications of conversation status, dropped frames and the number of frames in the capture buffer.

• … and more.  See our Release Notes in the NM3.2 installation directory for a complete list of new features and known issues with the Beta.

Enjoy!

The Network Monitor Team

A few weeks ago I visited the glitzy town of Las Vegas for the Interop Conference 2008. While I’m not much of a gambler, the lure of technology was more than enough to get me excited about the trip. Pack thousands of tech-heads in a luxurious hotel, present information about new technologies, and now you’ve got a hot time in the dessert. Buzz Words: Unified Communication and Virtualization

I don’t know if they do word counts on submitted Power Point presentations, but if I had to guess, both UC (Unified Communications) and Virtualization would be at the top of the list. And both technologies have a direct impact on the future of Network Monitor.

In terms of Virtualization, there are many challenges. The host machine can now house dozens to hundreds of virtual servers which means the backbone connection to the host machine requires a quick link. This means a 1 gig or even 10 gig network connection may be the norm for these configurations.  Making sure we can support these environments from the host machine and hosted VMs is going to be an important scenario to understand and support properly. 

For Unified Communications, the range of devices expands as your phone/pc/video conference merges.  And as wireless environments with remote users becomes the norm, trouble-shooting connections for voice and video become incredibly important in this domain.  And finally, interoperability between multiple devices and infrastructures requires a tool that can easily determine where interoperability problems may exist. Roaming the Exhibition Floor Jungle

The Interop Exhibition Floor is a microcosm of the hosting city, Las Vegas. Sounds from various booths ring out like slot machines and shows start on the hour as vendors attempt to garner your attention for longer periods of time. Booth babes and circus-like performances try to steal your eye away, in hopes of attracting you to their line of products. For me, however, I was curious about products that compete with or compliment Network Monitor.

What I found is that in general the protocol analyzer type tool that seems to be popular is more a of an aggregate tool that tells you where your network is sick. It may not tell you what the exact problem is, but it helps you monitor your network as a whole. I suppose that is where our tool would come in. Network Monitor is more targeted for solving specific problems and diving deep. Our conversation tree and soon to release process tracking creates a unique way to take a lot of data and filter down to a specific problem. Soaking up the Technology

As I introspect over my trip I try to find connections between our product’s future and the current state of technology. I realize that as things continue to accelerate, the need for a way to drill down from the very large to the very specific is of vast importance. And while we can continue to do general things, it will be more and more important to provide technology specific tools to help troubleshoot the dizzying array of protocols, devices and network mediums. As we continue to reach forward in this regard, we should be able to use our community to understand and provide tools to help us all solve difficult networking problems.

We’re very excited about showing off the beta of NM 3.2. Watch for it on our site, (http://connect.microsoft.com), in about a month! If you register into our community we’ll send you a note when it’s ready.

We have decided to release an SMB2 parser for Network Monitor 3.1 (released July 07) to hold people over untill the beta for Network Monitor 3.2 releases in early June.

Where can I get the SMB2 parser?

You can download SMB2.NPL parser, along with SPARSER.NPL, CER.NPL, FCCS.NPL, SCNA.NPL and SMB.NPL (all supporting parsers) on http://connect.microsoft.com under the Network Monitor 3 project. If you’ve already signed up you’ll see it as one of your active projects. If you need to sign up you will need to create a passport account and join our project. Once you are in on the Network Monitor 3 project, click on the Downloads link on the left. You will see SMB2 Parser as one of the selections.

How do I use the new SMB2 parser?

Look at the article on using the SSL parser (http://blogs.technet.com/netmon/archive/2007/10/23/new-ssl-public-parser-available-how-to-deal-with-new-parsers.aspx) in the sections “Where do I stick it?” and “Working with NPL Parser path”. The instructions for installing the SMB2 parsers are the same.

Happy SMB2 parsing!

Ever wonder how a network works? Maybe it just seemed so easy, and in your mind sending a file was just putting each byte on the wire to the receiving machine. That’s not too far from the truth but you’d be very selfish to think that the network was there for your bidding only. Since a network has to be shared between many users, strategies have been created to chop your file up and send it in chunks.

We’ve all done it before. We take all of the bits and pieces of junk we’ve collected over the years and are tasked with moving them from one location to another. We invite friends and relatives, and like ants, they arduously move each piece from house A to house B. If your house=file, and boxes=packets, then we’ve created an analogy for reassembly. If you are organized, you’ll label each box, 1 of 20, 2 of 20, and so on. When you get to the new location, you can put all your things back together using the labels you created and verify you received everything.

The architects of our Internet are not any cleverer. They’ve borrowed these same techniques to determine how a chopped up file can reach its destination and be reconstituted. As long as the data is sequenced, the other side should be able to put the pieces back together. The Transport Layer

In networking we like to talk about layers. You may have heard terms like “Layer 3 switch” or the “Network Layer”. Well, the architects of our Internet are just like everybody else. They get easily confused if too much is going on. Transferring network data from one place to another is a difficult problem. And a common strategy used to solve difficult problems is to divide and concur. Each layer is responsible for different things, and the transport layer is the one we use to define how files get chopped up and rebuilt.

We’ll narrow our focus down to TCP for a bit. This is the workhorse transporter for most networking applications. There are others, such as SPX/IPX, but TCP is by far the most popular. And it turns out RCP, SMB and HTTP can also fragment data. SMB might chop your file up into 4K chunks, and then each of these chunks could be further fragmented by TCP. Labeling Your Boxes

So in our analogy, labeling=sequencing. We could label each packet 1 of 20 and so on, and some protocols do use this strategy. But for TCP, we go a bit further and describe the number of bytes that are sent. We’ll say, for instance, that this packet contains Sequence 1000-2000. What we actually send is the first sequence number and the size, but this range can be derived from that data. When the other side gets the data, even if it’s out of order, it knows how to put the puzzle back together. Also, the receiving TCP will keep the sender up to date by acknowledging what it’s received so far. In the simplest scenario, the receiver ACKs the latest segment it received. If the sender gets an acknowledgment for sequence 2000, then that confirms the receiver has seen all data up to that point. Using NM3 to Reassemble the Data

When we capture data from the network, we are capturing it before the data has been put back together. But this can make it difficult to read, as you might imagine. With NM3 we’ve created a way to reassemble the data, so that you can see the data as it is seen by the application layer (there’s one of those darn layers again, see http://en.wikipedia.org/wiki/OSI_model). So when you click on a web page and get a trace, you can see the entire packet as sent by the browser, rather than a bunch of fragments after TCP has gotten to them. Data Before You Reassemble

So if you start a network trace and view a web page you’ll notice that the traffic that gets created shows HTTP and TCP. The HTTP packets are the headers sent/received by your web browser. But the TCP that is in between is also your browser traffic. The original HTTP packet was larger than would fit in single packet, so TCP has chopped it up for you.

In this example, you can see that the server has responded with a page. Frame 6 is a continuation of frame 5 and is where TCP has chunked up the data. Frame 7 is an acknowledgment so the server knows we are receiving data. And frame 8 is the final frame in the fragmented data.

To reassemble with NM3.1, you go to the Frames menu and select “Reassemble All Frames”. In NM3.2 we’ve created a more prominent button on the tool bar so this should be easier to find. Once the reassembly is complete a new window opens and contains all the original frames PLUS new frames for each reassembled piece.

So frame 9 is the new frame we inserted. While the Description doesn’t look much different, the devil’s in the details. Quite literally we have to look at the frame details to see the difference. Frame: Number: 9, Captured Frame Length = 4449, MediaType= PayloadHeader + PayloadHeader: Re-assembled Payload + Http: Response, HTTP/1.1, Status Code = 502, URL: http://239.255.255.250/

The major difference is that we don’t see Ethernet, IP, or TCP anymore. We’ve replaced the original network header with our own called PayloadHeader. This header contains info about the protocols that have been reassembled, as well as information we might need from those layers. We don’t show the original frame numbers, but you can enable some debug NPL to get this information if you need. Just look at Payload.NPL for more info.

If you were to open up HTTP Response in the details, you would also see that all the HTML and header information is in this one packet. This reassembling of the data makes is very easy to understand that data from the HTTP level.

Also realize that in cases where both HTTP and TCP are fragmenting data, there can be multiple levels of reassembly. In those cases you may see a PayloadHeader for each TCP fragmentation, and then another for the HTTP fragmentation. Reassembly FAQ Reassembly doesn’t seem to be working for me, why?

The reassembly feature does depend on two things. One, conversations have to be enabled. Make sure this option is enabled when you open a trace. Currently the option can be found on the start page. Two, you can only reassemble a saved trace. If you just started a trace, you’ll need to save and reopen it. Why do all the original frames show up?

When NM3 does reassembly, new frames are added in. Rather than remove the original frames, we’ve decided to leave them in as they still provide information about the original TCP data. Also, if there’s a problem with reassembly, perhaps due to missing packets, you could use the data to figure out why it didn’t reassemble correctly. Can I only see reassembled frames?

Well yes, you can filter on only those things that have been reassembled. As we mentioned above, we add a PayloadHeader, which is just a protocol we’ve devised. So you can filter out all reassembled data, but applying a filter of “PayloadHeader”. I actually create a color filter for this so they stick out.

What you can’t do, however, is see only data that has been reassembled and the data that was never originally fragmented. While we realized this would be useful, we haven’t exposed a way to make this possible. However, with the API and NM32, it would be possible to do this programmatically. Why doesn’t the Reassembly window have a conversation tree?

When we originally designed reassembly, the conversation window wasn’t really integral to the product yet. The simple work around is to save the capture file and reopen. Putting it All Together

Networks have devised ways to break apart data and rebuild them. But with protocol analyzers that capture this raw data, it’s sometimes difficult to follow. With the reassembly option in NM3 your network traffic is put back together making it easier to read. Add this ability with the new Process Tracking feature in the upcoming NM3.2 and finding the only the data you need will be easier than ever.